It can be challenging to run a group UX review, even when the participants know each other.  There is always someone who talks endlessly, raising nitpicks and alienating other team members, others who talk too little, and those at their back who are miles away doing their email.

Its also hard to cover all the material in an allotted time, particularly if a long interaction flow is being reviewed. I remember one usability inspection at Google where the aim was to review a 5-step signup process. Sounds easy, but we didn’t even make it halfway since we got bogged down with the visual design of the first page. This was a particular problem since the real usability and conversion challenges weren’t until the penultimate billing step.

For these and other challenges, its important to have a strong facilitator to keep things focused, constructive and rolling.

Its even more of a challenge in a room of 50+ strangers who – in a limited time period – want to give critical (and hopefully constructive) feedback on a product they haven’t seen before. This is the situation I find myself in San Francisco’s UX Eye for the Developer Guy Meetup, which I help run with – amongst others – Luca Candela, Ben Morrow, and Jenny Morrow. At “UX Eye”, startups in need of UX loving, are invited to present their product to a large group of UX practitioners including designers, researchers, and engineers.

When I run these sessions, its my job – as the facilitator – to keep things focused so as to cover all the material, to get as much input as possible from both loudmouths and those more softly-spoken, share some useful UX principles and methods, and hopefully be entertaining in the process.

Introducing The Voices In Your Head Protocol

My favourite approach for a group UX review is based on a simple protocol I call “The Voices in Your Head” (VIYH). Its an odd name for a UX methodology, but bear with me, all will become clear. A VIYH session consists of a modified usability test with one participant at the front of the room, systematically interrupted with audience input from the masses. This balance of methods – usability and group feedback – has several benefits:

  1. A well-moderated VIYH session is a very efficient way of systematically reviewing a complete interaction flow in a room of 50 strangers in a limited time. “Impossible!” you cry, well read on.
  2. The product team and audience gets to see an approximation of a one-on-one think-aloud usability session, glimpse some live moderation techniques, and observe actual user behaviour in the product. It still surprises me how few startups observe people using their products and are effectively “developing blind”. Getting feedback from a large audience can be stressful, but the product teams always learn something new and enjoy the experience. In fact, many of our participating startups are inspired to do user research themselves.
  3. The audience is forced to watch and think about actual user behaviour before conveying suggestions. I find this encourages deeper analysis, and a stepping beyond “visual design nitpicks” to more fundamental issues (for example, “the colour scheme is terrible” versus “the user can’t find the call-to-action to sign-up”).
  4. The whole process is a lot of fun – especially for the facilitator. See the catchphrases below.

How to run a VIYH Session.

Before the Session

As the facilitator, be sure to review the product in advance. This allows you to confirm the interaction flow to be reviewed, indeed make sure its ready to be reviewed(!), and come up with a scenario and persona for the usability session. In most cases, these are very simple. The persona tends to be simple as “a 20-30yo white collar worker who is not very tech-literate” (i.e. unlike everyone in the room), whilst the scenario might be “You have heard about product X from a friend. Go and find out about it, and sign up”.

Brief the startup team to make sure they don’t interrupt and inadvertently prompt the audience. I always encourage them to take good notes since the feedback often comes thick and fast.

If you are truly presenting to a roomful of strangers, chat some up beforehand, and select a suitable usability participant. Select someone focused and confident, but able to follow instructions. That last bit is important.

Schedule at least 40 minutes per product. This should be enough for an intro, a 4-5 step interaction flow, and closing discussion.

Starting Off

First things first, find a volunteer to be the usability participant and ask them to leave the room.  Then invite the Startup to give a two minute overview of the product so the audience have a reasonable idea what it is, and what specific UX-related questions they have. Ideally the audience would be completely fresh just like the participant. However, I find that an up-front overview helps keep things relevant and to time.

If the audience has limited experience in UX, and in particular in doing UX reviews, I present a quick checklist of things to look for. For example:

  1. Try and stay in the mindset of our target persona. You understand acronym soup, would our target persona?
  2. Is it obvious from the landing page what the product is? (with no background introduction).
  3. Follow the user’s actions, rather than looking around the page. Focus on the areas of the screen which get the user’s attention as they complete their task.
  4. Are there clear calls to action for completing each step of the task at hand.
  5. KISS.
  6. Is the wording concise and clear? (This is often an overlooked aspect of UX for a busy startup).

Brief the Participant

Now its time to get going. Invite the test subject back into the room. Tell them who they are (the persona) and the scenario (what it is they need to do).

Next the crucial bit. Tell the participant that you want them to go through a step at a time. At each step you’ll ask them what they would do next to achieve their goal, and to think-aloud through their decision process. Tell them they are allowed to move around with their mouse, but ask them not to click anywhere until you say (this step by step approach is derived from the Cognitive Walkthrough UX Inspection methodology). Depending on the sense of humour of the participant you might want to reassure them that its the product being tested, not them. Or not. I like to keep things light-hearted and relaxed, and find a joke about how “all the data collected will not leave the room” goes down well.

Brief the Audience!

Don’t forget about the audience. They have a key role too. Tell them that its important not to shout out and prompt the user. In fact they also have personas to act out. They are going to be “the voices in the user’s head”. At each step of the session, they will silently watch the participant do their thing. When the participant is finished, the audience then get their chance to give feedback on anything and everything. Tell them that as “Voices” they are continuously working in the background, but only speak when they are told to do so. There’s a lot of them after all, and the participant has a small head. Lastly, these Voices are not nitpickers. They like to pipe up if they see something they like, as well as raising areas for improvement.

Step 1: The Incoming Channel: google.com

I like to start the flow before the participant even sees the product. As facilitator, there is a very important point to make here – that the user’s experience starts before they get to your website. SEO, SEM and other marketing channels are UX too.

  • Typically I start off on google.com and ask the user to do a search to start their task, and review the corresponding search results. After, the user has indicated which result they would click, open it up to the audience for 30 seconds of SEO, SEM and product naming suggestions.
  • This is handy for getting some initial feedback on the name of the product and how easy it is to spell, a frequent challenge in these days of a cluttered DNS name space.
  • After 30 seconds of group discussion, tell the Voices to ssshhh. Ask the participant to click on their search result and/or type in the URL directly.

Step 2: The Product Landing Page

Now we get to the meat of the session as we start to look at the product. On each page (i.e. each step of the follow), the review proceeds through 2 distinct phases: (1) the usability participant speaks, and (2) the voices (the audience) speak.


First the participant gets a minute to look at the landing page and decide what they would do next to complete their task. The facilitator should follow standard usability interview practice. Let the user have some uninterrupted time to think and act. Prompt them to think-aloud and/or stay on track as need be.

  • If anyone from the audience says anything, ask them to be quiet since they are well-behaving “Voices in the participants’ head”. Depending on your style, you might like to shout at the audience and/or tell them to “shut up” in a mock stern voice. A cow bell can come in handy too.
  • Occasionally the participant has no idea what to do. They may not even know what the product is! This is of course great feedback for the product team. Let the participant “suffer” for a while, before asking the Voices for a suggestion as to what to do next.

When the participant has indicated what their next step would be (just indicated mind you remember you’ve ask them not to click), its the audiences’ turn. Ask the participant to freeze, and open up the proceedings to the audience:

  • Have some fun. Go drill sergeant on them if they are being quiet. Typically, I have the opposite problem and have to ask the Voices to put hands up and speak in turn. Cut them off after 20 seconds so you can get input from as many people as possible.
  • Since the whole audience is contributing, make sure people speak clearly. Don’t be afraid to ask them to stand up.
  • Voices are not allowed to interrupt other Voices.
  • Watch the clock. Particularly for less important pages with obvious issues, try to keep the group review concise. You can allow a few minutes more for the key steps with the main UX issues.
  • When time is up, tell the Voices to be quiet. Its time to give the participant control of their head again.

Step 3: Onto the next product page

And now its back to the participant. Unfreeze them. Congratulate them for listening quietly to the voices in their head, and ask them to do their intended action. Again prompt them to think-aloud, and stay on task.

  • If they get completely stuck, get a suggestion from the the audience, or give them a clue. Hopefully you as the pre-briefed facilitator know what the desired path is.
  • Again if the voices speak up, tell them to be quiet. Surely they must have got the message by now?

When the participant has said what they will do next, open it up again to the audience again. If not all of the would-be commenters had time to speak in the previous step, they will now be champing at the bit! Remember to watch the time.

Repeat Step 3

Keep cycling between the usability test, and the Voices until you are out of time, or more ideally have completed the interaction flow.

Close the Session

Finish the proceedings by asking everyone – participant and audience – what they see as the top issues and any recommendations for the startup. As facilitator, guide them to focus on the big significant issues – a nice exercise in issue prioritization.

Finally, I invite the startup to say a few words. Typically they are overwhelmed (positively!) with feedback, and have filled several pages with notes. There are always a few fresh surprise issues and ideas new to the team.

Conclusion

I find “The Voices in Your Head” protocol allows a facilitator to get a lot of focused feedback on an interaction flow in an efficient manner, balancing task focus and some objectivity (the poor user at the front), whilst giving everyone a chance to speak and have some fun in the process!

Let me know how it goes if you try it. And if you want to see it in action, swing by the next UX Eye for the Developer Guy Meetup in the SF Bay Area.

Thanks to Braden Kowitz for help with the smiley graphics!