Last week I had the pleasure of running
some training and mentoring for moderating usability tests. I have been able to
gain a lot of experience in running usability tests; working for one agency I
was running up to 8 sessions a day! Within the industry there is a variety of
skill levels and expertise in this area. I have witnessed moderators who act
like a chameleon, adapting to each participant and skillfully eliciting
feedback like a puppet master and have also been horrified with clumsily
structured sessions where the participant was uneasy and plied with direction
on how to act and left in no doubt about the feedback the moderator would like.
Moderation is not a difficult skill to
learn, however the difference between good and bad moderation is huge and can
have major implication for your project. I genuinely believe that anyone can
learn the skills required with the right guidance. Like any skill, moderation
takes time to master and mentoring can help. This post covers some of the
basics for moderation; getting these right should lead to an effective
usability test.
1. Start on the right note: The key to a
successful session is engagement. From the minute you first interact with the
participant it is crucial to set them at ease and make them feel comfortable. This
rapport building is about gaining the participant's trust as well as letting
them know you are in charge. Mutual trust building helps participant feel
comfortable that the session will be run in a manner where they are not going
to be judged or made to feel uneasy. This comes down to a friendly but
professional demeanour. Rapport can be built quickly, you need to be friendly
and warm, be aware of your body language, ask some open questions about their
day, or job etc. actively listen to their small talk and reflect that you are
interested in what they have to say.
2. Make the process crystal clear: Setting
boundaries with participants as well as providing a clear understanding of the
process is crucial to making sure that they know what to expect and what is
expected of them. This can overcome some potential issues regarding the anxiety
some participants feel about undertaking a test,
the desire of participants to please the moderator (tell the moderator what
they think you want to know) and their potential diversion into trying to solve
the design issues during the test.
A good script can be utilised
to communicate key parameters to participants including:
a.
That you want to understand how
they would use the interface in their typical environment.
b.
That you will evaluate (and
ultimately improve) the website by observing their interactions with the site and
that with this understanding, you will work out how it needs to be designed; make
it clear that the participant doesn’t need to worry about re-designing the site
during the session.
c.
That you are not testing them.
A successful usability test is one where participants behave as normally as
possible (or as close as is realistic in a unnatural setting). Other tactics
can be employed such as avoiding the term “task” and using “activity” instead.
Also avoid comments like “good” or “well done”. While it is unrealistic to
total eradicate the sense that they are being testing, it can be minimised.
d.
That you are independent of the
design and that you won’t be offended by negative feedback. Independence is
crucial to promote honest feedback (see Usabilitytesting: Does independence matter?) The reverse can be true, making out that you
expect negative feedback can also adversely effect a session.
3. The masterful art of
deflection: Some participants will seek out assistance
from you as the moderator (a good introduction script should reduce the
likelihood of this happening). When this happens the key is to deflect the
question while maintaining the participant’s engagement. For example deflecting
the question poorly such as, “My role is not to answer your question” can
actually do harm by making the participant feel that their feedback is
unimportant. A better approach is to use phrases such as “What do you think?”
or “Let’s discuss that in a moment”. Maintaining a respectful tone of voice is the
key to deflecting the question.
4. Not leading participants in their discussion or behaviour: Eliciting non-prompted and honest feedback is fundamental to the
role of effective moderation. The way you phrase your questions to participants
can lead to completely different responses and it is important that
participants are not lead to an answer. An example of a leading question would
be “Did you find creating a password difficult?” The use of a non-leading
question such as “how did you find creating a password?” will elicit a more
true response.
5. Get comfortable with silence: Silence is
one of the most effective moderation techniques. As outlined above, questions
can impact a participant’s response, especially during a task. This presents a
challenge, as any stakeholder viewing the session will want you to ask why, “why did the participant select that option?” “Why
didn’t the participant sign-up?” etc. Instead, a better choice is to encourage
natural behavior by deferring discussion to after the activity or to an
appropriate pause.
Silence is difficult, and
sometime you want to encourage your participant to continue – using some
minimal encouragers, such as “I see”, “tell me more about that”, “and then…”, are
a nice supplement to silence, if you feel your participant needs encouragement
without impacting on their train of thought or their subsequent behaviours.
It can be argued that
asking probing questions as things happens is likely to reveal more insightful
feedback and at times this can be of value. You need to ask yourself, will
asking this question now have an adverse impact on behaviour? If so, does the
insight gained outweigh changing their natural behavior.
6. Listen: As someone much smarter than
myself once said, “we have two ears and one mouth so that we can listen twice
as much as we speak” and when moderating we should follow this advice, and then
take it to the extreme. Active listening will make your participant feel
comfortable, make them feel heard, and allow trust to be built, hopefully supporting
them to participate in the test more fully. The best way to start active
listening, is to genuinely listen and show an interest in what the participant
is saying (don’t tune out – remember you don’t have to be friends and it will
be over in 60 minutes or so!). You can show an interest in their responses
through your body language/posture, eye contact, use of minimal encouragers,
asking open questions and use of attentive silence. Finally, active listening
requires the moderator to get some feedback from the participant that their
message is being understood in the way it was meant to be. This can be done by
reflecting back to the participant what it is that they just said eg.
Participant: “I didn’t like setting up the password, it was hard” – Moderator:
“so you are saying that the password set up was challenging?” Apart from this
showing the participant that you have actually listened, it allows you to check
that you have understood what the participant has communicated to you,
validating your conclusions.
Conclusions
Mastering the art of moderation can take
years, running lots of sessions and making lots of mistakes. That said,
learning to gain useful insights from usability testing can be quite straight
forward. The most crucial elements are learning to listen, making it clear to
the participant what is expected of them and starting on the right note.
