Last week I had the pleasure of running some training and mentoring for moderating usability tests. I have been able to gain a lot of experience in running usability tests; working for one agency I was running up to 8 sessions a day! Within the industry there is a variety of skill levels and expertise in this area. I have witnessed moderators who act like a chameleon, adapting to each participant and skillfully eliciting feedback like a puppet master and have also been horrified with clumsily structured sessions where the participant was uneasy and plied with direction on how to act and left in no doubt about the feedback the moderator would like.
Moderation is not a difficult skill to learn, however the difference between good and bad moderation is huge and can have major implication for your project. I genuinely believe that anyone can learn the skills required with the right guidance. Like any skill, moderation takes time to master and mentoring can help. This post covers some of the basics for moderation; getting these right should lead to an effective usability test.
1. Start on the right note: The key to a successful session is engagement. From the minute you first interact with the participant it is crucial to set them at ease and make them feel comfortable. This rapport building is about gaining the participant's trust as well as letting them know you are in charge. Mutual trust building helps participant feel comfortable that the session will be run in a manner where they are not going to be judged or made to feel uneasy. This comes down to a friendly but professional demeanour. Rapport can be built quickly, you need to be friendly and warm, be aware of your body language, ask some open questions about their day, or job etc. actively listen to their small talk and reflect that you are interested in what they have to say.
2. Make the process crystal clear: Setting boundaries with participants as well as providing a clear understanding of the process is crucial to making sure that they know what to expect and what is expected of them. This can overcome some potential issues regarding the anxiety some participants feel about undertaking a test, the desire of participants to please the moderator (tell the moderator what they think you want to know) and their potential diversion into trying to solve the design issues during the test.
A good script can be utilised to communicate key parameters to participants including:
a. That you want to understand how they would use the interface in their typical environment.
b. That you will evaluate (and ultimately improve) the website by observing their interactions with the site and that with this understanding, you will work out how it needs to be designed; make it clear that the participant doesn’t need to worry about re-designing the site during the session.
c. That you are not testing them. A successful usability test is one where participants behave as normally as possible (or as close as is realistic in a unnatural setting). Other tactics can be employed such as avoiding the term “task” and using “activity” instead. Also avoid comments like “good” or “well done”. While it is unrealistic to total eradicate the sense that they are being testing, it can be minimised.
d. That you are independent of the design and that you won’t be offended by negative feedback. Independence is crucial to promote honest feedback (see Usabilitytesting: Does independence matter? The reverse can be true, making out that you expect negative feedback can also adversely effect a session.
3. The masterful art of deflection: Some participants will seek out assistance from you as the moderator (a good introduction script should reduce the likelihood of this happening). When this happens the key is to deflect the question while maintaining the participant’s engagement. For example deflecting the question poorly such as, “My role is not to answer your question” can actually do harm by making the participant feel that their feedback is unimportant. A better approach is to use phrases such as “What do you think?” or “Let’s discuss that in a moment”. Maintaining a respectful tone of voice is the key to deflecting the question.
4. Not leading participants in their discussion or behaviour: Eliciting non-prompted and honest feedback is fundamental to the role of effective moderation. The way you phrase your questions to participants can lead to completely different responses and it is important that participants are not lead to an answer. An example of a leading question would be “Did you find creating a password difficult?” The use of a non-leading question such as “how did you find creating a password?” will elicit a more true response.
5. Get comfortable with silence: Silence is one of the most effective moderation techniques. As outlined above, questions can impact a participant’s response, especially during a task. This presents a challenge, as any stakeholder viewing the session will want you to ask why, “why did the participant select that option?” “Why didn’t the participant sign-up?” etc. Instead, a better choice is to encourage natural behavior by deferring discussion to after the activity or to an appropriate pause.
Silence is difficult, and sometime you want to encourage your participant to continue – using some minimal encouragers, such as “I see”, “tell me more about that”, “and then…”, are a nice supplement to silence, if you feel your participant needs encouragement without impacting on their train of thought or their subsequent behaviours.
It can be argued that asking probing questions as things happens is likely to reveal more insightful feedback and at times this can be of value. You need to ask yourself, will asking this question now have an adverse impact on behaviour? If so, does the insight gained outweigh changing their natural behavior.
6. Listen: As someone much smarter than myself once said, “we have two ears and one mouth so that we can listen twice as much as we speak” and when moderating we should follow this advice, and then take it to the extreme. Active listening will make your participant feel comfortable, make them feel heard, and allow trust to be built, hopefully supporting them to participate in the test more fully. The best way to start active listening, is to genuinely listen and show an interest in what the participant is saying (don’t tune out – remember you don’t have to be friends and it will be over in 60 minutes or so!). You can show an interest in their responses through your body language/posture, eye contact, use of minimal encouragers, asking open questions and use of attentive silence. Finally, active listening requires the moderator to get some feedback from the participant that their message is being understood in the way it was meant to be. This can be done by reflecting back to the participant what it is that they just said eg. Participant: “I didn’t like setting up the password, it was hard” – Moderator: “so you are saying that the password set up was challenging?” Apart from this showing the participant that you have actually listened, it allows you to check that you have understood what the participant has communicated to you, validating your conclusions.
Mastering the art of moderation can take years, running lots of sessions and making lots of mistakes. That said, learning to gain useful insights from usability testing can be quite straight forward. The most crucial elements are learning to listen, making it clear to the participant what is expected of them and starting on the right note.