Usability testing. Does independence matter?

I caught up with a former client recently and got onto the topic of independence in usability testing a design. They were lamenting the fact that there are very few companies who focus on usability testing without also providing their own design services. Having tested my own designs and tested the designs of my colleagues, as well as working as a third party brought in to assess an agency’s design, I have a strong view on the role of independence. I feel that it is impossible to be truly objective with your own design. This is not to say that you shouldn't test your own design. Confused? Let me explain.

Generally speaking, there are two questions which usability testing answers: 1) How usable is an interface? 2) How can an interface be improved? Often these questions are explored together, however identifying which is the most important will dictate whether independence is required.

When the goal of testing a design is to gain an accurate measure of how usable it is, the testing requires an independent practitioner with no vested interest in a positive or negative outcome. The person or organisation which created the design has an interest in a positive outcome which will impact their assessment of the interface's usability. This is not to suggest that a designer intentionally goes out of their way to present the results in a misleading manner, although I am sure this has happened. Put simply, the assumptions which were made during the design phase make it extremely difficult to objectively assess behaviour during a usability test. The outcome is more likely to be favourable because the practitioner is more likely to acknowledge the behaviour which supports their assumptions (the salience effect). Furthermore, an agency who has designed an interface is also compromised in their assessment of usability, even in cases where separate teams design and evaluate an interface. Again this is because they have a vested interest in a positive outcome. This is never more evident than during difficult projects and where budgets tighten.

There are a number of scenarios where I think independence matters:

• A project where significant investment has been made to design an interface
• Interfaces that will have a significant impact on the bottom line i.e. a new eCommerce platform
• Projects which are highly strategic for an organisation and numerous stakeholders are involved
• Projects where the team must report to a steering committee or senior management
• Interfaces that depart from industry or design conventions

In all of the above scenarios, being able to state that an interface has been independently tested and is usable is of great value to the designers and the project owners. It also represents a risk mitigation strategy for organisations in identifying readiness for deployment.

So having argued all the pro's of independence, when does independence become less important? When the goal of usability testing is purely focused on enhancing the design. Why? Because a good designer should be able to see the flaws in their execution and ultimately have an interest in creating the best possible design. Hopefully they are motivated to identify opportunities for improvement.

Ironically it seems the UX marketplace is moving away from independent usability testing despite the value that it offers to organisations as well as the integrity it affords the design process itself. Focusing on the objectives of usability testing will help to clarify whether independence is necessary. I think that if understanding usability is the goal, one agency or designer cannot objectively test their own design.

What do you think?

Data just got fashionable: 5 learnings from Andrew Bleeker

Analytics and data have become very popular in the last few years. Two things seem to have contributed to this, one has been Big Data and the other has been the Obama campaign which highlighted data and analytics as a key ingredient to their success. So when I stumbled across Andrew Bleeker's presentation from the recent DNA://13 conference on the ABC's Big Ideas program I was over the moon. His presentation makes for great viewing and can be found on the Big Ideas site. Also, the slide pack is on the DNA://13 conference site.

It was great to see that Andrew Bleeker's presentation style was clever, clear and engaging. All things a good data presentation should be. He is a fantastic advertisement for the practical application of data to make better decisions.

My take on Andrew's main points:

1. Focus efforts on your key objectives and avoid getting side tracked. I thought this was a simple, yet critical message.
2. They used data to improve the efficiency of their campaign, that is, to increase the likelihood  that the dollars spent on communicating to voters would result in a vote and or donation. They avoided demographics and profiled the market in terms of likelihood to vote and likelihood to vote Obama. This allowed the campaign to focus on the voters who would influence the outcome, avoiding those who were  going to vote Obama already and those who definitely would not. Ultimately this is why we use data, to make better decisions, be it for optimising a landing page or fine tuning the message in an email. 
3. Distrust of authority is a major challenge for all brands when trying to convey a message. He noted that their research had consistently reiterated this fact and that it represented a barrier to communicating a message. He talked about social media as a mechanism for overcoming this by using a trusted source (i.e. a friend or  connection) to carry the message for the campaign. I think this mentality of distrust can be a real barrier for some organisations. They often need to accept that people don't care about their brand and once they can do this, they can begin to understand how to communicate with their customers.  
4. Where possible lead with the facts. This was an effective way to communicate a message.  Apparently dogs and children were also effective at encouraging people to share content however not necessarily in achieving their objectives. Andrew made a clear distinction between general engagement (re-tweets, shares) and the end objective (donations and votes).
5. Test. The discussion around testing was incredible. During the campaign they had 4000 iterations of one donations page. The final version converted 4 times better than the original. Andrew suggested that without the volume of donations they received online they would have had dramatically less funding than the Republicans.

To wrap up, there is much to learn from this Andrew Bleeker presentation but there is one phrase  that stood out:

 "less spin on the ball"

I think this relates to the need to treat your customer or audience with respect.  Gone are the days of selling people things they don't need and here is a landscape in which relationships are more important than ever. If you get a minute watch the presentation or view the slides. I would love to know what you think. Is my man crush justified? 

The first post

This is a blog about improving user experiences.    

          

Why the focus on user experience?

I firmly believe that customer centric thinking is the best way to achieve business success. In general, I view business in an uncomplicated manner: company X wants to sell their products or services, and there are potential customers who need or want those products. By understanding the needs of both parties we can design solutions where customers buy from company X.

If fruit was 'designed', surely the result would be a banana.

Why call the blog Pragmatic?

I think that a pragmatic approach is key to helping any organisation improve their user experience. That is, a pragmatic approach in everything, from choosing a technology solution, to finding the balance between business requirements and user needs, right through to negotiating the needs of competing stakeholders within a company.

With this blog I am hoping to be able to share some of my knowledge and engage with other professionals who are dealing with similar, and different, challenges in the digital space.

About me

I am a digital professional based on Melbourne, Australia and work with leading companies to assist them to get more from their interfaces and products. Much of my work has been research and design based with a focus on websites, apps, and other interfaces. When helping these companies I draw on my experience of having observed well over one thousand people using interfaces, countless hours trawling analytics and other quant data sources, plus numerous interviews and focus groups, as well as an understanding of best practice. While most companies and products are unique at the end of the day it is about people.

My specific areas of interest include:

•  Measurement: As the quote attributed to Peter Drucker goes “if you can’t measure it, you can’t manage it”. Without measurement we have no understanding of our progress towards our goals. This applies across most aspects of life and business, be it in brand or campaign tracking, or monitoring your personal fitness. Measurement delivers the power to make better decisions. And in the digital space we can measure so much!

• Solving the real problems: Too often we get caught up in the detail and don't view problems strategically enough. I think Seth Goddin’s classic post Bear Shaving, illustrates this brilliantly.

• Goals and strategy: From clearly articulated goals comes clear vision and purpose. When groups of people have a common understanding of the end goal and how they are going to get there, success is much more likely.

My pet peeves:

• Users instead of I: A positive trend within the digital industry is the acknowledgement that decisions should be made based on the needs of end user. On the flip side, I now hear more and more people presenting their opinion from the user perspective such as “users want to be able to…”.

• Bad research and people bagging research: Research seems to be a misunderstood discipline and like any, it requires a specialist skill set to do well, plus an understanding of when to apply the right methodology. So when I hear comments like “focus groups are a waste of time” it makes me mad. Sure, there is plenty of examples of bad research out there but that does not make research itself bad, just in the same way that one bad car does not make all cars bad.

• Ignorance and lack of respect for other professionals. Too often in the UX space people hold attitudes and preconceived views about other professions which are ill conceived and just plain ignorant. Some common views I have heard include, that developers ruin great design, that business analysts have no vision or imagination and that advertising agencies don’t get digital. News flash – just like the UX space there are good and bad people in all of these professions. As Abraham Lincoln said, “When you look for the bad in mankind, expecting to find it, you surely will.”

Let the blogging begin.