Over the last years we have participated in several international usability studies. In these studies our cross-bordering colleagues and we all conducted localised variations of the same usability study. In retrospect, we consider most of the large scale usability projects a waste of time and money. Seriously. The investments could have been used far more wisely. Let us tell you why and how.
Oh dear, the hassle!
As a huge world wide operating company you want to play it safe. You want to involve all your customers from around the globe, gathering feedback to develop the best website possible. Therefore it may seem sensible to organise an international qualitative usability study. Let me tell you what you are getting into when you invite six to eight participants from six different countries to evaluate your new concept.
You write (or translate) six interview guide lines. You instruct six recruiters to invite six to eight participants, with essentially the same characteristics. After that, six teams of two researchers all conduct the same interviews. To top it all off, one overal team has to co-ordinate the entire project. And here is the fun part: you will end up with a enormous amount of the same results. Why? The ‘rule’ of Jakob Nielsen still applies. The number of new insights declines with every additional participant.
It is more worthwhile to conduct several iterative usability studies, because in between studies you have the opportunity to improve and optimise your designs. You can use each iteration to verify whether the changes are indeed improvements. In other words, you can make some real progress. You could even organise each round of testing in a different country. Your own spin of ‘Eighty days around the world’.
What do you really want to know?
“But aren’t there any cultural differences to take into account?” you might ask. There might be, but remember, you requested a usability study. A usability study shows us click patterns and we follow your users on their journey through your website. We conclude whether the designs and content are supportive enough. But the perception and interaction of users will not differ that much between nationalities, because usability is pretty universal.
This is not what you want to know, is it? Your real (research) question is probably something like this: “But aren’t there any cultural differences when it comes to user needs?” There probably are. The motivation or needs of users could differ between nationalities. Or how they perceive your brand. Your desire to understand your users is admirable, but do not use a usability study for this purpose. That is a big no no!
Instead, you use methods tailored to map user needs, usage contexts, customer behaviour or markets. These methods can help you determine who your users are and what drives them. This can be done both qualitatively and quantitatively. By the way, you should do these studies before you start designing and building. These results determine what and how you should design your website. After that you can conduct your (iterative) usability studies.
When should you conduct an international usability study?
Localised testing is worthwhile when you discovered there are contextual differences. When specific elements of the website should work profoundly different in certain countries. For example, check-out flows.
In the Netherlands we primarily use iDeal as a payment method, while other countries prefer creditcards or Facebook payment. After paying we depend on PostNL for delivering our orders instead of FedEx or UPS. In other words, the check-out flow is worth exploring and researching, because each test will uncover different results. It will give you specific insights which can help you to optimise the flow for these individual markets.
Do you see the difference? An international (usability) study is valuable when there are contextual differences. Or when you want to understand your users better. But stop abusing usability studies. It hurts. You should first unravel your research questions and then find the fitting method. And one way or another, an international study is costly. Use it wisely.