International usability research: when is it worth the investment?

Over the last few years, we have taken part in several international usability studies. In these studies, our colleagues across borders and we all carried out localised versions of the same usability study. Looking back, we consider most of these large-scale usability projects to have been a waste of time and money. Seriously. The investment could have been put to far better use. Let us explain why and how.

Oh dear, the hassle!

As a major global company, you want to play it safe. You want to involve all your customers from around the world, gathering feedback to develop the best possible website. It may therefore seem sensible to organise an international qualitative usability study. Let me tell you what you’re getting yourself into when you invite six to eight participants from six different countries to evaluate your new concept.

You write (or translate) six interview guidelines. You instruct six recruiters to invite six to eight participants, all with essentially the same characteristics. After that, six teams of two researchers each conduct the same interviews. To top it all off, one overarching team has to coordinate the entire project. And here’s the fun bit: you’ll end up with a huge amount of identical results. Why? Jakob Nielsen’s ​‘rule’ still applies. The number of new insights decreases with every additional participant.

It is more worthwhile to carry out several iterative usability studies, because between studies you have the opportunity to improve and optimise your designs. You can use each iteration to verify whether the changes are indeed improvements. In other words, you can make some real progress. You could even organise each round of testing in a different country. Your own take on ​‘Around the World in Eighty Days’.

What do you really want to know?

“But aren’t there any cultural differences to take into account?” you might ask. There might be, but remember, you requested a usability study. A usability study shows us click patterns and we follow your users on their journey through your website. We assess whether the designs and content are supportive enough. But users’ perceptions and interactions won’t differ that much between nationalities, because usability is pretty universal.

This isn’t what you want to know, is it? Your real (research) question is probably something like this: ​“But aren’t there any cultural differences when it comes to user needs?” There probably are. Users’ motivations or needs might differ between nationalities. Or how they perceive your brand. Your desire to understand your users is admirable, but do not use a usability study for this purpose. That’s a big no-no!

Instead, you should use methods specifically designed to map user needs, usage contexts, customer behaviour or markets. These methods can help you determine who your users are and what drives them. This can be done both qualitatively and quantitatively. Incidentally, you should carry out these studies before you start designing and building. These results will determine what your website should look like and how it should be designed. After that, you can conduct your (iterative) usability studies.

When should you carry out an international usability study?

Localised testing is worthwhile when you discover there are contextual differences. When specific elements of the website need to function very differently in certain countries. For example, checkout flows.

In the Netherlands, we mainly use iDeal as a payment method, whilst other countries prefer credit cards or Facebook Pay. Once payment has been made, we rely on PostNL to deliver our orders rather than FedEx or UPS. In other words, the checkout flow is worth exploring and researching, as each test will reveal different results. This will provide you with specific insights that can help you optimise the flow for these individual markets.

Can you see the difference? An international (usability) study is valuable when there are contextual differences, or when you want to understand your users better. But stop overusing usability studies – it’s counterproductive. You should first clarify your research questions and then find the appropriate method. And one way or another, an international study is costly. Use it wisely.