Want to test a new proposition? Use Experiment Design

You’re working on a new product, or a new twist on an existing one. Your value proposition is being turned on its head. You want to do something innovative. But before you start building, you want to know: will customers actually be able to make sense of it? Do they understand it? Do they even want it in the first place?

Experiment Design helps you find answers to these kinds of questions. You test how users react before you set your product in stone.

But what exactly is Experiment Design? What does it look like? How do you go about it?

Let us give you a quick overview.

With Experiment Design, you can test real-world behaviour

Suppose you ask people: ​‘Would you sign up for product X?’ They’ll usually say yes straight away. But if you then develop your proposition and present it to people online, there’s a good chance they won’t sign up at all. Experiment Design tackles this. You don’t rely on what people say, but on what they do. 

You develop your ideas into a design that feels very real – without needing any development resources. You then launch it. You promote your experiment through channels such as adverts. An experiment like this feels authentic to users. And that’s how you find out: do they actually sign up for product X? How do they get through the onboarding process?

During an experiment, you never test just one version; you always test different variations. You learn where your target audience drops out, what they struggle with, and what works well – even before you’ve invested a lot of time and money in developing your new proposition.

An experiment provides focus

Experimentation is all about getting to know the real behaviour of your target audience. But it also has a valuable side benefit: it forces you to be very specific. All the ideas that you and your team have must be channelled into a product. You’re forced to ask yourself: if we had to launch something right now, what would be the bare minimum? If we didn’t have another year to discuss this together, what would absolutely have to be included?

An experiment creates focus. It sharpens your focus as a team. You have to think about what really matters. You have to start cutting things out and narrowing down your proposition.

Could you explain how an experiment like that works?

Broadly speaking, you go through five steps:

#1 Define your assumptions
You always start Experiment Design by setting out the key assumptions. What do you want to learn? What questions do you have? The assumptions you want to test determine what your design will look like.

#2 Create the design
You design your new proposition. Always use tools that require as little coding as possible. This ensures you can work quickly. You link what you build to tools that allow you to measure performance in the background. 

You’ll often create a single design, within which you’ll develop different propositions. You’ll make small variations. This way, you’ll learn which proposition – or which elements of the different propositions – work best. 

The design doesn’t have to be perfect. The point is to be able to test your assumptions quickly. And that’s often possible with a very simple design. At this stage, the aim is for the design to validate your assumptions – so it doesn’t need to look exactly like the final product. 

#3 Promote your experiment
Your design is ready, everything is set up. Now the question is: how do you ensure your target audience sees it? To draw attention to your design, you need to look for places where your target audience is. You can do this through adverts, or even by taking to the streets. 

For PostNL and bol, for example, we went door-to-door wearing PostNL T‑shirts to collect bol parcels. That way, you reach exactly the right target audience, in the right context. And that’s what it’s all about. Your experiment should take place as much as possible within your target audience’s natural habitat. 

#4 Run the experiment
Once your experiment is up and running, you’ll find out whether people understand the concept, whether they click through the pages, and whether they make the effort to explore your proposition. You’ll see how people actually behave, and that’s different from what happens in the lab. In the lab, for example, you ask participants to go through an onboarding process. They neatly follow all the steps. During an experiment, people simply stop if they get stuck or if they don’t find it interesting enough. That gives a very honest picture. At the end of the experiment, you’ll know what works well and what doesn’t catch on at all. 

#5 Time to tidy up 
After the experiment, you take everything offline. You tidy everything up properly. You gather the key insights. And then you ask yourself: what’s the logical next step? Often, you’ll want to run follow-up experiments to test new variations or validate other assumptions. You’re never really done experimenting. But at some point, you’ll know: this works; people are signing up; they’re trying to buy the product. That’s when you can start building.

An experiment can be exciting

It’s important to involve your stakeholders in your experiment and the thinking behind it. You’re going to test something, design something quickly to learn from it. With Experiment Design, you sometimes have to leave out a few details for the sake of speed. If stakeholders aren’t aware of this, or if they don’t understand the purpose of the experiment, resistance may arise: ​‘This isn’t our style; this isn’t how we want to do things.’ 

Once your stakeholders are on board and have got used to testing propositions in the real world, it really gets things moving. It becomes a habit. Trying something different every week. Until you get the result you want.

Combine experimental design with other forms of research

Experimental design always coexists with other forms of research. An experiment allows you to test actual behaviour, but after an experiment, for example, you won’t know why people do what they do. To find that out, you’ll need to do interviews with your target group or with people who took part in your experiment. 

Either way, it’s important to enrich your experiment with qualitative research. Don’t forget to find out the ​‘why’ behind the behaviour.