Multivariate testing: save time and refine your analyis

Multivariate testing: save time and refine your analyis

What is an MVT?

An A/B test is an experiment run on a website enabling an original page to be compared with another page (called the variation) to which a change has been made.

A multivariate test, also known as an MVT, is a form of A/B testing in which each variation includes several changes at the same time.

The idea is to simultaneously test all the possible combinations of several elements on the same page, so as to determine which combination has the greatest impact on your conversion rates.

Here’s a simple example:
Imagine you want to change both the background color and wording of a CTA.
For the color, you want to test yellow, blue and red.
For the wording, you want to test two phrasings: “Start” and “Buy”.

Kameleoon generates six versions of your page presenting the six possible combinations:

A mini glossary of multivariate test terms

Multivariate test: a multivariate test enables the conversion of one or several combinations of several elements to be tested.
MVT : MVT is the acronym for Multivariate Test.

Section : a section groups together several variations of an element. For example, the “color” section, or the “wording” section.

Variation : a variation is a proposal to change an element. It varies compared to the original version. In an MVT, we can create several variations that we group together in sections. For example, the variations of the “color” section would be “blue”, “red” and “yellow”. The variations of the “wording” section would be “Buy” and “Start”.

Combination : the combination designates several variations of different sections that we combine and display at the same time to the exposed visitors. A combination would be “Buy” in blue or “Start” in red, for example.

What is the value of MVTs for your optimization strategy?

To clearly identify the reasons that influence your conversion

The first advantage of an MVT compared to a classic A/B test is that it allows a greater understanding of the reasons that explain the differences in performance between different variations.

By systematically breaking down changes into combinations, we can isolate the elements of the tested page and measure the influence of each of them.

The multivariate test lets you understand how the elements of a page interact with each other and which ones are instrumental in improving performance.

To save time

Automatic implementation of the combinations will save you precious time. This is especially true with a large number of different factors.

If we add a change in the shape of the CTA (square or rectangular) to our previous example (3 colors, 2 wordings), we get to 2 x 2 x 3 = 12 different variations to test in order to identify the variation that best converts.

The automation of combination creation becomes a must.

In what situations should multivariate testing be used?

When you don’t have a clear-cut opinion on the hypotheses to be initially tested

The first stage of designing your A/B test is to imagine the different changes you want to test.

In certain situations, it’s hard to have an initial opinion on the element that will make the difference. Your CTA is not performing as expected — is the problem the wording, the color, the shape? It’s not easy to have a strong hypothesis right from the start.

In this case, an MVT is essential. You can test several elements at the same time, and let Kameleoon generate all the possible combinations for you. In this way, you won’t rule out any possibility from the outset, and will save considerable time thanks to automation.

When you’re working from an analytical perspective

As we said, a multivariate test can refine the analysis of results, by isolating the different components of the changes being tested.

If you’re working from an analytical perspective in which the interpretation and understanding of your A/B test results are key, then multivariate testing will be a good ally.

The question of minimum traffic

The question of traffic on the page being tested is an important one and should be considered before you start any MVTs.

Indeed, the higher the number of combinations, the more traffic you need.
Generally speaking, it is considered that we obtain an acceptable reliability rate in an A/B test:
– From 1,000 visits/variations if the page being tested has a high conversion rate (such as engagement on a landing page or click-through rate for a CTA button) and where the expected potential gain is high (for example, +10% or 20%) ;
– From 7,500 visits/variations if the conversion rate is low, which we find in filling in forms, for example (generally <5%).

So try to imagine how it is at the end of the conversion funnel, where conversion rates are generally weak and traffic is also low. Reaching 7,500 visits/variations becomes complicated.
And the more we multiply the number of variations (which happens quickly, with MVT), the more we need the traffic to be high to reach the critical threshold of number of conversions by variation.

In a nutshell, as the nature of MVTs is combinatorial, it is highly recommended that you select a finite and relatively low number of variables to change. There’s no point in using multiple variations if it’s to read results that are meaningless because they were not validated by a sufficiently large sample of visitors.

How does multivariate testing work?

On Kameleoon, you access it via the back office or the graphic editor.

All users with access to the A/B testing license automatically have access to the MVT tool.

1. Create your variations in the graphic editor

As with a classic A/B test, the first stage is to create the variations you want to test in the Kameleoon graphic editor.

Let’s go back to our example in which you want to change both the background color and wording of a CTA.

It’s very simple:
In the menu, you create the sections of your test; here these are “Color” and “Wording”.
Then, for each section, you create the variations you want. For example, in the “Color” section, you’d create a “yellow” variation, a “blue variation and a “red” variation. In the “Wording” section, a “Start” variation and a “Buy” variation.

Kameleoon will then automatically generate all the possible combinations of your different section variations.
So, with this example, we’d have:
“Start” in blue
“Start” in yellow
“Start” in red
“Buy” in blue
“Buy” in yellow
“Buy” in red

2. Allocate the traffic

Now you have to decide how you want to allocate the traffic on your website to the different combinations for the duration of the test.

One of the strengths of Kameleoon multivariate testing is that it proposes two approaches for allocating your traffic: by section or by combination.

By section

Section: a section groups together several variations of an element.

In our example, we have the Color and Wording sections.

The Color variations are “yellow”, “blue” and “red”.
The Wording variations are “Buy” and “Start”.

By default, the traffic will be distributed equally among all the variations.
Allocation by section is the “simplest”, since you have fewer sections than combinations.
Here, you will have to distribute 100% of your traffic between Yellow, Blue and Red, and 100% of your traffic between Buy and Start.

Kameleoon will then automatically generate the allocation to each of the combinations.

By combination

Combination: a combination designates several variations of different sections that we combine and display at the same time to the exposed visitors.

In our example, 6 combinations are possible:
“Start” in blue
“Start” in yellow
“Start” in red
“Buy” in blue
“Buy” in yellow
“Buy” in red

You can therefore decide manually how to allocate your traffic to each of the combinations, or, here too, decide to allocate them equally to all the variations.

This manner of allocating your traffic can become difficult to manage if you have many different variations since the number of combinations increases exponentially.

However, it is useful in certain specific situations, particularly when you want to exclude certain combinations from your test.

Here’s an example: you change the background color and the text color in a CTA. You don’t want your visitors to see the combination of black background/black text (because they would see… nothing). So, traffic allocation by combination will let you manually set the traffic at 0 on the combination of black background/black text.

3. Set up the KPIs to monitor

Once you have set the traffic allocation, you can set the goals to be measured during your test.

Here too, the approach is the same as for a classic test. You set up the KPIs to be monitored.

You will find all available information on setting up your reporting tools and your goals in our article on this subject

4. Simulate the test and then launch it

You can then view all the combinations you are testing in simulation mode, directly accessible at this stage of your project. This enables you to validate your choices before putting the test online.

The simulation mode, which sets the Kameleoon platform apart from others, is used in the same way as for a classic A/B test.

 

5. Monitor the results

On your results page, each combination is considered as a variation (in the sense of a classic A/B test).

You can therefore analyze, in detail, the level of impact each element has on your main KPIs, and ensure that you are taking the best informed decisions to change your page.

Conclusion

In conclusion, it is very valuable to use MVTs if you have a sufficient level of traffic. Be aware, though, that it takes a relatively long amount of time to obtain reliable results.

So while the use of MVTs should not be systematic, it can prove very useful in certain situations. And when these situations arise, it’s in your best interests to have a robust solution, like that of Kameleoon!

demo request kameleoon

Catherine Fournier

Catherine est Content & Events manager chez Kameleoon. Sa raison d'être : analyser tous les retours d'expérience de nos clients et consultants pour pouvoir vous partager les meilleures pratiques en optimisation de l'expérience utilisateur !