A/B testing: How our new technology ends the Flicker effect

A/B testing: How our new technology ends the Flicker effect

When we announced a major innovation a couple of weeks ago, the small world of A/B testing was all shaken up. It’s true that we tackled a problem with big consequences: the Flicker effect.

We’re glad that our innovation created such a buzz. Our customers gave us very positive feedback on our new technology which was automatically implemented on their sites and showed immediate results.

Other industry players also took the time to analyze the subject and share their comments. But we realized that there was a major incomprehension on the scope of our new technology.

We aim at giving new impulses and making the debate more constructive. That’s why we decided to unveil some details on the way our new, 100% Flicker-free technology works.

100% Flicker-free: true or false?

Quick recap: The Flicker effect is when on an A/B tested page, users see the reference version before the variation builds up.

Let me insist: Kameleoon’s new 100% Flicker-free technology really is an innovative solution to the Flicker effect on your sites!

There’s one thing all experts agree upon: The Flicker effect is induced by A/B testing solutions. We wanted to go beyond best practices that reduce the risk, and decided to attack the problem at its root and solve it once and for all. Some will tell you that there is no miracle solution. But we believe that one should always try to go beyond the status quo and find innovative solutions for widely spread problems – such as the Flicker effect.

“If I had asked people what they wanted, they would have said faster horses.” – Henry Ford, Founder of the Ford Motor Company

Where does the Flicker effect come from?

There’re really two reasons for the Flicker effect. This information is essential to understand how our new 100% Flicker-free technology works.

A. Users: what you can do

First, the Flicker effect can be caused by the loading of the script itself and the moment when it becomes available and is ready to execute the necessary changes on the pages of your site.

Best practice offered by A/B testing tools primarily treats this issue (but beware, not all advice is good advice, as we’ll see later on).

One thing’s for sure: The earlier your A/B testing script is available, the better.

Different factors impact this first reason for the Flicker effect (the first two in the list below are the most important ones, as they’re linked to the way the script is integrated into your site’s code):

  • Script hard-coded into the header of your site vs integration via a tag manager
  • Position of the script, in the page and with respect to other tags (top/bottom of the page…)
  • Weight of the script and performances of the hosting server (the lighter the script and the stronger the server, the faster the script is loaded by the visitor’s browser)
  • The browser’s caching capacities

Synchronous or asynchronous script: which one to choose?

You often hear about synchronous and asynchronous script and their influence on page loading and Flicker effect.

Synchronous means that the navigator blocks the execution and the display of the rest of the page as long as the script hasn’t loaded and isn’t available. This is because the browser considers that the script’s instructions are primary. The script must be executed before any other action is taken, which means that nothing else is executed while the script is still loading.

The consequence is simple: for as long as the script is loading, a white page will appear on your site. And if the server hosting the script doesn’t respond, the page remains white permanently: the site is no longer available.

Nobody wants that to happen to their site. However, the use of synchronous script is recommended by certain optimization tools. And unfortunately, some of their customers have seen it happen on their sites. Oops…

Asynchronous means that the script is loaded at the same time as other resources. So there’s no blocking. But neither is there a guarantee that the script is available before the other elements of your page. That means there is a potential risk of a Flicker effect, as elements supposed to be changed for an A/B test may appear before the script has loaded.

Four years ago, we developed a method called “asynchronous blocking integration”, to offer the best possible compromise between synchronous and asynchronous script.

The script is loaded asynchronously, but the page remains white as long as it is not available, or before a certain time (a second, for instance) hasn’t elapsed.

This method offers the advantages of the synchronous script (white page), but with additional security: if ever Kameleoon’s server (CDN) doesn’t answer, the page loads nevertheless once the second has elapsed.

Result: a performance similar to synchronous script as to the Flicker effect, but offering additional security.

We have always delivered relevant best practices to our customers and continue to innovate in this field as well.

B. A/B-Testing solutions: what they should provide

The second reason for the flicker effect is induced by the way the A/B testing solution interacts with the web page to implement changes.

One could think that once the page has loaded, the risk of a Flicker effect disappears. But it doesn’t. The factors that reduce the risk are the quality of the code of the software and the technologies it uses. The A/B testing solution should use as little CPU and RAM as possible while offering the fastest possible interaction with the necessary elements of the page.

Our new 100% Flicker-free technology concentrates on the second reason only.

A new 100% Flicker-free technology, available without additional installation and compatible with 92% of all browsers.

Our technological innovation impacts the way Kameleoon interacts with the elements on the page once the script is available.

The difficulties linked to the refreshment cycle of the browser have been taken into account. Changes are directly applied to the elements as soon as they appear on the page, without need for a white page, and without the risk of your website crashing.

For the geeks out there: our technology is based on the DOM Mutation Observers standards, as some of you have correctly guessed (well done, guys!).

It’s an industry standard implemented in all (mobile and desktop) navigators supported by Kameleoon, apart from IE 9 and 10 (which raise other issues as well, but here’s not the place to discuss them).

That means that the new Flicker-free technology is compatible with 92% of all existing navigators.

More good news: No change of code is needed to benefit from this new technology, it’s available automatically. If the browser isn’t compatible, the Flicker-free technology simply won’t be installed, and your tests continue to run normally. You just run the risk of the Flicker effect, but no more than with any other A/B testing tool.

In short, Kameleoon’s new Flicker-free technology is a great improvement, without negative side effects.

So what about best practices?

The performances of the new technology we developed put an end to the Flicker effect. But some rules must also be respected. Most of them are well known, some need to be revised.

In order to limit the Flicker effect you should follow some advice:

  • Install the script of your A/B testing solution as high up as possible in the header of your site.
  • Install the script using the asynchronous blocking integration as explained above. If your solution doesn’t offer this integration, we still recommend not using synchronous script. It gives better results for preventing the Flicker effect than an asynchronous script would (not as good as an asynchronous blocking integration though), but it creates the (although small) risk of crashing your entire site!
  • Don’t use tag managers, in order to keep control of the order in which scripts load.
  • Reduce the size of the script. Kameleoon developed pre-processing a couple of years ago that accelerates the script by loading only what is strictly necessary. If, for example, your A/B test does not need advanced targeting, there is no need to load the corresponding criteria.
  • Optimize the loading time of your site and all its elements.
  • Use a CDN to accelerate script loading.
  • Most of all don’t use redirection for your tests. Unfortunately, A/B test redirection can only be done via JavaScript. Unlike a “real” server redirect, (HTTP 302 for instance), which is very light, JaveScript redirect is heavy and will slow down page loading significantly. In order to execute a JavaScript redirect with an A/B testing solution, the first page and all its resources (images, scripts, etc.) have to be loaded prior to the loading of the second page. No Flicker effect for sure, but an initial loading that is extremely long. You should actually limit redirects to situations where there’s no alternative. A good example would be a test for a complete overhaul of your website, with the site hosted on a different server. In this case, the redirect is almost mandatory. In all other cases, please avoid using JavaScript redirects!

Some more best practices – which you won’t need if you use Kameleoon’s new 100% Flicker-free technology:

  • Use style sheet whenever possible. It hardly makes any difference if you use Kameleoon, you are free to choose the implementations that suit you best, without worrying about performance which is taken into account automatically.
  • Don’t use jQuery. Kameleoon doesn’t use external libraries, thus reducing the size of the script even further. New sites (many among you are overhauling your websites at the moment) are based on new frameworks such as Angular or React, which most of the time makes jQuery an unnecessary burden. Moreover, you should be careful: If your site already uses jQuery, it may not be sufficient to withdraw it from you’re A/B tests. If the jQuery script loads later, you can still have a Flicker effect. Such unnecessary worries!
  • Optimize the code of your changes in order not to add unnecessary JavaScript instructions. Kameleoon’s graphic editor doesn’t generate JavaScript instructions and uses an engine to apply modifications made by the user. So this is not an issue.

Another best practice, which we ourselves recommended in the past but is no longer necessary, is the “setInterval”, used to change the targeted elements as quickly as possible. This is the core of the problem we solved. The idea is to test different values to find the ideal parameter that makes the Flicker effect disappear without deteriorating the CPU time your A/B testing solution uses up. That’s exactly the challenge we took on. Our solution eliminates the “trial and error” method associated with setInterval and saves A/B test developers precious time. SetInterval, a polling solution, becomes redundant: modifications are applied to a given elements as soon as it is injected in the page’s DOM.

Conclusion

Best practices offered by editors should be followed. Without these rules, even the most innovative technology can’t hold its promises.

Our aim is to deliver continuous innovation to our customers. Thanks to our technology, they went from an already relatively small risk of Flicker effect to the total absence of it. And that’s a major improvement! 😉

Grégoire Thomas

Gregoire est Head of Marketing à Kameleoon. Il est responsable du lancement des nouveaux produits et de la croissance de l'écosystème Kameleoon.