Ask: The Difference Between a Multivariate Test and an A/B Test? Which should you use?

Ask: The Difference Between a Multivariate Test and an A/B Test? Which should you use? 

Both tests are trying to test which component you should change on your website to improve click thru rate and your funnel to your ultimate goal. 

Site visitors are bucketed into one version or the other. By tracking the way visitors interact with the page they are shown — the videos they watch, the buttons they click, or whether or not they sign up for a newsletter — you can determine which version of the page is most effective.

AB testing 

changes only “1” component at a time, such as font size, headline keyword, color of call to action button, etc.  (but not all at the same time which most makes this mistake) 

Multivariant testing 

compares a higher number of variables, and reveals more information about how these variables interact with one another. As in an A/B test, traffic to a page is split between 2 versions of the design. The purpose of a multivariate test, then, is to measure the effectiveness each design combination has on the ultimate goal. So there are more combinations in your testing but will reveal how the
components perform with each other.  See picture below with 1 image/2version and 1 headline/2 version testing

Should you use AB testing and Multivariant testing? 

Depends on your volume of website traffic:

Multivariant testing requires a large amount of traffic to complete. Since all experiments are fully factorial, too many changing elements at once can quickly add up to a very large number of possible combinations that must be tested. Even a site with fairly high traffic might have trouble completing a test with more than 25 combinations in a feasible amount of time.  But the biggest advantage is that it will test "combinations" of components which AB test does not do. 

A/B testing is a powerful and widely used testing method. Keeping the number of tracked variables small means these tests can deliver reliable data very quickly, as they do not require a large amount of traffic to run. This is especially helpful if your site has a small number of daily visitors. Splitting traffic into more than three or four segments would make it hard to finish a test. In fact, A/B testing is so speedy and easy to interpret that some large sites use it as their primary testing method, running cycles of tests one after another rather than more complex multivariate tests.