Multivariate testing (MVT) is a particular form of split testing by which several versions of several components of a webpage or website are dynamically combined to give multiple versions of the page or website. The goal is to measure how the different changes can affect conversion rate for a given objective (e.g., order rate, webform completion rate, subscription rate, etc.) and to give the winner page version. MVT may be also used for email campaign testing. It is a key tool of conversion rate optimization (CRO).
During the experiment each visitor is randomly exposed to one version of the page.
Below seven variables or elements are tested on a trial page and the F variable, the call to action associated with the button, has six differents values or versions (the original or control version and five challenger versions).
Only a small part of trial page versions created for the test:
Images source Monetate
Confronted to A/B testing, multivariate testing allow to test simultaneously several elements (e.g., image, button color, call to action, etc.) and the test may be realized on a succession of pages (e.g.,checkout process).
MVT also measures interaction between different tested elements and influence of each variable.
For instance, imagine a landing page test with two versions (A-B) of call to action and two versions (1-2) of an image. Two separate A/B testing may give call to action A and image 1 as tests winners, but a multivariate test may reveal that the better combination is in fact a page with B and 2.
Below, 3 variables are tested, headline, background color and action button. The experiment dashboard shows influence of each variable on results.
Image source Webster
High volumes of visits are often needed for multivariate testing results to be significant and reliable. 4 variables and 4 values for each result in 64 page versions to be tested. If the average transformation rate on the page is 5 %, 128 000 pages are needed to have on average 100 observed transformations per page version.
Some MVT produce several tens of thousands of distinct pages and require several millions of visits to give significant results.
In the example below, almost six months are needed to run a test with 3 variables and 3 variations for each.
Fortunately, MVT tools automatically calculate confidence interval for a given confidence level. They also quickly remove worse combination to prevent "consuming" visits. Experiment and test designs are based on Taguchi method, fractional factorial or others.
Below, a report shows best challenger versions against control page with confidence index.
Image source Maxymiser.
For more details on experiment procedure see multivariate testing tools.
MVT are mainly used for testing:
check out process