You’ve figured out that A/B testing is an effective (and low cost) tool for increasing the conversion rate of your emails, landing pages, paid ads, CTAs, or anything else related to your digital marketing efforts. Now that you’ve mastered the basics, it’s time to take you’re testing to the next level. (If you’re not familiar with A/B Testing, be sure to read our Beginner’s Guide to A/B Testing.)
You’ve probably been told that with A/B testing it’s important to focus your test by creating a clear hypothesis and testing only one variable at a time – and that’s true. But once you’ve mastered the basics, you can begin to play with concept testing and explore the all-important test for statistical significance.
Going beyond one variable testing
Concept testing is another technique for gaining important information through testing by focusing on an overall concept versus just one variable. For example, VONT client Suffolk University ran a test in which ad #1 focused on touting part-time and evening classes while ad #2 focused on receiving an MBA in 18 months. The variables of headlines, copy and CTA were different, yet that was not a concern because VONT was interested in learning which focus/concept people responded to more – part-time and evening classes, or a quicker path to an MBA. Another way to look at concept testing is that it still is a one hypothesis=one variable test – it’s just that the overall concept is the one variable.
No matter what type of testing you’re doing, it’s important to attain meaningful results. And that’s where the Chi Square Test of Statistical Significance comes into play.
The importance of chi square
You’ve run a test and received your data showing that one component is outperforming another. But how do you know the results are valid and don’t fall within the realm of chance? You turn to the Chi Squared Test of Statistical Significance, also known as Pearson’s Chi Squared Test. The test shows how likely it is that, given n sets of data, any differences between those sets are simply the result of chance. Put another way, the chi squared test shows how likely it is that examined variables are independent of each other. VONT Digital Strategist Will Plumley explains the importance of the test and what it provides: “With chi square analysis, VONT and each client make informed decisions on all of our test results rather than merely guessing that version “a” will continue to outperform version “b” in the future.”
The outcome of such a test is measured in a percentage of confidence – meaning that you can be X% sure that the results of your test are not due to chance, and that winner version Z will continue providing better results than loser version Y. Considering the cost/benefits trade-off of a longer test run, VONT usually waits until there is a 75% confidence level; to wait longer is often an exercise in diminishing returns.
At this point you may be wondering how – short of phoning your high school math teacher – you’ll run the chi squared test? Not to worry, as with so much in the digital world, there’s an app for that. After using several tools, VONT Digital Advertising Manager Sharon Roy has settled on SEO Book’s PPC G-Test Calculator: “It’s simple, yet accurate. It’s versatile, and allows the user to vary what is used as trials or successes. In addition, it provides results not just for A/B tests, but for tests with multiple variables – all you need to do is indicate the number of versions.” Here’s a link so you can check out the SEO Book calculator.
Putting it all together
With testing, it’s the overall process that is most important. The strength of the testing is in the repetition, in the continual search for higher performance. Plumley explains the process: “We test, analyze for statistical difference, call the winner with a high level of confidence, declare the winner the new “control,” and develop a new version to test against the control.” Doing one test won’t get you too far – but creating a process of testing over and again will enable you to continually improve your marketing campaign results.
About VONT Performance Digital Marketing
At VONT we believe that change is the only constant in the digital world – and that excites us. When tools and environments are constantly changing, new opportunities to help our clients achieve success are constantly arising. Each new advertising technology, social platform, or design approach allows us to improve on the results we achieve for our clients.
We believe in this idea of continual fine-tuning so much that we named our company VONT, which means to achieve exponential improvement in incremental steps. It is our core belief, and the reason why we are not simply a web design company or simply a digital advertising agency, but rather a long-term, single source partner providing a comprehensive array of web development and digital marketing capabilities.
In short, we’re here so that our clients achieve success in the ever-changing digital world. If you’d like to learn more about VONT and the work we’ve done with our client partners, visit our Work page. Or, if you have a question, contact us. We’ll get right back to you!
