Limitations with A/B split testing
I use AB testing (split testing) on my tea website a lot. It's a great way of checking whether design iterations or new functionality is actually useful for the users (instead of just my preference).
But if you're using, or plan to use AB testing, then be aware of its limitations. This article is about just one particular (but very practical) limitation I've encountered.
AB tests are rarely conclusive
We're all attracted by the magic of AB tests - "we changed the colour of a button and sales increased 60%!!!" - but the reality is slightly less glamorous.
Of the last 6 AB tests I've conducted on my site, 5 of them finished inconclusively. Those tests ended with both variants neck-and-neck, or separated by one or two percent. Even when the 2% increase was statistically significant, it's not wise to trust an AB test so implicitly (ie. are you going to base your major design decisions on a 1 or 2% variation from a single AB test?)
Lots of people like to cite Amazon when talking about user-testing, particularly a 1% increase in sales they achieved through a 100ms speed increase on the site. What people tend to forget is that Amazon is at the peak of optimisation - they've already user-tested and AB-tested the ass off their website, so a 1% increase is like squeezing that last drop of water out of the sponge. Us mere mortals should aim for big results, big improvements before we start worrying about the 1% tweaks out there.
AB tests are short-range
The most reliable way to run an AB test is with a page and a goal that are ideally 1 or 2 clicks away from each other. Eg. your product listing as your AB template, and your product page as a goal. The problem here is that it ignores your crucial metric for ecommerce - sales!
So, many people recommend AB tests should always have 'sale' as the goal - ie. measure all AB testing against whether it increases sales or not. However, that's a pretty flawed approach for many practical reasons:
- It means you can only AB test 1 thing at a time - eg. if you're testing homepage design changes, with 'sale' as the goal, then every other page on the site must stay the same (ie. cannot be subject to another AB test). Otherwise, your homepage results might be skewed depending on other AB test results across the site.
- You need to freeze your whole site - as above, if you're AB testing the homepage against 'sale' as a goal, then you can't mess around with your content in between, because it might screw with the results.
- You need large amounts of traffic - imagine your sales conversion rate is 1%. To be able to produce statistically significant increases in sales, you'll need to channel huge amounts of traffic through your sales funnel. Unless you have a high traffic site, that will take time.
Customers don't behave like robots
For an ecommerce website, this is the major limitation of an AB test - AB testing is 2 dimensional, measuring simple 'monkey-see-monkey-do' type behaviours. Your sales and marketing is 3D, incorporating many factors and variables which might not be related to your web design
For example, encouraging genuine engagement and stickiness might be more beneficial in the long run than just channeling more clicks on a button. Having a user remember your name, and having a good impression of your business might be more worthwhile than them clicking on buttons - it could be that some people are not ever going to be customers of yours, but they might refer you to other people.
AB testing is good, but don't ignore other tools
Go crazy AB testing things on your site, but don't base all your decision making on it. Make use of your friends and family to give you real feedback about what they're thinking and don't just measure what they do. And use your own experience too - remember that sometimes, breaking new ground can set you apart from your competitors.
As Don Draper says in an episode of MadMen: "Maybe they didn't mention that idea because they don't know about it yet. Give me a year to work on them and then test to see if they mention it."