skip to Main Content

3 reasons why A/B testing is always the right answer (Guide to Small Business E-Commerce Strategy)

Why you should A/B test, courtesy of mil8 on FlickrDo you A/B test your site’s content, commerce and customer experience? You should, no matter how big your site. Especially in a down economy, the worst decision you can make is to abandon testing. If anything, now is the perfect time to look for ways to improve. And testing is the best way to find the improvements that work best.

But what if you site gets little traffic? Just the other day someone told me, “My site doesn’t get enough traffic to bother with testing. Why should I wait six weeks – or more – to get data when I need business results now?”

Fair question. Here are 3 reasons why testing matters, no matter the size of your traffic:

  1. Time isn’t the issue. Conversion rate is. If your test case performs dramatically better than your original, you’ll probably know it (relatively) quickly. You can feel comfortable a 15% increase in conversion wasn’t just a fluke in as few as 200-250 conversions[*]. Said another way, you only need so many conversions to prove what works. The bigger the increase in conversion, the fewer conversions you’ll need to know it’s working. And the bigger the increase in conversion, the less time it takes to collect the data.
  2. Meaningful changes take less time to show up. If you don’t see a significant lift in a timely period, it may mean the new version isn’t working dramatically better than the original. If that’s the case, the amount of time it takes to collect the data isn’t the problem; the lack of lift is. Put your focus on the areas where you’ll see greatest improvement. Not sure where to start? Check out the tips here, here and here.
  3. Sometimes the cure is worse than the disease. What happens if your changed page actually performs worse than the original? You could lose sales and not know whether it was the new version that hurt you or some external factor that would have impacted sales in either case. If you don’t run the two versions side-by-side, how do you know which works better?

Yes, you’ll still need to run your tests for a bit of time. Anything less than a couple of weeks worth of data could be influenced by many factors, including day of week, time of day, holidays, what have you. But don’t focus on that. Focus on what you’ll learn and the results you’ll see.

Simply put, the right answer is never, “don’t test.”

* – This is, admittedly, a rough figure that assumes an 85% confidence level. To check statistical significance for your tests, use Google Website Optimizer’s built-in tool or download the Teasley Stat Calculator [Excel spreadsheet] here.

Are you getting enough value out of your small business website? Want to make sure you’re ready for the mobile, social, local web? Subscribe to thinks today and get more than just news. Get understanding.

Tim Peter

Tim Peter is the founder and president of Tim Peter & Associates. You can learn more about our company's strategy and digital marketing consulting services here or about Tim here.

This Post Has 14 Comments
  1. Tim, I couldn’t agree more that testing is the right way to gather data on improving website performance. I checked out the articles that you linked to, and they provided a good overview on how to choose the pages on which you should prioritize your testing.

    I’m curious how you think about prioritizing types of media within individual pages. When you have the option of altering marketing copy, “buy now” links, positioning of contact details, product tour movies, etc, how do you decide where to start?

  2. Jake,
    I have the perfect answer for this. But I’m going to make you wait a day or two for it, because it’s such a great question, it deserves an equally good answer. Give me a couple days, and I will write it up the answer as a separate post. Thanks for the comment and keep me honest on the follow-up.

Leave a Reply

Your email address will not be published. Required fields are marked *

Back To Top