Comments on: How much testing is too much testing? http://www.timpeter.com/blog/2009/03/23/how-much-testing-is-too-much-testing/?utm_source=rss&utm_medium=rss&utm_campaign=how-much-testing-is-too-much-testing Interactive marketing and e-commerce strategy blog for the local, mobile, social web Wed, 08 Feb 2012 18:08:35 +0000 http://wordpress.org/?v=2.9.1 hourly 1 By: Tim http://www.timpeter.com/blog/2009/03/23/how-much-testing-is-too-much-testing/comment-page-1/#comment-2939 Tim Thu, 26 Mar 2009 11:47:49 +0000 http://www.timpeter.com/blog/?p=650#comment-2939 Hi Tom, Thanks for the comment. And you're right, sometimes you need to make a decision. Having said that, with Google's scale, their tests won't have to go on forever. Not even close. Take a look at the chart above. With a sufficiently large improvement, you need as few as 100 conversions to find which version works best. Now, I doubt one shade of blue is going to have a 20% lift. But <strong>one recent test I did generated a 7% lift just by changing a font color</strong>. Given Google's 25 billion pageviews monthly (according to <a href="http://www.anrdoezrs.net/click-2955913-10521478" rel="nofollow">Compete</a>), how long will it take them to get 1,000 conversions? Not long at all. Even if the page they're testing gets 0.01% (i.e., 1/100th of 1%) of Google's traffic, assuming a 2% conversion rate, they'd see almost 1,700 conversions weekly (25 billion/30 days * .0001 * 2% = 1,667). So, Google could run this test surprisingly quickly. Of course, their scale is just <em>a tiny bit</em> larger than most, so testing 41 shades of blue simply isn't practical for most companies. But, <strong>testing is much faster than most people think</strong>, whether for Google or for smaller companies, so long as the test is designed for the scale of the organization. Thanks again for your comment. Look forward to seeing you here again soon. Tim Hi Tom,
Thanks for the comment. And you’re right, sometimes you need to make a decision.

Having said that, with Google’s scale, their tests won’t have to go on forever. Not even close. Take a look at the chart above. With a sufficiently large improvement, you need as few as 100 conversions to find which version works best. Now, I doubt one shade of blue is going to have a 20% lift. But one recent test I did generated a 7% lift just by changing a font color. Given Google’s 25 billion pageviews monthly (according to Compete), how long will it take them to get 1,000 conversions? Not long at all. Even if the page they’re testing gets 0.01% (i.e., 1/100th of 1%) of Google’s traffic, assuming a 2% conversion rate, they’d see almost 1,700 conversions weekly (25 billion/30 days * .0001 * 2% = 1,667). So, Google could run this test surprisingly quickly.

Of course, their scale is just a tiny bit larger than most, so testing 41 shades of blue simply isn’t practical for most companies. But, testing is much faster than most people think, whether for Google or for smaller companies, so long as the test is designed for the scale of the organization.

Thanks again for your comment. Look forward to seeing you here again soon.

Tim

]]> By: Tom Lindstrom http://www.timpeter.com/blog/2009/03/23/how-much-testing-is-too-much-testing/comment-page-1/#comment-2934 Tom Lindstrom Wed, 25 Mar 2009 14:01:49 +0000 http://www.timpeter.com/blog/?p=650#comment-2934 Testing is really important, but sometimes you must make a decision.The results of the testing are never immediate anyway so it´s going to take some time to see which one is better. If they are testing 41 shades of blue, they will have to go on forever :) Testing is really important, but sometimes you must make a decision.The results of the testing are never immediate anyway so it´s going to take some time to see which one is better. If they are testing 41 shades of blue, they will have to go on forever :)

]]>