skip to Main Content

How much testing is too much testing?

How much testing is too much?

Testing vs. design
When are designers right? When are engineers? How do you know how much testing is too much? Designer Douglas Bowman announced his decision to leave Google last week, largely for this reason:

“Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that. I’ve grown tired of debating such minuscule design decisions. There are more exciting design problems in this world to tackle.”

Shortly after Bowan’s announcement, such diverse outlets as Jeffrey Zeldman, Joe Clark and CNNMoney’s Jon Fortt questioned whether Google’s process is killing design (or as Jeffrey Veffer calls it “Design”). Clark goes so far as to suggest Google smothers designers.

Nonsense. Ignore this noise. While “form vs. function” or “design vs. engineering” makes for a great “religious debate,” ultimately it’s pointless. Great designers can help you be successful. So can great engineers. And when they work well together, truly amazing things can happen. But, not testing the work of either can lead you to failure. Falling into the trap of either/or is a poor decision. Making this an “x vs. y” ignores the power of great design AND testing.

Y’see, the flaw in this gnashing of teeth around the web is that the folks jumping all over Google on this one seem to assume only one path leads to success. Either you have a great designer or you’re doomed to failure. Many people point to Steve Jobs’ passion for design fueling Apple’s turnaround – and rightfully so. But, despite its (supposed) lack of design savvy, Google seems to be doing pretty well for itself despite how few design gurus are throwing parades in its honor.

Jeffrey Veffer points out,

“…this is not design (or “Design”) that is the equivalent to window dressing or chrome or garish menus, but really a holistic approach to approaching problems… [Bill Buxton] mentioned that design is probably the “most negative” profession out there as it required a continual progression from a blank sheet of paper (millions of possibilities) to exactly one, with all the rest being thrown out!” [emphasis mine]

Isn’t A/B testing one way of throwing out possibilities? And isn’t A/B testing within the reach of most businesses?

Think about this. The number of conversion actions you need to see to have a statistically significant sample for a given improvement looks like this:

Sample Improvement Size
100 20%
1,000 6.3%
10,000 2%
100,000 0.63%

(Source: Tim Ash, “Landing Page Optimization”)

So, sure, testing 41 shades of blue is overkill. For you. But not necessarily for Google. Given Google’s 140 million unique visitors monthly, about how long do you think it would take Google to test 41 shades of blue and get 100,000 samples each? Not long at all. And given that 0.63% of Google’s Q4 revenues equals about $36 million, it might make sense for them.

Who knows? Maybe with the additional revenue Google makes from this test, they can afford to hire more designers.

My point here isn’t to defend Google. They can defend themselves. Nor is it to criticize Bowman; he can – and should – work for whomever he chooses, using whatever methodologies suit him.

The point is that testing works. I’ve tested major changes – loved by designers – that had little effect. I’ve tested minor tweaks – despised by engineers – that drove big lifts in revenues. But we only knew what worked and what didn’t because we tested. Companies that work towards an optimization culture get the best results from both their designers AND their engineers. Anything less is a sucker’s bet.

Disagree? Think designers are nuts and engineers know best? Or vice versa? Don’t be shy. Tell me what you think in the comments below.

Are you getting enough value out of your small business website? Want to make sure your business makes the most of the local, mobile, social web? thinks helps you understand how to grow your business via the web, every day. Get more than just news. Get understanding. Add thinks to your feed reader today.

Or subscribe via email.

And while you’re at it, don’t forget to follow me on Twitter.

Image credit: Scott Schram via Flickr using Creative Commons Attribution 2.0 Generic.

Technorati Tags:
, , , , , , , , , , , , , , ,

Tim Peter is the founder and president of Tim Peter & Associates. You can learn more about our company's strategy and digital marketing consulting services here or about Tim here.

This Post Has 0 Comments

  1. Testing is really important, but sometimes you must make a decision.The results of the testing are never immediate anyway so it´s going to take some time to see which one is better. If they are testing 41 shades of blue, they will have to go on forever 🙂

  2. Hi Tom,
    Thanks for the comment. And you’re right, sometimes you need to make a decision.

    Having said that, with Google’s scale, their tests won’t have to go on forever. Not even close. Take a look at the chart above. With a sufficiently large improvement, you need as few as 100 conversions to find which version works best. Now, I doubt one shade of blue is going to have a 20% lift. But one recent test I did generated a 7% lift just by changing a font color. Given Google’s 25 billion pageviews monthly (according to Compete), how long will it take them to get 1,000 conversions? Not long at all. Even if the page they’re testing gets 0.01% (i.e., 1/100th of 1%) of Google’s traffic, assuming a 2% conversion rate, they’d see almost 1,700 conversions weekly (25 billion/30 days * .0001 * 2% = 1,667). So, Google could run this test surprisingly quickly.

    Of course, their scale is just a tiny bit larger than most, so testing 41 shades of blue simply isn’t practical for most companies. But, testing is much faster than most people think, whether for Google or for smaller companies, so long as the test is designed for the scale of the organization.

    Thanks again for your comment. Look forward to seeing you here again soon.


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Back To Top