thinks

Yesterday, we started looking at how to build the right metric for your marketing. We’d gotten as far as tracking monthly unique visitors by typed/referred domain.

Throughout the process, we had made a number of assumptions, such as what type of URL would get the best results and what duration to track unique visitors. Once you’ve made these types of decisions, move on. Debate your assumptions until your team fully understands the pros and cons of each assumption. But, unless you’ve got better options, some tracking is better than none. From this point forward, you will be tracking the trends of your metrics and validating that the trend correlates to business results. You may find better ways to correlate later. But don’t change the underlying data if you can help it. As long as you’re comparing apples to apples, it doesn’t much matter if the apples are rotten.

OK, now back to our case study. So, what happened next?

To this point in the process, we already knew how we were going to track awareness of the media, by type. What we needed now was a way to track that awareness through to purchase.

Fortunately, this proved relatively easy. When each customer came to our landing page, the site placed a cookie - a small data file containing some distinct identifiers - into their browser. One of those identifiers was a key that told us their source ID - that they’d seen the landing page and whether they’d come from the URL for the billboard or from the print media. The problem was that our e-commerce engine didn’t have a database field for that value. The solution: a simple database that captured both the order number and the source ID. Finally, we made sure to capture information such as name, address, and email address on all purchases, to see if these were new customers or matched individuals already in our database.

To complete the picture, the team developed a weekly dashboard showing the identified metrics:

• Monthly unique visitors, by source ID (remember, these told us whether the customer saw the billboard or the print advertisement)
• Net change in monthly unique visitors from the prior week (later versions of the test incorporated year-over-year, once enough data existed)
• Sales generated, by source ID
• Net change in sales
• Ratio of sales to unique visitors, by source ID (aka “conversion rate”)
• Net change in conversion

That’s it. At a glance, we had a good sense of which media type brought in both prospects and customers, and the change in those numbers over time.

More important, we were able to make changes to the landing page and the ad copy over time to improve the capture rates for both media, increasing sales among a new customer group. And that’s really why having the right metric makes all the difference.

How did we match up to the 7 keys of successful metrics? Quite well. Clearly, our dashboard was tied to business results and were actionable. They also were timely, trended, segmented and meaningful. They also showed us precisely what we wanted to know. And that’s a good day’s work.

Like this post? Tell us about it in the comments. And subscribe to thinks, so you’ll never miss an update.

A comment from thinks reader Robin the other day asked how to measure adoption of a new program. In fact, Robin’s comment gets to the heart of the matter for many people new to online marketing: how do you choose the right metric for your marketing efforts? While thinks has looked at the keys to successful web metrics before, today we’re going to take a look at one specific case for how to build the right metrics for your marketing.

First, you’ve got to start with the business goal. A metric unrelated to your business goal is, put simply, useless. In this case, the goal was to compare the effectiveness of print and outdoor advertising (billboards, mostly) among specific demographic segments in several markets. This might sound complex, but it’s really asking some very simple questions:

1. Did customers see the advertising?
2. Did one type of media drive a better response than another?

So, we were trying to count:

That’s it.

That outline provided the basis for the metrics we’d develop. Anything that didn’t measure those three factors didn’t matter. Here’s what we did.

First, we started with how to count the number of people who saw the advertising. In this case, we were marketing to Spanish-speaking customers, which required us to develop a unique Spanish-language landing page. Since we wanted to ensure the page design didn’t influence results, we knew all traffic would have to go to the same page. There are ways to track the influence of those differences, but for a limited duration, low budget test, that was overkill.

If we wanted to know how often those people came, we could look at visits to the page or possibly page views. But our goal was to see how many people we reached. The best metric to track the number of people you reach is unique visitor counts. As you’re about to see, it won’t tell you everything. But it will start you on the right path. Since the test was due to run for several months, and since we knew our typical sales window was three weeks, we decided tracking monthly unique visitors provided the best picture. Weekly or daily counts would have been too short and over-counted traffic. Quarterly or annual counts would have been too long a period and under-counted the results. Just like Goldilocks’ porridge, monthly was just right.

But deciding on unique visitor counts was just the starting point. Since the media would appear in both print and outdoor, we wanted to segment the customers to see which type of media drove traffic. Even though all traffic was going to the same landing page, there’s no need to use the same URL for both. In fact, using different URL’s allowed us to track where customers saw the media. We opted to use a “vanity URL” - a domain name specific to the campaign - each place we ran media, one for print, another for billboards.

While there is nothing wrong with using URL’s like www.example.com/media1 and www.example.com/media2 in your marketing - and there are many good reasons to do so - in this case there was value in having distinct domains for each media channel. First, the campaign was in Spanish. We felt having the domain name appear as www.ejemplo.com (that’s “example” in Spanish) was more likely to reassure our customers than www.example.com/página. Second, we wanted to know whether billboards or print were driving the traffic. Customers who saw our brand may have simply gone to our traditional domain. That’s almost always a Good Thing, but for purposes of this test, wouldn’t tell us what we wanted to know. Keeping the domains custom and exclusive to the media type increased the likelihood that the customer had seen, and was reacting to, that specific medium. So, by using the URL ejemplo-uno.com (not the actual domain) for billboards and ejemplo-dos.com in print media, we could then track which medium the customer had seen. This changed our metric to monthly unique visitors by typed/referred domain.

Notice throughout this how the design of the tracking influenced the design of the campaign and vice versa. When planning for your campaigns, you also must plan for what you’re looking to measure. Almost like Murphy’s Law, ignoring what you’re trying to track while building your campaign tactics guarantees you won’t have the data later to support your efforts.

At this point, we had enough information to begin tracking how many people acted on the ads. But awareness isn’t a goal. Revenue is. Tomorrow, we’ll take a look at how the team was able to track the efficiency of the different media types and how that grew revenues.

Like this post? Subscribe to thinks and never miss an update.

A friend forwarded the results of a marketing study to me this morning. It’s from a political campaign, but it highlights key points for any marketing plan:

1. Repeat your message to help influence a “purchase decision” (voting, in this case)
2. Use multiple contact channels to reinforce your message
3. Too much repetition turns off buyers
4. Make sure your analytics are in place. That will show what works and allows you to improve your message, channel and frequency

#4 might just be the most important one. Do you track your key business metrics for all of your marketing?

Avinash Kaushik on 4Q and Measuring Customer Satisfaction

During my discussion of web analytics and business metrics yesterday, I left out one critical metric: customer satisfaction. Until now, few tools existed that made it easy for small business owners and marketing managers to hear what their customers had to say online. That changed last week when Avinash Kaushik and iPerceptions announced the launch of 4Q, a qualitative analysis tool designed to measure customer satisfaction.

I sat down with Avinash (well, emailed, but you get the point) to talk about qualtitative analysis and how site owners can make the most of 4Q. We also talked a bit about the future of the tool and how it came to be.

Tim: First, Avinash, congratulations on the launch of 4Q and thanks for taking the time to do this interview. I’m very impressed by 4Q and happy with the results it’s providing for my site so far.

Avinash: Thanks so much Tim, I am absolutely thrilled that you are finding it to be of value. I personally never get tired of analyzing VOC (voice of the customer), every site is so unique.

Tim: Additionally, I love that 4Q asks your three greatest survey questions ever. Can you talk a bit about what led you to develop 4Q and what your goals are for the project?

Avinash: Several reasons. As you know I inhibit the world of quantitative web analytics and there is a tendency to actually believe in the power of all the numbers. Make no mistake they can be powerful, but for a complex interaction medium like the web, people and habits are evolving too fast to actually get lots of insights into customer intent and preferences. So one thing was the limitations when it comes to the power of pure quant stuff to spit out actionable insights.

A secondary reason was that it is quite atrocious how little most website owner’s and decision makers know about their actual customers. I know that sounds hard to believe, but it is true.

4Q is our humble attempt at helping decision makers have key customer data (in their own voices) as they consider what to do with their sites. It is also to help facilitate the evolution from simply using the “What” type data to “What + Why” type data.

Oh and it is kind of cool and “web 2.0″ to put out something free and of value!

Tim: What benefits should site owners expect from 4Q?

Avinash: Simple:

• Tap into insights directly from their customers.
• Eliminate the need to interpret or throw in your own opinions or overlay “here is what I think works”.

If you know why people come to your site, you’ll design it better. If you know what people find tough about your site, you’ll improve it. If you know they love it, you can go ask your boss for a raise!

Tim: On your blog and in your book, Web Analytics: An Hour a Day, you’ve talked about 6 recommendations for conducting surveys. 4Q does a great job on most of those. One thing I’m missing is how to tie this data to your regular clickstream or behavioral data. How can site owners integrate 4Q most effectively with their behavioral analysis to create a complete picture of their customers’ experience?

Avinash: Connecting the two sets of data would be important, especially as you squeeze all the initial set of juice out in the first say six months or a year.

For GA (Google Analytics), specifically, the best option is to pass a common parameter between both. So, for example, you can pass the anonymous cookie id to the survey and to GA and then use that to tie [them together]. Another alternative is if your survey has a unique value that identifies it, then pass it as a “user defined value” into GA, which also allows you to tie the data and slice and dice.

Tim: And for others?

Avinash: There are some where there is an option to pass a unique identifier (like the user defined value) and there are others where you can import the survey data into the web analytics [tool] after it has been processed. Some of the paid solutions have this option. Please check with your vendor and I am sure they have it.

Tim: 4Q shows its data in real time, without indicating when it’s reached a statistically valid sample. Any concerns about site owners reacting to data that may lead them down the wrong path? How can site owners react most appropriately, particularly if their site receives limited traffic (i.e., a statistically invalid sample)?

Avinash: Can I cheat? Here is a comment I wrote on my blog that addresses this specifically (and there is even a pretty table to go with it!)

The short answer is that for a given time period if you get approximately 300 responses (that’s it) then the data is statistically significant. But I caution and point out that one of the key outputs of 4Q is open text VOC and there, even with small numbers, you are kosher because customers are helping identify issues with your website.

Tim: 4Q is free for its users. Can you talk about the business model? Do you plan to charge users for 4Q at some point in the future?

Avinash: 4Q is and will remain free. That is our plan for now. No plans for a paid version. There are very good complex paid surveys in the marketplace that solve different problems. 4Q is in a different space.

Tim: Are there future development plans for 4Q that you can talk about? I know it just came out last week but, c’mon, we’re impatient folk here on the Interwebs.

Avinash: I think you are going to see some improvements soon, it has been only a few days but we already have so much great feedback.

Don’t hold me to dates yet (!!) but expect to see improvement on the survey invite. I would love for us to add a response rate report, ease of use of adding more sites under one account, and maybe easier to export data out.

So more fun stuff to come. I want 4Q to be truly a customer driven innovation. I plan to stay involved in the evolutionary process of 4Q for the near future, or until they no longer let me (I am sure they are going to get tired of me bugging them!).

Tim: Thanks again for taking the time to talk with us about 4Q. Is there anything you’d like to add?

Avinash: Its my pleasure, thanks for the opportunity.

Tim: Anytime.

How are you measuring customer satisfaction with your website? Tell us about it in the comments. And as a reminder, you can subscribe to 4Q here

Technorati Tags:
business, e-commerce, e-commerce strategy, marketing, small business e-commerce, small business e-commerce strategy

Getting Successful Business Metrics from your Web Analytics Tool

I met analytics guru Marshall Sponder at a party last night. He’s passionate about analytics and doing some exciting work in measuring the value of social media. I’m a big numbers nerd, too. Comes with the territory as a baseball fan and as a geek marketer. Like baseball, you can count everything in ecommerce. Seriously. You name it, ecommerce has a stat for it. And there are many things you should measure. As the saying goes, “you can’t manage what you can’t measure.”

But not all metrics are created equal. For instance, conversion rate, which so many people live by, has some fundamental flaws. While I don’t feel as strong about as Avinash Kaushik, I agree that conversion rate is not the most useful metric. All useful web metrics share 2 key attributes:

1. Web metrics are tied to business results. These aren’t “web metrics” at all. They’re business metrics. If the thing you’re measuring doesn’t tie back to a business result, it’s useless. Sales volume, sales revenue, inventory turn, costs reduced. These are all Real Things. “Fuzzy” metrics don’t pay the rent. And if you’ve got bright young kids on your staff, passionate about web analytics, don’t squash their enthusiasm. But let them know that some web metrics are more important for making payroll, too.
2. Web metrics are actionable. Measuring stuff is great, but to turn my earlier quote on its head: “Measuring stuff you can’t manage is stupid.” For example, conversion rate is tied to business results. But if fails the actionable test. Actionable web metrics also share these 5 features:

1. Actionable web metrics are timely. Finding out what you sold six months ago is great, but it makes it very hard to repeat the process. While you don’t need to measure everything daily, checking that you’re going in the right direction once a month or less may not give you enough time to correct for any problems.
2. Actionable web metrics are precise. Don’t confuse precision with accuracy. While Avinash Kaushik explains the difference between precision and accuracy better than I can, I’ll summarize with my favorite quote about this topic: Apples are apples. It doesn’t matter if your apples are rotten as long as you’re comparing ‘em to other rotten apples.”
3. Actionable web metrics are segmented. Improving your web metrics requires pulling certain levers. Segments quickly help you figure out which levers to pull.
4. Actionable web metrics are trended. A number in isolation tells you nothing. You’ve got to see what direction they’re moving, too. I personally like to see a week at a time and how the same week stacks up against the prior year, for instance.
5. Actionable web metrics are meaningful. Statistical significance* is important, too, but meaning is more important. Here’s why. Assume you get 200 sales from 1,500 visitors vs. 20 sales from 30 visitors. While the 66% conversion rate of the latter is better, in statistical significance terms, 200 sales is better than 20 in the Real World. Only focus on statistical significance when you’re measuring meaningful results. 4 sales on 5 visitors is statistically better than 1 sale from 5 visitors, but, unless your margin per sale is huge (and there are cases where that would be sensational), would you care that much?

OK, so theory aside, what should you track? Well, that varies by business (remember, they’re business metrics, not just web metrics). As an example, though, here’s a representative set of the things I pay attention to and how often:

• Sales trend by segment (every day and weekly aggregate)
• Unique visitors trend by segment (every day and weekly aggregate)
• Revenue trend by segment (every day and weekly aggregate)
• Bounce rate by traffic source (weekly)
• Bounce rate by campaign (weekly)
• Bounce rate for top pages (weekly)
• Days to purchase (monthly)
• Visits to purchase (monthly)
• Top abandoned pages (monthly)
• Site overlay (monthly)

And I like to look at them by these key analytic segments:

• Typed URL
• Paid search
• Natural search
• Internal search
• Referred
• Email
• Repeat visitors
• “New” visitors

I don’t always drill down into the details by segment if the trends hold to targets, but it’s handy to have the data if you deviate from the trend or goal. Seasonal adjustments in your trends are a nice touch, too, if you can handle it. (See, I told you that ecommerce can count everything. And you thought I was kidding).

I don’t suggest you need to track all these things. Some businesses may only care about select subsets. The key point is that each of these meet the requirements for useful metrics. What items work for your business? Let me know in the comments.

* - By the way, if you need to determine statistical significance, Brian Teasley has a great set of statistical significance calculators available here.

Seth Godin writes, “…when we start delivering numbers with that level of accuracy, people can’t help but believe them.”

I’m a big believer in analytics and using data to support business decisions. Just don’t get blindsided by unsupported data. And, for God’s sake, don’t perpetrate this kind of silliness yourself. I’m 84.7% sure that’s a bad idea.