Last week at Emetrics I was speaking with a company doing vertical search. We discussed metrics like number of “next page” clicks, time to first click, and lots more, in order to measure the user experience. Metrics often take the place of real data, e.g. for inferring things like relevance of the search results.
So, thought experiment: Visitor A does a search, clicks on 15 links. Visitor B does a search, doesn’t click on any links. Visitor C does a search, clicks on 2 links. Average clicks per search is 5.66. Which visitor had the best experience?
The answer is that it depends on what the search term was.
- Visitor A typed santa barbara photos. They want to see lots of ’em. The fact that they looked at 15 is a good sign.
- Visitor B typed time in santa barbara. Poof, the time is on the results page. No need to click further.
- Visitor C typed santa barbara tourism, and was perhaps confused by both http://www.santabarbaraca.com/ and http://www.santabarbaraca.gov/. Still, the first two results seem to indicate a success.
So, it would be misleading to compute an average, and compare each result against the average. If you categorized your searches into terms like information, browsing, and general, you could group your searches and compute your metrics by category. You’d want your information searches to have low clicks, and browsing searches to have high clicks.
Absolutely. But you might want to go one step further and create a target. With a priori goals, it’s hard to know whether you’re getting anywhere. If you’re not sure what the absolute value of the target should be, you can always express it in terms of a percent improvement over recent experience.
Your categories might be considered objectives and we’d want some definitions for each objective. For example, while it seems obvious that for some of these objectives we want to minimize clicks and for others we’d like to maximize them, they may depend on a site’s philosophy. To use an off-line example, 7-11 wants to get buyers in and out of a store as quickly as possible while Wal-Mart intentionally designs stores to “capture” consumers for as long as possible with the hopes that “always the low price” will cause you to buy more.
On-line we need to consider the ads that are displayed with these search terms. Are there some objectives that fewer (or even zero) ads might be a better fit for the desired experience?
Many of us create metrics for metrics’ sake. It’s often worthwhile to step back and figure out what we’re trying to accomplish.