Lou,

I'm not sure this is a problem specific to search analytics -- trying to figure out the ROI on political ads or a public relations campaign can be tough, too.

I think as with many things, this probably requires having a good, hard think about what success would look like. Is it simply more visitors to your site? Is it something like greater awareness of your company, which might be measured by an outside survey firm? Is it more sales? Better polling numbers?

As with many things related to marketing (which is what I think a Web site often is for a business), I think you've got to properly define those goals to figure out how to connect big picture to site search metrics -- for example, maybe you want to drive every single visitor as easily as possible to your shopping cart functionality, or you just want to measure how many times the phone rings in your sales department?

Maybe this is all stating the obvious but it seems to me too many people jump to choosing a tool without first deciding what it is they're building.

Best of luck.

Colleen

While this discussion is still fresh on my mind with Lou yesterday...

KPIs: The KPIs I have listed, (and all the others, which may be metrics not KPIs BTW depending on your needs), are meant to be starting points 'from the top', a place to begin if you will. They are meant to act as guides that lead you down a "structured path" to be able to perform data analysis. Now, what does that really provide? It provides you with boundaries to work within and think about. Why would you need such a thing? Because there is SOOOO much data out there that you have to be able to decipher the "signals through the noise" and having these structured paths (KPIs) 'to walk down' provide you with those ways of doing just that. REMEMBER all data is valuable, not all of it is useful. You need ways to cut through the clutter.

Mapping: Now is a perfect simple world I/we could take your business model, a set of associated KPIs, your business goals, and deliver you actionable insights. Now, I emphasize the word 'simple' here, because in essence that is what I do for my clients, but here's the kicker... each and every company / client / etc are unique and that means I would have to create a cookie-cutter template that everyone would have to abide by. Now, to a degree I/we will be doing just that in the book and Lou has been pestering me to put some of that on the blog as well, but again it is only a starting point. What you must do after this is look at your individual business model, business goals, time, money, resources, technology, processes, and more to determine which KPI(s) will produce the best results for you.

Example: While giving my talk in Spain last week a person from the audience said (referring to slide 43 of my presentation) that for them "Zero /Null Results" was their number one KPI for search analytics. For them it was a very simple way to understand what content people were looking for in comparison to what content there was or wasn't on the website. And for them that works, and they are happy and it produces great insights that they can take action on. And that is the ultimate goal of what we are trying to accomplish with data analysis: some 'nugget' of information (an insight) that can be applied (action) to achieve better results (business & user).

Colleen's comments above: You make a couple of great points and that this issue is not limited to search analysis.

"trying to figure out the ROI on political ads or a public relations campaign can be tough, too."
-- I disagree with this as a blanket statement for ROI, but I'm not saying it's easy either. It's a multifaceted issue that we are not entirely equipped for today and that is the ability to measure & track to a degree we'd like between channels (multichannel metrics). That being said, if it's online it can be tracked, measured, and monetized.

"this probably requires having a good, hard think about what success would look like"
-- Couldn't agree more! I did a talk at the NY UPA in February on something I created called "Designing Outcomes". What is does it breaks down the process into 3 major areas: Business Goals, UX, & Analytics and 'forces' you to think about how to use the data from each to "design the desired outcome". The easiest way of doing this is to start at the end (desired result) and work backwards.

"too many people jump to choosing a tool without first deciding what it is they're building."
-- Oh indeed they do! The way it should be is as follows: Choose your people first. Then define you processes based upon your time, money, people. Then based upon all that ONLY THEN should you choose your technology. I am such an advocate or this method that there is simply no argument whatsoever. To prove this point to clients I have previously challenged their existing team who usually has Omniture or CoreMetrics, and I will use only myself and Google Analytics to get the best results on "X & Y". To date I've never lost. Technology only comes into play when you have maximized your internal resources (people & processes).

Lou,
I'll be discussing metrics-driven research in my upcoming talk at DRC (http://trex.id.iit.edu/events/drc/2009/speakers/tomer-sharon.html).
I will describe a 4-step model for measuring the user experience success (or failure) of a new Google product.
1. The process involves defining user experience goals for the product, that are separated from its business goals but are very relevant to its success or failure.
2. Next, UX goals are translated to signals in the interface that would tell whether success is achieved or not.
3. Then, signals are translated into specific metrics. For example, a UX goal would be "Efficiency", a signal for that goals is a measure of how lost users are when they complete a certain task, and a metric would be a "Lostness" score (that is calculated by a lostness formula).
4. The last step of the process is working with the product team (designers, engineers, and product managers) to make it happen.
I'd love to talk more about it (I'm currently at UPA 2009. Are you?).
Tomer