IA Summit talk on search analytics
Rich Wiggins and I presented on search analytics at the recent IA Summit in Las Vegas. The talk was quite popular; it was one of five that attendees asked to have repeated. And in the week or so since we posted it on SlideShare, it's had over 1,200 views and has been favorited 20 times.
You can play the embedded version below; enjoy!
—Lou Rosenfeld
Comments
Caught it at the Summit; really great presentation. I actually saw a preview of some of this last June, just prior to starting a new job, and have since applied many of the analytics methods covered then to internal projects. We've gained fast, representative, and valuable data using these methods.
One of the projects involved using query logs to assess the quality of a search engine. We started by identifying the most popular queries, which we grouped into broad categories. We then sampled queries from each of those groupings and tried them out using the search engine, applying various measurement criteria.
We found it pretty easy to develop some compelling quantitative metrics about the quality of search. Better yet, they could be expressed as simple percentage values that were easy to summarize in top-level reporting to management. The method wasn't perfect, but it gave us some great benchmarking data that we can use to set objectives for improving the search user experience.
One note: We did find that you can't completely ignore the long tail. The short head is the #1 priority, but looking only there excludes more complex searches that are not at all uncommon, but are unlikely to be phrased in the same way by more than one person.
Posted by: John Ferrara | April 8, 2007 06:40 PM
I'd like to echo John's comment about the long tail, and I'm interested in others thoughts on how it can add to richness to what we understand about the short head.
Right now I'm weighing up to what degree I should review a long tail so as to judge how stemming and synonym behaviours of our search tool need to reflect how users think. I always worry that a significant number of the one off search phrases actually add up to some kind of magical mental model which I may miss ... something that clears up ambiguous thinking patterns, or shows that a short head result actually represents only a small percentage of users thinking on a given topic.
I'm leaning towards using an analysis tool such as The Mole - does anyone do something similar?
Posted by: Brian Lyall | April 10, 2007 08:03 PM
John's been doing some great SA work at Vanguard; perhaps an article is due my friend?
Brian, I've never heard of the Mole; can you give some background on it?
Posted by: Lou | April 10, 2007 09:30 PM