August 9, 2007 12:00 PM
As part of our ongoing research of the UX environment, we recently took a closer look at the six major analyst firms (Aberdeen, AMR, Forrester, Gartner, IDC, and Yankee). We were hoping to determine if the analysts were paying much attention to user experience, so we searched a variety of UX-related terms (21, to be precise) on their respective web sites. We then looked at which firms paid attention to which UX topics, how these firms stacked up against each other, and how they compared to the web's overall UX consciousness.
The following table summarizes our general findings (highly recommended: view the full-size version—much easier to read). We used Google searches as our yardstick to gauge our UX terms' relative strengths across the web. So, for example, of all the UX-related documents that Google knows about, 10.7% mention knowledge management. We did the same thing for each analyst site, using their own local search systems, and then we averaged those numbers:
Which numbers really stood out among the analyst firms?
- Aberdeen is focusing on web analytics, which represents 23.7% of their UX-related documents (versus 3.8% on average for all the analysts).
- AMR pays a great deal of attention to the related areas of content management (38% versus 25.8% on average) and knowledge management (21.8% versus 11.9%).
- Forrester appears relatively strong in areas that are relatively new, such as experience design (2% versus 0.7%), interaction design (2.1% versus 0.2%), interface design (2.5% versus 0.9%), SEO (2.8% versus 0.5%), UCD (2.7% versus 0.2%), and web analytics (8.4% versus 3.8%). User experience, itself a recent term, is the most common term among Forrester's search results (16.4% versus 11%).
- Gartner's bread and butter is information management (34% versus 25%); not surprisingly, they also take the top position in information architecture (5.6% versus 2.6%).
- IDC's numbers are, overall, closest to average; if they have a specific focus, it's content management (32.6% versus 25.8%).
- Branding seems to dominate Yankee's mindspace (50.9% versus 13.9%). Perhaps they're also quite interested in the branding aspects of user experience (23.4% versus 11%).
In terms of the relative strengths of each UX topic, the analysts seem focused on organizing information; the top two topics are content management (25.8%) and information management (25%), with knowledge management coming at #4 (11.9%). So it's surprising that information architecture checked in at a low 2.6%. And, in a different way, it's surprising that the user experience, often confused with information architecture, ranks #5 at 11%. Branding took position #3 at 13.7%, which seems reasonable as analysts' clients typically have large and established marketing budgets. At 3.8%, web analytics is in sixth place; while this may sound low, it's probably rising quickly. It'll be interesting to check back in a year.
Our second illustration shows how the analysts, on average, stack up against the Web's UX consciousness as a whole (as expressed in Google search results):
It's interesting to look at the discrepancies here. The analysts "lead" the web on these topics:
- Experience design (0.7% versus 0.2%)
- Industrial design (1% versus 0.3%)
- Information architecture (2.6% versus 0.2%)
- Information management (25% versus 11.1%)
- Interaction design (0.9% versus 0.3%)
- Search analytics (0.2% versus 0.02%)
- User experience (11% versus 7.9%)
With the exception of information management, the analysts' numbers are small, though larger than those witnessed on the web as a whole. So are the analysts leaders here? It's possible, but quite hard to tell without longitudinal data; we'll revisit the numbers in a year.
What about topics where the analysts trail the web?
- Ergonomics (2.1% for the web versus 0.7% for the analysts)
- Graphic design (13.4% versus 0.7%)
- Human-computer interaction (2.6% versus 0.2%)
- Search engine optimization (2.4% versus 0.5%)
- Technical communication (0.2% versus 0.05%)
- Usability engineering (0.1% versus 0.04%)
- Web analytics (9.1% versus 3.8%)
Interestingly these topics, with the exception of SEO and web analytics, all represent fairly established fields. Do analysts forgo these areas as insufficiently innovative? If so, many of these field's practitioners would surely take issue. Conversely, SEO and web analytics are new fields which are not only considered quite innovative, but have accrued legions of software vendors. Given that the mission of many analysts is to help managers understand new technologies, it's especially strange that the analysts have not paid more attention to these two newer areas.
We're sure there are plenty of problems with our interpretation of these data, so we'd especially like to hear your thoughts. Let us know if you'd like the raw data and we'll send you the spreadsheet. We hope to repeat this study again in a year, and we're considering doing the same thing in other areas where we'd like to gauge UX mindshare, so please let us know how we can do better.
Comments
I'm an analyst in the Customer Experience group at Forrester Research.
In noting the lack of reports on topics like graphic design, human-computer interaction, and usability engineering, you ask, "Do analysts forgo these areas as insufficiently innovative? If so, many of these field's practitioners would surely take issue." That's not the case at all – but we're not writing for these field's practitioners. In fact, when I go to industry conferences like UPA, I'm almost always met with a dead stare when I mention that I work for Forrester. Forrester who? Many UX practitioners just aren't tuned into analyst research.
Who do we write for? Typically it's executives and managers in marketing, customer experience, or eBusiness departments. Their major pain points fall into categories like benchmarking their customer experiences, building a business case for UX within their organizations, and transforming the way their companies do business. As any good UX practitioner would do, we try to understand our audience and then provide the content that is going to best meet their needs.
So it shouldn't be surprising that we write a lot about UX organizational structure, customer-centric cultures, design processes, evaluation methods, personas, interactive design agencies and usability vendors, UX offshoring, UX management in global or multinational firms, ROI, strategy, and what we call Experience-Based Differentiation. We help our clients assess the user experience and value of emerging technologies like rich Internet apps, social media, and mobile. We help them understand specific customer groups like older users and Gen Y by providing deep behavioral data. We also just published our initial research report on desirability – a topic that's dwarfed by usability in "the web’s overall UX consciousness."
Thanks for doing this study. I think it's great that someone is examining the UX focus (or lack thereof) of the analyst firms. However, I think your list of topics needs to be modified to better match the UX needs of the typical analyst audience.
Posted by: Kerry Bodine | August 10, 2007 10:21 AM
Kerry, thanks for reading and for commenting; I really appreciate it. Would you mind sharing some of the terms you think we should add to our list?
Posted by: Lou Rosenfeld | August 10, 2007 10:46 AM
For starters, I'd make the terms a bit more generic. For example, a search for "usability engineering" on the Forrester site returns 23 docs for the past 12 months. In contrast, "usability" returns 307.
Posted by: Kerry Bodine | August 10, 2007 11:21 AM
Hi Kerry; I'm not sure that'd help, except in the case of "usability". Some one word terms (e.g., "ergonomics") aren't retrieving much as it is, while we wouldn't be helped by searching terms like "information" and "architecture" separately..
Posted by: Lou Rosenfeld | August 10, 2007 11:37 AM
From the UX perspective I am interested in the places I can get relevant data. I have found eConsultancy in the UK a good source for this but I wish I could find more.
Posted by: James Breeze | August 10, 2007 08:17 PM
I think this analysis is a fantastic idea and as many turn to you for guidance, publishing your results will help raise UX conscious, too. Even by helping many understand UX better by reviewing your related terms.
It still surprises me how many people confuse "ux design" with those terms you searched on, supposing it a trendy synonym for one or some of them.
I have a small fear that gauging "UX consciousness" on keyword frequency comparisons may really only prove frequency of misuse of the relatively new terms you searched on, as many expert marketers, including SEO specialists, will use them loosely. But one would have to read all of the articles to see about that.
Thank you!
Posted by: uxdesign.com | August 13, 2007 05:44 PM
It's nice to see that Kerry responded. It is a direct reflection of Forrester's long-term commitment to topics of this nature. However, Kerry tends to be quite LARGELY 'usability'-centric in her involvement in the industry. Her reports about salaries all came from the UPA. And all of her statements/observations are clearly slanted from that narrow perspective as a slice of our practice.
Unfortunately, she is one of the few resources in the industry who has dedicated as much time to the discipline to be taken seriously. Bottom Line: There's plenty of room for some healthy competition (with low barriers to entry).
You noted several typical topics that various analysts cover. The main problem is that these topics are covered in isolation from one another since the really relevant issues are currently in the cracks between them all. Even at Forrester I had been contacting the isolated analysts to help them see how interrelated their research should be. They all seemed to be sympathetic but suffered from one big issue: the holistic view required by our discipline is not supported by their current business model.
So I've changed tactics. I am now following the energy of Enterprise 2.0. Why? Because web-based stuff gets too caught up in issues related to Search Engine Optimization (which is, in my opinion, the same bastardization of a relationship that CRM has been) and other eCommerce specific considerations. And even when we're looking at a group supporting an internet vs. an intranet, half of the challenges as practitioners are similar to those already stated: the organizational challenges of practitioners inside of an enterprise with a business model that thwarts design thinking and design operating.
In my opinion, our greatest untapped professional potential is in the Enterprise where the design of an interface now can directly be tied to productivity and compliance. These are topics that are starting to turn heads. And these are the areas where we do not come head to head with marketeers pretending to understand the nuances of our practice.
Posted by: Paula Thornton | August 13, 2007 07:47 PM