The problems with experts

As I wrote earlier, I participated in an ExperTease game sponsored by search engine Ask at the Web 2.0 Expo. The game was a clever of way conveying an idea that Ask has up its sleeve: returning “expert” search results.

In the game, audience members were invited to identify themselves as experts on various topics such as mashups, venture capital and Silicon Valley gossip. If you answered correctly, you went up on stage to compete with experts on other topics. During the game you only got a point if you could correctly answer the question in your area of expertise and the other contestants couldn’t correctly answer a question on your topic.

The premise was that you will get the best answers to complex questions if you ask an expert. What happened in the room actually served to prove the fallacy of experts.

One of the questions was “What’s the largest acquisition in the Silicon Valley in the last year?” The expert’s answer (and the correct answer according to the questioner) was Google’s acquisition of DoubleClick at $3.1 billion. An audience member pointed out that this was edged out by Cisco’s acquisition of WebEx for $3.2 billion.

Relying on an “expert” has a number of problems:

It’s impossible for one person to know everything, even on a specific topic. As much as Mike Arrington knows about startups in Silicon Valley and John Battelle knows about search, there are fine-grained details that other people know better.

Experts may have conflicts of interest. You many not always get the complete picture from experts because of confidentially agreements. You might also get a biased picture because they’re trying to sell a specific product.

Experts have day jobs. They don’t make a living answering your questions for free. But they’re willing to offer nuggets of information.

Some questions don’t have a single right answer. Many questions have nuanced answers or answers that differ based on the person who is asking the question. In comparing values of acquisitions, do you net out the $300 million in cash WebEx had, making the cost to Cisco $2.9 billion? If someone asks, “What’s the best digital camera?” the best answer is based on the criteria used.

Not all experts are experts. For example, the scientists that the Bush administration keeps trotting out to deny global warming. Astroturf organizations with impressive, honorable sounding names trot such out fake experts to protect their industry’s revenues.

Instead of a single expert, the best answers can be found among communities where people who are passionate about specific topics gather. Take bulletin boards. If you’re looking for advice on audio/video gear, AVS Forum is a great place to go. If you want advice on travel, is a great resource.

In these forums, the participation of the community constantly refines the answers. The FAQs on these sites are some of the best syntheses of topic-specific knowledge out there; they’ve been beaten to life by constant discussion (and flame wars.)

Blogs also serve as communities. Although the blogger typically “publishes” an article, it’s not uncommon to see bloggers learn from their audience through comments and emails. They then update their own posts based on that feedback.

I’d love to see a blogging platform offer a Wiki version of blogs, allowing anyone to change the content of the main post by the author. Readers coming to the blog would see the “published” version as created by the author, but also be able to see the community edited version with the changes highlighted. I’m betting that you’ll get a lot of insider participation on stories.


About Rakesh Agrawal

Rakesh Agrawal is Senior Director of product at Amazon (Audible). Previously, he launched local and mobile products for Microsoft and AOL. He tweets at @rakeshlobster.
This entry was posted in search. Bookmark the permalink.

2 Responses to The problems with experts

  1. Alan Tai says:

    Agreed on all points. This is probably the reason that “answer sites” such as Yahoo! Answers and Ask Metafilter are more entertainment than actual reliable resources. You could probably argue that the answer sites are nothing more than forums, except the ratio of experts to spammers, wise-cracking kids, and just plain misinformed (but kindhearted) joes is disappointingly small. The real domain experts hang out at THE places to talk about their passion; e.g. AVS forum for audio/video (as you mentioned) and DPReview for photography. Unfortunately, most folks, however, don’t know about these sites initially. Blogs are also good resources. Now, if there were a good vertical search engine to dig through these top forums… (Google groups doesn’t count)

    As for the the wiki version of blogs, it sounds like a good idea at first. Unfortunately, I think it would quickly devolve into a fantastical ficlets story, with disparate opinions and tangets galore. Part of the reason wikipedia works is that there is (for the most part) some agreement on the facts. Also, you lose out on the tree structure and quoting inherent in blog comments and forums. I like it how it is…

  2. As much as I enjoy and appreciate specialist forums, and I realise that an ‘Expert’, on stage, under quick response may be capable of mistakes, I still think the value of professional experts is being undervalued in the rush to praise users.

    While the end product of a forum debate might yield the right answer, it’ll take a while for that answer to form. Indeed, it might never be satisfactorily be resolved. And in the meantime, how do you know who to trust without investing a lot of time and effort into researching the subject?

    Meanwhile any expert worth their salt will not only have an idea of an answer, but will also research it to make sure they’re right. And still end up with a correct answer before the forum has finished.

    Personally I think the optimum method is to combine the two and get the best of both worlds, rather than promoting one over the other.

Comments are closed.