Jeopardy! contestants Ken Jennings and Brad Rutter got a spanking the likes of which they’ve never seen by Watson, an IBM supercomputer, last night.
How would Watson do against that other font of knowledge — Google? Watson is optimized to understand human language, which can be full of ambiguity, wordplay, sarcasm and puns. Google makes some accommodations for language, but keywords do the heavy lifting.
Google did both much better than I expected and about as well as I expected. It did much better in the sense that I expected Watson to runaway from Google as much as it ran away from Jennings and Rutter. It did as well as I expected in that before I entered each clue into Google, I predicted how well Google would do. For example, I knew the Beatles lyrics category would be a piece of cake for Google. Those predictions were generally correct.
Watson had the correct response for 79% of the clues. (This includes clues for which Watson didn’t buzz in.) Google had the correct response in the first position 56% of the time and in the first 10 positions 79% of the time.
Google and Watson tended to struggle on the same clues. Both had trouble on Final Frontiers, Alternate Meanings and The Art of the Steal categories.
The only clue that Watson got wrong and Google nailed was “The first modern crossword puzzle is published & Oreo cookies are introduced” in the category Name The Decade. Google’s answer was found on this page that provides explanations of the New York Times crossword. If Watson could have heard Jennings’ incorrect answer of “1920s,” his backup answer of “1910s” would have been correct.
Finding a document with the right answer is one thing. Synthesizing that information into an answer is another significant challenge. Google only partially does that. I scored Google based on the snippet returned. While it wasn’t generally able to provide the precision of Watson, it did provide the right parts of the document with the answer.
As for Final Jeopardy!, neither Watson or Google got it right.
The clue in U.S. Cities was “Its largest airport is named for a World War II hero; its second largest, for a World War II battle.” Watson hesitantly answered Toronto. Google’s best guess would’ve been Arizona. Even geographically challenged Americans would know that neither answer fit the parameters of the category. (I hope.)
Oddly, that’s a relatively easy clue for humans. Think of U.S. cities, narrow it to the big ones, narrow that to ones that have multiple airports and then go through the list. Kennedy… Dulles… LAX… SFO… Hobby… Love…
A few thoughts on search from this exercise:
- Google still has serious problems with content duplication. One query generated seven pages of essentially the same AP story about Jeopardy!
- Content farms got in the way of the best answer a few times.
- Wikipedia frequently provided the correct answer.
- The day that Google can synthesize answers to queries is closer than I thought. Bad news for publishers.
Also see Danny Sullivan’s great analysis of the underlying technologies and the differences between natural language processing and keyword search.
Jeopardy! Watson vs. Google, Game 1 comparison. Each cell represents a question. Grid position maps to the position on the Jeopardy! game board; you can see the clues and responses at the Jeopardy! Archive. The colors indicate Watson's performance; the numbers and "X"s indicate Google's performance. Watson is only given credit for answers in its #1 position. If you give Watson credit for having the answer in any of its 3 positions, it would pick up 6 more questions.
Methodology:
- I used a private browsing session to avoid any influence from my search history.
- I ran the exact text of the clue (as provided on the Jeopardy! Archive) through Google.
- I excluded all results that were specific to the Jeopardy! match.
- I looked for the text of the answer on the search results page, including the result title and snippet.
- If the correct response appeared in the title or snippet, Google got credit. For the Name the Decade category, I gave Google credit if a year appeared that was in the correct decade. That would not be a correct response for the game, but I felt it was a better comparison for these purposes.
- Search results change based on time, new content published, geography and other factors. You may not be able to duplicate these results. The optimal tests would be done on Google’s index prior to the airing of these episodes.
Disclosure: I have several good friends who work at Google and went to high school with co-founder and CEO Larry Page.
Rakesh Agrawal's blog 
Pingback: Tweets that mention Google does remarkably well in comparison to Watson. I was surprised. -- Topsy.com
Pingback: Quora
Great analysis. Any chance you can share more specifics. I’m particularly interested in the Content Farm angle. Which sites did you consider Content Farms? How often did those sites provide the right answer? What about Answer sites? How often did the correct answer come from them (and which ones)? Thanx
Pingback: Local search is starting to get more social « reDesign mobile
Pingback: Google Hotpot a strong competitor to Yelp « reDesign mobile
fantastic submit, very informative. I’m wondering why the opposite specialists of this sector do not understand this. You should continue your writing. I am confident, you’ve a great readers’ base already!
Hello there, just became alert to your blog through Google, and found that it is truly informative. I am gonna watch out for brussels. I will appreciate if you continue this in future. Many people will be benefited from your writing. Cheers!
It’s perfect time to make some plans for the future and it is time to be happy. I’ve read this post and if I could I want to suggest you some interesting things or tips. Maybe you can write next articles referring to this article. I wish to read even more things about it!
Great post. I was checking constantly this blog and I’m impressed! Extremely useful information particularly the last part 🙂 I care for such info a lot. I was looking for this certain info for a long time. Thank you and good luck.
hello there and thank you for your info � I have certainly picked up something new from right here. I did however expertise a few technical points using this website, as I experienced to reload the site lots of times previous to I could get it to load properly. I had been wondering if your web hosting is OK? Not that I am complaining, but sluggish loading instances times will very frequently affect your placement in google and can damage your high quality score if ads and marketing with Adwords. Well I�m adding this RSS to my e-mail and can look out for much more of your respective fascinating content. Make sure you update this again very soon..
Wonderful goods from you, man. I’ve understand your stuff previous to and you are just too fantastic. I really like what you have acquired here, really like what you are stating and the way in which you say it. You make it enjoyable and you still take care of to keep it smart. I cant wait to read much more from you. This is really a great website.
Pretty nice post. I just stumbled upon your weblog and wanted to say that I have truly enjoyed browsing your blog posts. After all I�ll be subscribing to your feed and I hope you write again soon!
I like the helpful information you provide in your articles. I will bookmark your weblog and check again here regularly. I’m quite certain I�ll learn many new stuff right here! Best of luck for the next!
I think this is one of the most vital info for me. And i am glad reading your article. But wanna remark on some general things, The site style is perfect, the articles is really great : D. Good job, cheers
We’re a group of volunteers and starting a new scheme in our community. Your site provided us with valuable information to work on. You have done an impressive job and our whole community will be grateful to you.
Undeniably believe that which you stated. Your favorite reason appeared to be on the web the simplest thing to be aware of. I say to you, I definitely get annoyed while people consider worries that they just do not know about. You managed to hit the nail upon the top and also defined out the whole thing without having side-effects , people can take a signal. Will likely be back to get more. Thanks
This is really interesting, You’re a very skilled blogger. I have joined your rss feed and look forward to seeking more of your fantastic post. Also, I’ve shared your website in my social networks!
Hello There. I found your blog using msn. This is an extremely well written article. I�ll make sure to bookmark it and return to read more of your useful info. Thanks for the post. I will certainly comeback.
I loved as much as you will receive carried out right here. The sketch is tasteful, your authored subject matter stylish. nonetheless, you command get bought an impatience over that you wish be delivering the following. unwell unquestionably come further formerly again since exactly the same nearly a lot often inside case you shield this hike.
Hi, i think that i saw you visited my weblog thus i came to �return the favor�.I am attempting to find things to improve my web site!I suppose its ok to use some of your ideas!!
Simply wish to say your article is as amazing. The clearness in your post is simply great and i can assume you are an expert on this subject. Well with your permission allow me to grab your RSS feed to keep up to date with forthcoming post. Thanks a million and please continue the enjoyable work.
Its like you read my mind! You appear to know so much about this, like you wrote the book in it or something. I think that you could do with a few pics to drive the message home a bit, but other than that, this is excellent blog. An excellent read. I’ll certainly be back.
Thank you for the good writeup. It in fact was a amusement account it. Look advanced to far added agreeable from you! By the way, how could we communicate?
Hello there, You’ve done an incredible job. I will certainly digg it and personally suggest to my friends. I am confident they will be benefited from this website.
Wonderful beat ! I wish to apprentice while you amend your site, how can i subscribe for a blog site? The account aided me a acceptable deal. I had been tiny bit acquainted of this your broadcast offered bright clear concept
I’m extremely impressed with your writing skills and also with the layout on your weblog. Is this a paid theme or did you modify it yourself? Anyway keep up the nice quality writing, it�s rare to see a great blog like this one these days..
Attractive section of content. I just stumbled upon your blog and in accession capital to assert that I get actually enjoyed account your blog posts. Anyway I�ll be subscribing to your feeds and even I achievement you access consistently rapidly.
My brother recommended I might like this website. He was entirely right. This post truly made my day. You cann’t imagine simply how much time I had spent for this info! Thanks!
I do not even know how I ended up here, but I thought this post was good. I don’t know who you are but certainly you are going to a famous blogger if you are not already 😉 Cheers!
Heya i am for the first time here. I came across this board and I find It truly useful & it helped me out a lot. I hope to give something back and help others like you aided me.
I was recommended this blog by my cousin. I’m not sure whether this post is written by him as no one else know such detailed about my difficulty. You are wonderful! Thanks!
Great blog here! Also your web site loads up fast! What host are you using? Can I get your affiliate link to your host? I wish my site loaded up as quickly as yours lol
Wow, awesome blog layout! How long have you been blogging for? you made blogging look easy. The overall look of your site is wonderful, let alone the content!
I�m not sure where you are getting your information, but good topic. I needs to spend some time learning more or understanding more. Thanks for magnificent information I was looking for this info for my mission.
You actually make it seem so easy with your presentation but I find this topic to be really something which I think I would never understand. It seems too complex and extremely broad for me. I am looking forward for your next post, I�ll try to get the hang of it!
I’ve been surfing online more than three hours today, yet I never found any interesting article like yours. It�s pretty worth enough for me. Personally, if all webmasters and bloggers made good content as you did, the web will be much more useful than ever before.
I think this is a real great article post.Much thanks again. Want more.