February 23, 2011

Innovating with market research

Filed under: marketing, research — Rakesh Agrawal @ 1:23 pm

One of the things that people often ask me is how does someone goes about creating innovative product concepts. What are the research methods and tools to use?

One of the most powerful tools is just watching what customers do and identifying problems and opportunities for improvement. Many customers have a hard time articulating needs. Or they will incorrectly estimate their needs. Every time I’ve done survey research asking about local tasks, 30% of consumers claim they would check movie showtimes daily. No one does that!

The solutions consumers articulate are also typically much more limited than what’s possible.

A great example of that are the new apps that let you take a picture of a check to deposit into your account. (USAA had one of the earliest ones; Chase has been advertising theirs heavily.)

If you were to ask customers how you could improve the check deposit experience you’d likely get answers like:

  • Make the lines at the bank shorter.
  • Have an express lane for simple transactions.
  • At the ATM, put a pen on a chain so I don’t have to find a pen to fill out the deposit envelope.
  • Get rid of the deposit envelope and let me put checks right in the ATM. (Newer ATMs now allow you to do this.)

But you’d be unlikely to have customers tell you “let me take a picture of the check with my iPhone and put the money in my account without the bank ever holding the check.”

They don’t know enough about banking regulations to know that this is even possible.

Everyone I know who has seen the iPhone check deposit thinks it’s magic. That’s how truly innovative products should feel.

About these ads

February 17, 2011

I’m the king of my castle, not a serf

Filed under: journalism, social networking — Rakesh Agrawal @ 11:14 am

David Carr recently wrote in The New York Times about the devaluing of professional content. He contends that sites like the Huffington Post, Facebook and Twitter have turned Americans into serfs.

Carr makes the mistake of associating low-cost and no-cost content with low-quality content. Yes, there is a lot of shit content out there created for free.  But some of the best content out there is also created for free.

Someone should ask Carr: why do sources talk to reporters for free? Are those sources also serfs?

The reasons people talk to reporters are some of the same reasons they participate in social media.

A little background on me before I continue: I went to the top journalism school in the country and launched one of the first online newspapers. I quickly left the business because it was too hard to change mindsets.

Carr talks about the bifurcation into two streams: professional and amateur. There are really more than that. I’ll simplify and say amateur, professional reporter and expert. The middle stream (where many reporters sit) is being squeezed on both sides.

In the amateur stream, you’ve got the likes of Demand Media and Associated Content with amateurs who write low-quality content for small amounts of money. This content has some monetary value, usually in the form of AdSense ads.

You also have people creating amateur content purely to share among their friends. These are the jokes, status updates, photos, etc. that litter Facebook and Twitter.  This content has pretty much no monetary value. The value is in the sharing with people that you care about. Unfortunately for the mainstream media, there are only 24 hours in a day. If I spend 15 minutes looking at pictures from a friend’s trip to Barcelona, that’s 15 minutes I’m not spending watching TV or reading the newspaper.

In the expert stream, you’ve got people who write high-quality content for a variety of other reasons.

At the risk of being immodest, I write high-quality content on Quora and my blogs about topics I’m passionate about. (The future of media being one of them.) I do this because I enjoy the intellectual challenge. I enjoy meeting like-minded people. It helps me flesh out my own thinking. It helps me with my business. And I do this for free because I’m (usually) gainfully employed and can afford to write for free.

My analysis of search, mobile, local and travel-related topics will be much better than what a NYT journalist will put out because I live and breathe this stuff. And I can create that content in a fraction of the time.

There are many people like me — and many who are much more prominent. Danny Sullivan writes about search. Fred Wilson writes about startups and entrepreneurship. Robert Reich writes about economics and politics.

These are people at the top of their fields. These are the same people that reporters seek out when they’re looking for experts to quote on a topic. I’ve never been able to get journalists to reconcile the fact that they bitch about the low quality of blogosphere and then often quote expert bloggers in their stories.

The quality of the average piece of content in the blogosphere is less than the quality of the average piece of content in The New York Times. But there are places to go for people who want the higher quality content.

This answer on Quora about why Borders in bankruptcy and Barnes & Noble isn’t is way better than anything I’ve seen in the mainstream media about Borders. It also points out one big benefit of social media versus mainstream media for experts: you don’t have to worry about about an ill-informed or malicious reporter taking what you said out of context or misinterpreting. What you write goes out as you wrote it.

There are more experts in the blogosphere than there are in newsrooms. That’s just a matter of numbers. Few people in newsrooms have formal training, expertise or experience in the areas they write cover. CNN’s Dr. Sanjay Gupta is a rare exception. In addition to covering medical topics for CNN, he’s a practicing neurosurgeon. (Oddly, TV tends to have more topic experts than newspapers.)

The highest quality content will come from people who are deeply passionate about a space and live it, not from someone who writes about it for a paycheck. Yes, this brings up the issue of bias. But the media “solve” this issue by just quoting two people with opposite views and leaving the reader to decide. That’s no solution.

Many (though by no means all) journalists are stuck in the middle ground, where they’re paid a lot more than the amateurs but don’t have the knowledge or skills of the experts

February 16, 2011

Watson vs. Google

Filed under: google, search — Rakesh Agrawal @ 2:50 am

Jeopardy! contestants Ken Jennings and Brad Rutter got a spanking the likes of which they’ve never seen by Watson, an IBM supercomputer, last night.

How would Watson do against that other font of knowledge — Google? Watson is optimized to understand human language, which can be full of ambiguity, wordplay, sarcasm and puns. Google makes some accommodations for language, but keywords do the heavy lifting.

Google did both much better than I expected and about as well as I expected. It did much better in the sense that I expected Watson to runaway from Google as much as it ran away from Jennings and Rutter. It did as well as I expected in that before I entered each clue into Google, I predicted how well Google would do. For example, I knew the Beatles lyrics category would be a piece of cake for Google. Those predictions were generally correct.

Watson had the correct response for 79% of the clues. (This includes clues for which Watson didn’t buzz in.) Google had the correct response in the first position 56% of the time and in the first 10 positions 79% of the time.

Google and Watson tended to struggle on the same clues. Both had trouble on Final Frontiers, Alternate Meanings and The Art of the Steal categories.

The only clue that Watson got wrong and Google nailed was “The first modern crossword puzzle is published & Oreo cookies are introduced” in the category Name The Decade. Google’s answer was found on this page that provides explanations of the New York Times crossword. If Watson could have heard Jennings’ incorrect answer of “1920s,” his backup answer of “1910s” would have been correct.

Finding a document with the right answer is one thing. Synthesizing that information into an answer is another significant challenge. Google only partially does that. I scored Google based on the snippet returned. While it wasn’t generally able to provide the precision of Watson, it did provide the right parts of the document with the answer.

As for Final Jeopardy!, neither Watson or Google got it right.

The clue in U.S. Cities was “Its largest airport is named for a World War II hero; its second largest, for a World War II battle.” Watson hesitantly answered Toronto. Google’s best guess would’ve been Arizona. Even geographically challenged Americans would know that neither answer fit the parameters of the category. (I hope.)

Oddly, that’s a relatively easy clue for humans. Think of U.S. cities, narrow it to the big ones, narrow that to ones that have multiple airports and then go through the list. Kennedy… Dulles… LAX… SFO… Hobby… Love…

Mmmmm…. deep dish.

A few thoughts on search from this exercise:

  • Google still has serious problems with content duplication. One query generated seven pages of essentially the same AP story about Jeopardy!
  • Content farms got in the way of the best answer a few times.
  • Wikipedia frequently provided the correct answer.
  • The day that Google can synthesize answers to queries is closer than I thought. Bad news for publishers.

Also see Danny Sullivan’s great analysis of the underlying technologies and the differences between natural language processing and keyword search.

Jeopardy! Watson vs. Google, Game 1 analysis

Jeopardy! Watson vs. Google, Game 1 comparison. Each cell represents a question. Grid position maps to the position on the Jeopardy! game board; you can see the clues and responses at the Jeopardy! Archive. The colors indicate Watson's performance; the numbers and "X"s indicate Google's performance. Watson is only given credit for answers in its #1 position. If you give Watson credit for having the answer in any of its 3 positions, it would pick up 6 more questions.


  • I used a private browsing session to avoid any influence from my search history.
  • I ran the exact text of the clue (as provided on the Jeopardy! Archive) through Google.
  • I excluded all results that were specific to the Jeopardy! match.
  • I looked for the text of the answer on the search results page, including the result title and snippet.
  • If the correct response appeared in the title or snippet, Google got credit. For the Name the Decade category, I gave Google credit if a year appeared that was in the correct decade. That would not be a correct response for the game, but I felt it was a better comparison for these purposes.
  • Search results change based on time, new content published, geography and other factors. You may not be able to duplicate these results. The optimal tests would be done on Google’s index prior to the airing of these episodes.

Disclosure: I have several good friends who work at Google and went to high school with co-founder and CEO Larry Page.

February 15, 2011

Why Apple is right on subscriptions

Filed under: apple, ipad, journalism, newspapers, publishing — Rakesh Agrawal @ 3:08 pm

Apple has built an incredibly valuable distribution platform. It’s perfectly reasonable for them to say “if you make money using our platform, we want a piece of it.”

Distribution costs money. Facebook charges developers on its platform 30% for using Facebook Credits. It’s not uncommon for distributors to take significantly more than Apple is taking. I’ve done deals where the other company does all of the heavy lifting and my company received more than half the revenue for bringing in customers. Google pays AdSense distributors 68% of the revenue it earns off content network ads. And that’s for no-name distributors. That number can be much higher for branded players.

Groupon has local businesses (which have hard costs for each person served) discount their product or service 50%, then takes 50% of that and sticks the business with 3% for processing. Merchants net 22% of what they would normally charge.

Retail gross margins are generally much higher than 30%. Would anyone argue that Apple should sell Microsoft Office in Apple stores at cost?

A seamless experience can benefit all parties. As a consumer, a key appeal of the App Store is that it’s seamless. I recently purchased two wonderful coffee table books on Vincent Van Gogh and Monet. These were quick and easy purchases. But I’ve resisted buying other products because I’ve just never been motivated enough to fill out all of my information again.

Game publishers such as Rovio (of Angry Birds) fame have been paying 30% for a long time. Apple is essentially closing a loophole that allowed some publishers to free ride on its platform.

We are still very early in the tablet content race. As popular as the iPad is, only a small portion of the population has one. If publishers believe that their content is so valuable and object to Apple’s terms, they should shun the iPad and develop for platforms with lower distribution costs. Heck, a brand name publisher like The New York Times could offer a platform provider an exclusive partnership in exchange for lower fees or even upfront payments or marketing support.

If their content is really that valuable, fewer people will buy iPads and Apple will be forced to lower its distribution fee. That’s how the free market works.

The sad reality is that publishers don’t have that kind of market power. If they opt out of iPad, it’s more likely that they’ll lose. That’s not Apple’s fault.

February 2, 2011

The Daily is a solid effort facing huge challenges

Filed under: apple, ipad, journalism, media, newspapers — Rakesh Agrawal @ 7:50 pm
Feature story on prison inmates making toys in The Daily.

This feature story on prison inmates making toys in The Daily looked promising; it turned out to be only a short video.

Today we saw the unveiling of Rupert Murdoch’s new iPad-based newspaper, The Daily.

It looks very different from apps from the Washington Post, New York Times, Wall Street Journal and USA Today. The impression is distinctly more of a magazine than a newspaper. My first reaction was “It sure is purty.”

While other apps, like the Journal’s and the Post’s, have bizarre navigation modes, navigating The Daily feels very natural. (With the exception of occasional stutters, likely caused by the app’s heavy use of graphics and animation.) Tabs are readily available to flip among sections.

The first issue was clearly designed as a showcase with numerous interactive elements such as user polls, timelines and quizzes. Multimedia elements include extensive use of photos, videos and even a gratuitous 360-degree photo to illustrate a story about Venice sinking into the sea. (Not exactly breaking news there.) A Super Bowl feature included an animation that explains a shovel pass. It would’ve been more useful if it also included a video.

The content is of mixed quality, though much of it is has the quality and depth of Parade Magazine. (Much of it is wire copy.) A feature on inmates in Louisiana making toys for kids with a “only in The Daily” starburst looked promising. It turned out just to be a short video clip.

There are print design conventions that some magazines use, such as two-page spreads. While that can be somewhat excused in shovelware magazines exported from inDesign, it’s hard to justify for a publication specifically designed for a tablet.

Fashion news in The Daily

I will never be interested in fashion news. I don't need to see it.

The Daily integrates with social networks, but it is clumsy. A feature on Rihanna has her Tweet stream embedded. Tweeting a story defaults to the oh-so-catchy “Check out this article from The Daily” instead of something that might actually inspire a click. The Facebook equivalent contains some information about the article, but only after generic text and HTML that Facebook doesn’t render.

The Daily suffers from several significant issues:

  • The content is vapid. It’s as if someone took a look at USA Today and said, “Whoa! This is too intellectual.”
  • It contains a lot of crap I don’t want. I’m not interested in women’s fashion or celebrity gossip. On the other hand, there’s not enough tech or business news. I’m not convinced that in 2011 it makes sense to program for an abstract general audience that doesn’t exist. News consumption is increasingly driven by friends and colleagues. The New York Times or the Post, with their long histories, have a chance (albeit slim) of using their editorial voices. Creating one from scratch seems like a Herculean challenge.
  • It’s not timely. Although they claim that there will be more frequent updates for big events, the bulk of the content will be updated only once a day. In 2011, The Daily might as well be The Fortnightly.
  • There is no interaction with reporters. Yes, reporters get things wrong. Just this week, David Pogue had an error in one of his columns. I tweeted the error to him, he responded and it was fixed in a few hours. The Daily doesn’t provide email addresses or Twitter handles of its contributors. There are no bios. These are things that most newspapers got right several years ago.
  • There is no way to dig deep. A story about data consumption on cell phones caught my eye. But it was one paragraph. Clicking on it did nothing. Even 140-character Tweets often contain links. A movie review for Cold Weather made me interested enough to want to know when it was playing. Sorry, you can’t do that here.
  • You can’t search. I wanted to tweet a story about Quora that I’d read. But there wasn’t an easy way for me to find it. I had to flip through every page, just like I would in a real newspaper. That’s ridiculous.
  • Update: Exploding content. Can’t read the news today? Too bad, it goes away tomorrow. My hard work on yesterday’s crossword was wiped away when I launched the app this morning. With physical newspapers you can at least keep things around and until you want to get rid of them.
Football animation.

This animation explains a shovel pass.

Despite its flaws, the bottom line is that The Daily is the best incarnation of an online newspaper I’ve seen. The question is how big is that market?

Estimates are that Murdoch is spending $500,000 a week for The Daily, or an annual budget of $26 million. While that’s tiny compared to traditional publishers, it’s gargantuan compared with funding that tech startups receive. Then there’s the $30 million already spent to get to launch.

Based on the proposed subscription price of $40/year and subtracting Apple’s cut, the venture would need to have about 930,000 subscribers to break even. As a print publication, that would make it the third largest paper in the country, behind the Journal and USA Today. Even if you assume that The Daily could make as much per user on advertising as it does on subscriptions, that’s 465,000 subs to break even. (For comparison, this assumption would also mean that The Daily makes more per user on advertising than the most successful Internet advertising company — Google.)

Those are huge numbers and I’m very skeptical that The Daily can do it.

Disclosure: I worked for from 1998-1999.

See also: Why iPad magazines aren’t selling well

The Silver is the New Black Theme. Blog at


Get every new post delivered to your Inbox.

Join 227 other followers

%d bloggers like this: