reDesign

January 23, 2012

Twitter and Google are both responsible for you not being able to search tweets

Filed under: facebook, google, search, twitter — Rakesh Agrawal @ 7:45 pm

Chris Dixon ignited a firestorm on his blog when he said it was Twitter’s fault that Google doesn’t index tweets. It’s the fault of both parties, really. Neither has the moral high ground.

Twitter does not block Google from crawling their site. Google does crawl Twitter and index tweets. You can see this by going to:

https://www.google.com/search?sourceid=chrome&ie=UTF-8&q=site%3Atwitter.com%2Frakeshlobster

You will see some of my tweets in Google’s search index.

Right now, Google does this by crawling Twitter, just like it does for everything else. But crawling takes resources (bandwidth, compute cycles). For a site that is updated as much as Twitter, it would take a lot of resources to keep a reasonably fresh index.

This also has an impact on the crawled site. Twitter has enough stability problems without Google increasing its crawl rate. If Google cranked it up, it would make Twitter less stable.

Another alternative is for Twitter to provide a feed of new tweets to Google. This is what they do for bing and what they previously did for Google, before their agreement ended. This would take less computing power and bandwidth on both sides.

Twitter used to charge Google for this. It was a rare case where Google paid for content. According to Google, Twitter decided to stop licensing this feed to them.

So there are two ways to get more tweets into Google:

  • Twitter could provide a feed (either for free or a mutually agreed upon fee).
  • Google could increase it’s crawl frequency.

A third, more complicated way to solve this is for Twitter to provide Google a real-time query API. This would require that Twitter build a decent search engine first.

There’s a broader question here, that Yelp has raised a few times, including in the senate hearings: should Google be allowed to use its dominance in Web search to make its way into other spaces like local and social?

If  social search was not part of Google’s dominant Web search, no one would care whether it included Twitter or Facebook results because no one would use it. But because Search plus Your World is so prominent in Google search, Twitter and Facebook care.

See my series on Google and antitrust for a deeper exploration of this.

About these ads

5 Comments »

  1. [...] results — thanks to what it sees as favoritism of Google’s own Google+ network. But this particular brouhaha is only the latest manifestation of a much deeper problem between the two, like a fight over the toothpaste, or who did the laundry last. The reality is that both sides need [...]

    Pingback by Why Google and Twitter need to kiss and make up — Tech News and Analysis — January 24, 2012 @ 10:36 am

  2. Twitter Should Open Up Its API —- To Google http://bit.ly/zLxkkx

    Comment by Paramendra Kumar Bhagat — January 24, 2012 @ 2:53 pm

  3. Twitter Should Open Up Its API ?

    Comment by orange business phones — January 28, 2012 @ 1:03 am

  4. [...] in those results — thanks to what it sees as favoritism of Google’s own Google+ network. But this particular brouhaha is only the latest manifestation of a much deeper problem between the two, like a fight over the toothpaste, or who did the laundry last. The reality is that both sides need [...]

    Pingback by Why Google and Twitter need to kiss and make up – - Tech News AggregatorTech News Aggregator — February 2, 2012 @ 3:27 am

  5. Купить Spajs у Алексея Хабарова. Skype: spice27rus

    Купить, Spajs в городе Habarovsk.

    Контакты для связи: (Постоянно работает Skype: spice27rus) Осторожно!!! Возможны различные фейки.

    Mail: spice27rus@mail.ru (Основная почта.)
    Skype: spice27rus

    Comment by Spice Habarovsk — October 11, 2013 @ 3:43 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Silver is the New Black Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 134 other followers

%d bloggers like this: