Chris Dixon ignited a firestorm on his blog when he said it was Twitter’s fault that Google doesn’t index tweets. It’s the fault of both parties, really. Neither has the moral high ground.

Twitter does not block Google from crawling their site. Google does crawl Twitter and index tweets. You can see this by going to:

https://www.google.com/search?sourceid=chrome&ie=UTF-8&q=site%3Atwitter.com%2Frakeshlobster

You will see some of my tweets in Google’s search index.

Right now, Google does this by crawling Twitter, just like it does for everything else. But crawling takes resources (bandwidth, compute cycles). For a site that is updated as much as Twitter, it would take a lot of resources to keep a reasonably fresh index.

This also has an impact on the crawled site. Twitter has enough stability problems without Google increasing its crawl rate. If Google cranked it up, it would make Twitter less stable.

Another alternative is for Twitter to provide a feed of new tweets to Google. This is what they do for bing and what they previously did for Google, before their agreement ended. This would take less computing power and bandwidth on both sides.

Twitter used to charge Google for this. It was a rare case where Google paid for content. According to Google, Twitter decided to stop licensing this feed to them.

So there are two ways to get more tweets into Google:

• Twitter could provide a feed (either for free or a mutually agreed upon fee).
• Google could increase it’s crawl frequency.

A third, more complicated way to solve this is for Twitter to provide Google a real-time query API. This would require that Twitter build a decent search engine first.

There’s a broader question here, that Yelp has raised a few times, including in the senate hearings: should Google be allowed to use its dominance in Web search to make its way into other spaces like local and social?

If  social search was not part of Google’s dominant Web search, no one would care whether it included Twitter or Facebook results because no one would use it. But because Search plus Your World is so prominent in Google search, Twitter and Facebook care.

See my series on Google and antitrust for a deeper exploration of this.