How Twitter Could Bring Search Up to Speed

Continued from page 1

An upswing in the number of marketers and companies that have joined Twitter over the past year also reflects big interest in using Twitter to tap into the online trends. Sullivan admits that right now the average Twitter user isn't turning to the service for the majority of its searches, but he says that marketers who want to track the sentiment of their brands and products are finding it an indispensable tool. "Is it essential to search Twitter because you're a brand manager?" asks Sullivan. "Yeah, it's becoming very important." He notes that outside search engines, such as Google, cannot easily go through Twitter's data because it is not releasing it. "If I were over at Google, I'd definitely be concerned about that," he says.

Meanwhile, Twitter is clearly thinking about ways to better mine its users' tweets. When you search Twitter, its search engine looks for keywords in the most recent tweets, explains Biz Stone, one of the company's cofounders. The results are then ranked based on the time when they were posted; for some popular topics, this can mean just seconds ago. The search technology used for this was developed at a startup called Summize, which Twitter acquired last July. Until recently, Twitter users needed to go to the address search.twitter.com to use the technology: a link was buried at the bottom of a user's homepage. Twitter is now testing a "more integrated" search box at the top of the page for a small number of users, says Stone.

As simple as it sounds, dealing with the volume of messages posted on Twitter in real time poses a big challenge, according to Stone. "Our biggest technical challenge right now is not that exciting: it's just scaling from an operations and engineering standpoint," he says. In other words, engineers have to make sure that the system doesn't crash under the weight of all the messages that Twitter handles.

Summize was given access to a higher volume of tweets than most third-party developers. In the early days of Twitter, it wasn't a problem to let developers use such a large volume of tweets, but as the service has grown, the application programming interface (API) that allows for such access "requires special design and attention," Stone says. Engineers at Twitter have also been working on a new, more scalable solution. "We'll need to be thoughtful about how, when, and who we partner with to test this technology when it's ready," he adds.

When asked whether Twitter and Google could work together to build a real-time search engine, Stone is a bit more optimistic. "We're huge fans of Google," he says. "And we'd be delighted to partner or work with them in the future."