Tuesday, July 08, 2008

Digging a Smarter Crowd

Continued from page 1

By Erica Naone

Paul Lamere, a staff engineer on the Search Inside the Music recommendation project at Sun Microsystems, says that while it can be difficult to build recommendation engines that can handle vast quantities of information and calculations, the nature of Digg makes the problem a bit easier. He says that unlike systems such as Amazon's, in which the number of items in the database is constantly growing, Digg limits its recommendation engine to items that users selected within the past 30 days, which keeps the data store from getting too large. What's more, splitting recommendations by topic also turns out to help with scaling, since it reduces the amount of data that needs to be processed at once. Lamere notes, however, that by making recommendations based only on users, rather than on features of the articles themselves, there's a risk of driving diversity out of the system. "It's the rich-get-richer phenomenon," he says, adding that recommendation engines that factor in the characteristics of products or articles can balance popular items by bringing forward lesser-known items with similar qualities.

Although the Digg recommendation engine became available to users, in an experimental version, only a few days ago, Kast says that it's already having an effect on the way the site functions. "There's been a huge spike of digging activity on the site," he notes, "and substantial increases in the number of unique diggers." Kast says that the company hopes this will ultimately improve the quality of the website. If more users become more active in selecting stories early on, he explains, Digg's algorithms will have better statistics to work with when promoting stories to the front page.

John Riedl, a professor of computer science at the University of Minnesota who studies recommenders, says that Digg's entry into the field of recommendations is interesting because news has a very different character from e-commerce. While shopping sites are dealing with fads that play out over the course of weeks and months, news sites are dealing with fads that could pass in the space of a few hours. The time pressure, he says, makes it hard to come up with a system that can sort out stories that are both up-to-date and high quality. Riedl says that he sees Digg's move as part of the next step in changing how information reaches people. "I'd like to see information disseminated because it's the stuff that's most interesting to us individually, based on our tastes and our unique qualities as people," he says. "I don't know if Digg's nailed it yet, but I think it's an incredible opportunity."