jump to navigation

Hints, tips and cheats to better datamining June 17, 2008

Posted by jeremyliew in datamining.

As web based product development and game development both become more iterative, better datamining and analysis becomes more and more important. But the data generated by users behavior can be almost overwhelming. How should a startup think about getting the most insight and value from their data?

Anand Rajaraman is a co-founder of Kosmix, a Lightspeed portfolio company, and also teaches a datamining class at Stanford. He knows a thing or two about the subject, and he suggests that more data usually beats better algorithms:

Different student teams in my class adopted different approaches to the [Netflix challenge] problem, using both published algorithms and novel ideas. Of these, the results from two of the teams illustrate a broader point. Team A came up with a very sophisticated algorithm using the Netflix data. Team B used a very simple algorithm, but they added in additional data beyond the Netflix set: information about movie genres from the Internet Movie Database (IMDB). Guess which team did better?

Team B got much better results, close to the best results on the Netflix leaderboard!! I’m really happy for them, and they’re going to tune their algorithm and take a crack at the grand prize. But the bigger point is, adding more, independent data usually beats out designing ever-better algorithms to analyze an existing data set. I’m often suprised [sic] that many people in the business, and even in academia, don’t realize this.

Another fine illustration of this principle comes from Google. Most people think Google’s success is due to their brilliant algorithms, especially PageRank. In reality, the two big innovations that Larry and Sergey introduced, that really took search to the next level in 1998, were:

1. The recognition that hyperlinks were an important measure of popularity — a link to a webpage counts as a vote for it.
2. The use of anchortext (the text of hyperlinks) in the web index, giving it a weight close to the page title.

First generation search engines had used only the text of the web pages themselves. The addition of these two additional data sets — hyperlinks and anchortext — took Google’s search to the next level. The PageRank algorithm itself is a minor detail — any halfway decent algorithm that exploited this additional data would have produced roughly comparable results.

In a followup post, he notes that:

1. More data is usually better than more complex algorithms because complex algorithms don’t scale as well (computationally) and
2. More independent data is better than more of the same data, but if data was originally sparse, then more of the same data can help a lot too.

Mayank Bawa of Aster Data chimes in to say that running simple analysis over complete datasets is better than running more complex data over sampled datasets for two reasons:

1. The freedom of big data allows us to bring in related datasets that provide contextual richness.
2. Simple algorithms allow us to identify small nuances by leveraging contextual richness in the data.

In other words, since human behavior is complex, and some behavior crossmatches are rare, using a sample of data will cause some important but rare correlations to be lost into the noise.

He also points out that Google takes a similar approach to datamining.

This is good stuff.


1. Amy Jo Kim - June 18, 2008

great find — fascinating stuff — thanks for posting about this.

2. Parker Emmott - June 20, 2008

Great words of wisdom here. Thanks for sharing. But if more is better, this seems to leave the more important question unanswered… How does one know what data sets to focus on? If Google’s breakthrough came from identifying the critically relevant but previously ignored data sets of hyperlinks and anchortext, what guidelines can startups deduce for identifying the data sets that are critical to them? Interesting question…

3. dj Chang - June 21, 2008

I am a macroeconomist and this post is stupid.

1. Google search is a rich AI engine with huge numbers of simple rules. Generalizing this to data-mining wisdom is comparing Apples and Oranges.

2. If you seek a significant relationship among variables, you don’t need millions of data points. However, the relationship may be non-linear and require complex time-series corrections to reduce the noise.

Human behavior is complex. Millions of rules or sophisticated analytics – those are your options.

I know the rigid nature of data mining tools. As a result, behavior targeting has been a failure. Don’t let that be confused with true needs in data mining.

PS: I do appreciate the post from Saran Chari.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: