Wisdom of crowds, or just crowdiness of crowds?

On April 15th, 2007 an excellent article in the NY Times magazine asked “Is Justin Timberlake the product of cumulative advantage?“. It describes an experiment in which a group of unknown songs was exposed to different sets of people who were able to listen, rate and download them. One group of people had no exposure to what songs others were listening to, rating and downloading. The other eight groups were exposed to feedback on what others in their group (and only their group) were doing. Here is the conclusion:

… if people know what they like regardless of what … other people like, the most successful songs should draw about the same … market share in both the independent and social-influence conditions … And … the “best” ones … should become hits in all social-influence worlds.

… we found… exactly the opposite. In all the social-influence worlds, the most popular songs were much more popular … than in the independent condition. At the same time, however, the particular songs that became hits were different in different worlds… Introducing social influence into human decision making, in other words, didn’t just make the hits bigger; it also made them more unpredictable.

… intrinsic “quality,” which we measured in terms of a song’s popularity in the independent condition, did help to explain success in the social-influence condition. … But the impact of a listener’s own reactions is easily overwhelmed by his or her reactions to others.

… Because the long-run success of a song depends so sensitively on the decisions of a few early-arriving individuals, whose choices are subsequently amplified and eventually locked in by the cumulative-advantage process, and because the particular individuals who play this important role are chosen randomly and may make different decisions from one moment to the next, the resulting unpredictability is inherent to the nature of the market.

(This is heavily edited – read the whole thing. )

This is fascinating stuff.

Jordan Schwartz and Tim O’Reilly point to this as an example of the “Hive Mind”:

You can see these same processes at work in social bookmarking and ranking services like Digg and Del.icio.us. Individual users rate a site as being interesting, causing other users to visit it and, in turn, assess whether the site is interesting enough to rate it as interesting. Over time, certain sites gain momentum and rise to the top of the heap, even though most individuals only ever see a small fraction of the options.

There is a bit of an implication that the BEST sites rise to the top due to these social processes – a sort of “Wisdom of the Crowds” effect.

Josh Porter at Bodarko draws the opposite conclusion, that

This result has huge implications for all social web sites, especially those that show aggregate data. Digg, for example, shows aggregate data everywhere on the site. This experiment, in addition to several other issues that I wrote about in Digg’s Design Dilemma, suggest that the results there are socially influenced to such an extent that it would be hard indeed to know where the quality lies…

Others have posted in the past about why the Wisdom of Crowds fails on Digg.

Its worth going back to Surowiecki’s book to understand why. Summarizing Kottke:

In order for a crowd to be smart … it needs to satisfy four conditions:

1. Diversity. A group with many different points of view will make better decisions than one where everyone knows the same information.

2. Independence. “People’s opinions are not determined by those around them.”

3. Decentralization. “Power does not fully reside in one central location, and many of the important decisions are made by individuals based on their own local and specific knowledge rather than by an omniscient or farseeing planner.”

4. Aggregation. You need some way of determining the group’s answer from the individual (i.e. unbiased) responses of its members..

Digg fails on all four of these criteria; it is not a completely diverse group of users, Digg intrinsically influences its users’ opinions, there are certain Digger’s who have more power to push a story to the front page than others, and Digging does not occur in a vacuum.

Harvard Business review has another explanation for why crowds don’t always identify the best solutions by way of the Marquis de Condorcet, a 17th Century mathematician:

Groups will do better than individuals in choosing a correct answer, and big groups better than little ones, as long as two conditions are met: the majority response “wins,” and each person is more likely than not to be correct.

(Italics mine)

In a large enough sample set of options, the likelihood that a person can choose the “best” article or “best” song rapidly drops below 50%, so this second criteria fails.

Ultimately, there are a couple of key takeaways for people who run social media sites:

1. If you’re trying to iterate towards a “best answer” then keep feedback loops to a minimum, at least before users “vote” on their own. (e.g. Hotornot, espgame)
2. If you’re trying to create “hits” out of some of your content (and don’t care if it’s the “most worthy” content – you only care that they are hits), then display feedback and popularity constantly, as this will effect user behavior and exacerbate the size of the hits (e.g. Youtube, Digg, American Idol?
3. If you want to “guide” user behavior in a certain direction, provide feedback that validates or shows the popularity of that behavior. This is consistent with my prior post on game mechanics applied to social media: keeping score.

Can you think of more examples of these three best practices? If so, please add them in comments.