jump to navigation

Wisdom of crowds, or just crowdiness of crowds? May 1, 2007

Posted by jeremyliew in Consumer internet, gaming, social media, social networks, user generated content, web 2.0.
trackback

On April 15th, 2007 an excellent article in the NY Times magazine asked “Is Justin Timberlake the product of cumulative advantage?“. It describes an experiment in which a group of unknown songs was exposed to different sets of people who were able to listen, rate and download them. One group of people had no exposure to what songs others were listening to, rating and downloading. The other eight groups were exposed to feedback on what others in their group (and only their group) were doing. Here is the conclusion:

… if people know what they like regardless of what … other people like, the most successful songs should draw about the same … market share in both the independent and social-influence conditions … And … the “best” ones … should become hits in all social-influence worlds.

… we found… exactly the opposite. In all the social-influence worlds, the most popular songs were much more popular … than in the independent condition. At the same time, however, the particular songs that became hits were different in different worlds… Introducing social influence into human decision making, in other words, didn’t just make the hits bigger; it also made them more unpredictable.

… intrinsic “quality,” which we measured in terms of a song’s popularity in the independent condition, did help to explain success in the social-influence condition. … But the impact of a listener’s own reactions is easily overwhelmed by his or her reactions to others.

… Because the long-run success of a song depends so sensitively on the decisions of a few early-arriving individuals, whose choices are subsequently amplified and eventually locked in by the cumulative-advantage process, and because the particular individuals who play this important role are chosen randomly and may make different decisions from one moment to the next, the resulting unpredictability is inherent to the nature of the market.

(This is heavily edited – read the whole thing. )

This is fascinating stuff.

Jordan Schwartz and Tim O’Reilly point to this as an example of the “Hive Mind”:

You can see these same processes at work in social bookmarking and ranking services like Digg and Del.icio.us. Individual users rate a site as being interesting, causing other users to visit it and, in turn, assess whether the site is interesting enough to rate it as interesting. Over time, certain sites gain momentum and rise to the top of the heap, even though most individuals only ever see a small fraction of the options.

There is a bit of an implication that the BEST sites rise to the top due to these social processes – a sort of “Wisdom of the Crowds” effect.

Josh Porter at Bodarko draws the opposite conclusion, that

This result has huge implications for all social web sites, especially those that show aggregate data. Digg, for example, shows aggregate data everywhere on the site. This experiment, in addition to several other issues that I wrote about in Digg’s Design Dilemma, suggest that the results there are socially influenced to such an extent that it would be hard indeed to know where the quality lies…

Others have posted in the past about why the Wisdom of Crowds fails on Digg.

Its worth going back to Surowiecki’s book to understand why. Summarizing Kottke:

In order for a crowd to be smart … it needs to satisfy four conditions:

1. Diversity. A group with many different points of view will make better decisions than one where everyone knows the same information.

2. Independence. “People’s opinions are not determined by those around them.”

3. Decentralization. “Power does not fully reside in one central location, and many of the important decisions are made by individuals based on their own local and specific knowledge rather than by an omniscient or farseeing planner.”

4. Aggregation. You need some way of determining the group’s answer from the individual (i.e. unbiased) responses of its members..

Digg fails on all four of these criteria; it is not a completely diverse group of users, Digg intrinsically influences its users’ opinions, there are certain Digger’s who have more power to push a story to the front page than others, and Digging does not occur in a vacuum.

Harvard Business review has another explanation for why crowds don’t always identify the best solutions by way of the Marquis de Condorcet, a 17th Century mathematician:

Groups will do better than individuals in choosing a correct answer, and big groups better than little ones, as long as two conditions are met: the majority response “wins,” and each person is more likely than not to be correct.

(Italics mine)

In a large enough sample set of options, the likelihood that a person can choose the “best” article or “best” song rapidly drops below 50%, so this second criteria fails.

Ultimately, there are a couple of key takeaways for people who run social media sites:

1. If you’re trying to iterate towards a “best answer” then keep feedback loops to a minimum, at least before users “vote” on their own. (e.g. Hotornot, espgame)
2. If you’re trying to create “hits” out of some of your content (and don’t care if it’s the “most worthy” content – you only care that they are hits), then display feedback and popularity constantly, as this will effect user behavior and exacerbate the size of the hits (e.g. Youtube, Digg, American Idol?
3. If you want to “guide” user behavior in a certain direction, provide feedback that validates or shows the popularity of that behavior. This is consistent with my prior post on game mechanics applied to social media: keeping score.

Can you think of more examples of these three best practices? If so, please add them in comments.

Comments»

1. Uri L. - May 1, 2007

In a way, Last.fm radio player implements practice 1 – as u can simply mark if u like a song or not, without seeing any community stats related to it. So it’s kinda leaned into an unbiased voting process…

I wonder what about all the answers services…which lead the mission to provide the most accurate answer utilizing wisdom of the crowds. Looking at some of them, I see that they don’t follow practice 1 that much.

2. wrwebb - May 2, 2007

Your timing with this post is ridiculously good. Digg is complete chaos right now.

I put a post on my blog giving some background:

http://www.blog.robwebb2k.com/2007/05/digg-implosion.html

or you can just go straight there to see the mayhem. Unclear how this will impact Digg in the long run. Definitely having a huge impact in the short run:

http://www.digg.com/

3. Dave McClure - May 2, 2007

interesting post jeremy. good stuff :)

4. stephane lee - May 2, 2007

Very interesting post, Jeremy.

I think what practice you choose depends on the “filtering” power that you want the crowd to have.

Do you want the crowd to produce a note (8/10,…) or do you want them to filter things, sort them out to get a concensus, even if it’s not trustworthy ?

Both ways are valuable, and I think that combining the two would make sense :
Phase 1/ Let a restricted community share and sort content
Phase 2/ Ask individual people to rate the top results from Phase 1

What do you think ?

PS : As for Digg, I think the ratio – number of diggs / number of clicks – could be a balancing act. Maybe they already do ?

5. karim tahwi - May 3, 2007

Great post Jeremy.

I think you have hit this on the nose. The issue is herding versus crowd wisdom. Digg starts as wisdom of the crowds but crosses over to herding when the number of independents gets ambushed by the dependents. Ultimately, Digg is a herding platform because most people are consuming selection (or manipulating selection) rather than making it.

We (www.my-currency.com) are using prediction markets and other social apps to aggregate future values of properties. We do not close information loops but actually encourage sharing so as to improve the aggregate knowledge in the community. Although we risk diluting independence and diversity, we feel markets manage this efficiently by allowing people to vote a problem up or down to varying degrees. We also measure performance over time and assign reputation scoring to individuals so their market impact is more substantial as they are proving to have knowledge. So our model gets smarter – as long as we continue to sponge new knowledge participants.

6. More on using crowds to your advantage « Lightspeed Venture Partners Blog - May 3, 2007

[…] More on using crowds to your advantage May 3, 2007 Posted by jeremyliew in Uncategorized. trackback On April 29th, 2007, the Boston Globe published an interesting article in praise of peer pressure. Coming shortly on the heels of the NY Times article about cumulative advantage, it gives a separate set of examples on how to use the public display of popularity to shape user behavior, a topic I covered in a recent post. […]

7. Pascal Van Hecke - May 3, 2007

Great post, as always – but the first “or” should be “of” in your title :-)

8. Viral marketing, randomness and the difficulty of controlling growth in social media « Lightspeed Venture Partners Blog - September 13, 2007

[…] finding in May, how you don’t necessarily get the wisdom of crowds, but sometimes just the crowdiness of crowds. It was based on a NY Time’s article that showed how randomness can have a big impact on the […]

9. embed - September 13, 2007

Good stuff Jeremy. I continue to be impressed.

10. Website Hits Are Shoddy Data, Now More Than Ever « - April 16, 2008

[…] acting separately, consecutively, and/or concurrently: the rich get richer (e.g., googlearchy, cumulative advantage), and popularity is incredibly volatile (e.g., a single post/song/video goes viral, a blog’s […]

11. Facebook selling digital gifts at a $35m run rate « Lightspeed Venture Partners Blog - September 2, 2008

[…] from the group of the first 20 gifts. This points to the self reinforcing nature of popularity (the crowdiness of crowds rather than the wisdom of crowds) when popularity data is made […]

12. How to use popularity lists to influence your users behavior « Lightspeed Venture Partners Blog - May 20, 2009

[…] product management, social media, user generated content. trackback Two years ago I asked if crowds generated wisdom, or simply crowdiness. Today’s WSJ has a really interesting article on the same topic, concluding that popularity […]

13. How to use popularity lists to influence your users behavior | Bookmarks - May 20, 2009

[…] $(function(){ $("#startbox").html("Two years ago I asked if crowds generated wisdom, or simply crowdiness. Today’s WSJ has a really interesting article on the same topic, concluding that popularity is […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 35 other followers

%d bloggers like this: