Popularity – just random or science of success?

July 13, 2007

Nothing is more powerful than an idea whose time has come. The invisible hand of the market sets in motion a clear chain of causality:

  1. If you have a good idea (quality)
  2. experts/gatekeepers will recognize its potential (predictions based on experience)
  3. and as a result it will turn out to be a success (as peoples’ decision making is based on quality)

Textbook examples of inferior products winning out (QWERTY-keyboard, VHS, etc.) show that this is not always the case, as certain other factors come into play as well (network effects, lock-ins, etc.). These examples, however, are only mentioned as mere exceptions to the rule that the best will win. Is this really the case and does the rule really exist in general?

Do experts know anything?

In contrast to the amount of money spent on experts’ predictions, there is very little evidence that experts have any clue of what the future will hold. One reason could be that experts need a better predictive process to improve their forecasting. But as Nassim Taleb suggests in his book The Black Swan: The Impact of the Highly Improbable (excellent review by James Surowiecki) the very reason for their poor performance is that the job of forecasting is simply impossible. We assume that historical patterns enable us to extrapolate what will happen. In contrast to this assumption, Taleb argues, history tells us that the radical outliers outside the realm of regular expectations have had the biggest impact and that these outliers account for just about everything of significance around us. In a world of classic bell curve distributions (coined ‘Mediocristan’ by Taleb) most things happen close to the middle and are therefore relatively easy to predict. But then there are a few events that shape the world which happen outside the center of a bell curve (in ‘Extremistan’). They represent a break of what has come before and because of this very nature of those events experts fail in predicting them. They happen in every domain — bestsellers, technological innovations, stock market returns, etc. — and outweigh everything else. Extracting generizable stories from these events in hindsight can make us believe that they could have been predicted — which might be emotionally satisfying but is at the same time practically useless.

Does quality matter?

But since all media companies employ experts to assess the potential of a certain idea (music, movies, books, etc.), there must be certain criteria all can agree on to have an impact on the success. Maybe it’s quality? Let’s assume we are able to objectively identify the quality of one of these products — does it really matter?

According to a study by three Columbia sociologists quality’s impact on the popularity is very limited:

“In our study, … 14,000 participants … were asked to listen to, rate and, if they chose, download songs by bands they had never heard of. Some of the participants saw only the names of the songs and bands, while others also saw how many times the songs had been downloaded by previous participants. This second group – in what we called the “social influence” condition – was further split into eight parallel “worlds” such that participants could see the prior downloads of people only in their own world. …

In all the social-influence worlds, the most popular songs were much more popular (and the least popular songs were less popular) than in the independent condition. At the same time, however, the particular songs that became hits were different in different worlds, just as cumulative-advantage theory would predict. …

In fact, intrinsic “quality,” which we measured in terms of a song’s popularity in the independent condition, did help to explain success in the social-influence condition. …. But the impact of a listener’s own reactions is easily overwhelmed by his or her reactions to others. The song “Lockdown,” by 52metro, for example, ranked 26th out of 48 in quality; yet it was the No. 1 song in one social-influence world, and 40th in another. Overall, a song in the Top 5 in terms of quality had only a 50 percent chance of finishing in the Top 5 of success.”

Although a 50 percent chance for a quality song to end up in the Top 5 of success doesn’t sound too bad to me (even though the universe only consists of 48 songs), the impact of quality seems to be fairly limited and what becomes a hit seems to be determined largely by the previous choices of others, making it a random walk. Or as Cory Doctorow so tellingly coined the implications of power laws for media products “Content isn’t king. … Conversation is king. Content is just something to talk about”. The more people I have that I can talk with about a certain song, movie, or book the more relevant it is for me and the more likely I am to choose it based on the choice of others. Prove of this can be seen when a devastating book review leads to an uplift in sales almost by the same degree as raving reviews.

Nobody knows anything?

So if experts’ predictions aren’t worth much for finding the next hit and the choice of others matters just as much as the quality, how much can I then know in advance about the chance for a success? Unfortunately not very much.

Surowiecki offers some hope, however: by involving a diversified, independent, and decentralized crowd of people and aggregating their evaluations in advance of publishing we have a very good chance to come up with significantly better predictions. Or in Surowiecki’s reply to the famous assertion by screenwriter William Goldman: “Nobody knows anything.” “But everybody, it turns out, may know something.”

Advertisements

Wisdom of Crowds – or stupidity of the mob?

July 5, 2007

A couple of days ago I came across a review of Andrew Keen‘s book The Cult of the Amateur. As in Keen‘s (must read!) debate with Chris Anderson (podcast) he is making the point that today’s internet — Web 2.0 — is killing our culture because the traditional gatekeepers are being removed, opening cultural, economic, and political life up to amateurs. With a negative impact on quality, as the crowd would only be able to produce mediocrity. To quote Keen in his own words from his debate with Anderson:

“I still think that the wisdom that I value — the scarcity, to put it in economic terms — is not in the crowd, but in people with talent and experience, whether they exist in political life, in economic life or cultural life.”

To defend the pre-Web 2.0 state of affairs that Keen prefers two conditions would have to be true:

  1. The people in the gatekeeper positions are people with talent and experience, enabling them to discover all the other talent in an effective way and foster it efficiently
  2. The result of this process is output of high quality

Just focusing on publishing as one example of the cultural life this is clearly not the case (as previously discussed):

  • the traditional process of screening new authors and estimating the sales is clearly neither effective nor efficient: a lot of talent goes undiscovered, as the process of discovery is not only based on talent and not all talent is getting through; a lot of other talent is overestimated (at least from a sales point of view), resulting in return rates of about 40% on average
  • as everybody familiar with the bestseller lists can attest they are not really a beacon for literary excellence, since the publishing houses are not operating in an economical vacuum and therefore have to publish what sells — to the crowds, who are by Keen‘s terms responsible for mediocrity in the Web 2.0 environment; I argue that because economic rational dictates what gets published and what doesn’t today the same is already true in the pre-Web 2.0 world

These obvious shortcomings in today’s publishing are at the heart of what we are trying to solve with quilp — to be launched shortly — and what I believe the technologies around Web 2.0 are best suited to enable people to do. I therefore couldn’t disagree more with Keen‘s argument, neither with his underlying observation that the current process is effective and ensures quality nor with the assumption that quality has to suffer if this process is opened up to amateurs.

I believe that there is a tremendous amount of unused creative potential out there that we can tap into and open up to everybody by providing the right tools as an enabler for sharing and evaluating ideas without outdated bottlenecks.

Strangely enough, at the same time that Keene favors a few gatekeepers over selections by the crowds he cites an article from the Wall Street Journal to prove his point: of the 900,000 registered users at Digg.com, only 30 were responsible for submitting one-third of the postings on the home page. Isn’t this the selection by the few experts that he is so fond of? Or is it a question about the authority of being an expert? But who is better suited to determine who the expert is in a certain field — a few employees at a publisher or newspaper or a crowd of people especially interested in a specific topic?

I think what Keen really wants to stress with this quote is the room for manipulation he is seeing in amateurs vs. professionals, their hidden agendas. If I recall the events that lead to the current war in Iraq correctly, however, it is the professional media that has been mislead the most and the independent amateurs that were giving a voice to a more pluralistic debate. That the professional media should be immune to hidden agendas especially if they are owned by arms industry magnates like in France (Lagardere, Dassault) is highly questionable, as Jürgen Altwegg points out in the Frankfurter Allgemeine Zeitung (July 4th, 2007); the internet, as Altwegg — a professional journalist by Keene‘s standards — argues, gives more room for tough questions, putting more pressure on journalists in turn to address these issues.

As Surowiecki points it out in his excellent book The Wisdom of Crowds there are certain elements required to make a crowd’s decision wise. Arguing against or in favor of crowdsourcing in general terms therefore doesn’t make a lot of sense. The design of the tools enabling this process is crucial for the value it manages to provide.

P.S. Funny enough, I only came across the review of Keen‘s book debating the value of crowdsourcing through one of my favorite crowdsourcing tools: using the affinities with other readers which I discovered through my favorites on del.icio.us I can now quickly filter millions of documents, as other gatekeepers which I determined as being relevant to my interests are doing the filtering for me.

Update (July 7th, 2007):
There has been quite some coverage of this topic recently – some interesting links:

Update (July 10th, 2007):
“A Luddite argument is one in which some broadly useful technology is opposed on the grounds that it will discomfit the people who benefit from the inefficiency the technology destroys. An argument is especially Luddite if the discomfort of the newly challenged professionals is presented as a general social crisis, rather than as trouble for a special interest.” Clay Shirky in Andrew Keen: Rescuing ‘Luddite’ from the Luddites (July 9th, 2007)

And here another post by Clay Shirky, quoting Scott Bradner: The Internet means you don’t have to convince anyone else that something is a good idea before trying it. The upshot is that the internet’s output is data, but its product is freedom.(July 10th, 2007)

Update (July 20th, 2007):
The Good, the Bad, And the ‘Web 2.0’ (Andrew Keen and David Weinberger in Wall Street Journal; July 18th, 2007)