Mauboussin: Why the Market is Smarter Than You Are

Most people who write about the wisdom of crowds offer colorful anecdotes about crowd intelligence but don’t explain why groups are smarter than individuals.  In a recent piece, Explaining the Wisdom of Crowds, Legg Mason strategist Michael Mauboussin solves this mystery.

Drawing on a book by Scott Page (The Difference) and some of his own experiments at Columbia Business School, Mauboussin first observes that, in order for crowd wisdom to prevail, several conditions must be met:

• The group must be cognitively diverse (the individuals in the group must have different approaches to analyzing and solving problems).
• There must be a way to aggregate each individual’s opinion into a collective view (such as a stock market).
• There must be incentives: rewards for being right, punishment for being wrong.

In most cases, the stock market easily satisfies all three conditions.  (Mauboussin has argued in the past that part of what happens during bubbles is that there is reduced cognitive diversity, as most investors begin to think and invest alike). 

Mauboussin then describes three distinct types of situations in which groups are wiser than individuals. In the first, the "needle-in-the-haystack," a subset of the group knows the correct answer, but the majority does not (Mauboussin uses the example of the audience on Who Wants to Be a Millionaire).  Here, the crowd’s "best guess" is superior because the subset of people who know the answer ensure that the correct answer is the most popular choice (everyone else’s guess is random).

In the second and third cases, which are closer to the situation in the stock market, no one in the crowd knows the correct answer, so it must be estimated.  (In the stock market, no one knows what stocks are worth because the valuation process is subjective and because the key variable–future cash flows–is unknown).  In these cases, Mauboussin attributes the crowd’s wisdom to Page’s "diversity prediction theorem," which states the following:

Collective error (the difference between the correct answer and the group’s average estimate) = average individual error minus prediction diversity (the dispersion of the individual estimates).

Mauboussin invokes his Columbia experiments–"guess the number of jelly beans in the jar" and "predict the Oscar winners"–to show how this theorem works.  He also summarizes the theory’s implications, all of which apply to investing:

1. A diverse crowd will always predict better than the average individual.  Not sometimes.  Always. 
2. Collective predictive ability is driven by average individual accuracy and diversity.  In other words, a crowd composed of simpletons who think the same way will be less intelligent than one composed of independent-minded geniuses or experts.
3. The group is often smarter than not only the average individual, but the smartest individual.

The last point is critical for investors.  Because of our innate overconfidence, we quickly dismiss the fact that a diverse crowd is always smarter than the average individual.  (Most of us think we’re above average).  What is far harder to dismiss is the fact that the crowd is often smarter than all individuals. 

In instances in which the crowd is not smarter than all individuals, moreover, the individuals who beat the crowd usually change with each successive prediction (which gives the investment media a parade of new gurus and "fallen stars" to write about).  Just because you are occasionally right when the market is wrong, in other words, doesn’t mean you usually will be.