, , ,

Reading an article like this reminds me of the work by psychologist Philip Tetlock who studied expert predictions in areas such as economics and politics. What he found was truly shocking. They are on average only slightly better at predicting than random guessings, and are worse than basic extrapolations.

Even crude extrapolations are more reliable than human predictions — Philip Tetlock

Further, they tend to exhibit higher degree of confidence in their predictions. They tend to assign narrower 95 percent confidence intervals for their predictions. A confidence interval is a range of values (interval) for possible outcomes. The 95 percent simply means that they are 95 percent confident that the true value will fall within this range, or in the other words, they believe that there is a 95 percent chance that the true value will be in this interval. A narrow confident interval implies that they are more confidence (less variance) about their predictions. It turns out that the true values fall outside of this range more than 5 percent of the times. That is a sign of overconfidence.

Rather than asking why, it is more useful to know when they are more likely to be wrong.

In some domains, experts are more likely to outperform the average person: imagine playing Go against a 9 dan professional or playing tennis against Roger Federer. Michael Mauboussin and Beth Azar argued that expert performance largely depends on the characteristics of their domains of competence. In the area where outcomes are largely probabilistic and high degrees of freedom (a lot of variables), experts tend to perform poorly against a collection of average persons. For instance, pundits are often wrong about the future outcomes of the market and economy.

Interestingly, high media exposure tends to correlate with poor predictions. Well-known forecasters are more likely to make bold predictions and end up being wrong than their lesser-known peers. More reason to be wary of talking heads on the radio and television.

Since there is no disincentive to making wrong predictions, it is not at all surprising that media pundits are still predicting several market crashes in the last five years, while the S&P 500 index has tripled its value in the same period. But a good news is: Google and other websites are now making it easier to keep these pundits more accountable.

Finally, I like to finish off with a graph to illustrate that even a respectable Australian financial media is not immune from this trap. More detail here. It is a funny read.