William L. Coney Blog: Are There Statistics on Inaccurate Forecasts?

Tuesday, May 16, 2017

Are There Statistics on Inaccurate Forecasts?

How often do forecasts fail? How many business forecasts are accurate? Are there statistics on inaccurate forecasts? It’s a good question. It relates nicely to my view that all business plans are wrong, with the addendum, but vital. While I almost never host guest posts on this blog – opinions here are mine – but I liked this one enough to post it here.

It’s a Quora answer, by Hubertus Hofkirchner, to the question Are there any statistics available on forecasting failures?

Here’s what he wrote:

This is a dangerously simplifying question in a highly complex subject. Let’s forget crystal balls, Gandalf, and Harry Potter for a moment. Let’s use Prediction Science. There are three hidden levels to this question.

Level 1 – Simple Prediction

A forecast impacted by human action can never be 100% certain, because humans will react to forecasts with unforeseen actions which in turn can change the future dramatically, in accordance with chaos theory.

We can only measure the accuracy level and forecast bias of a specific method for multiple predictions – not a single one – with regards to specific forecast topics.

So: What level of unavoidable inaccuracy or bias do we chose to call a failure? What is the commercial worth of more accuracy compared to the cost of producing it with a more expensive method?

Level 2 – Decision Making

Human decisions combine forecasts with their subjective (and never known) purpose and value judgements. This brings the next level: every human action has a purpose. The decision maker can act on an inaccurate or biased forecast but still achieve the actual purpose successfully.

So: Do we chose to call a success then a prediction failure?

Level 3 – Deception Intent

Also, the question assumes a one-way causality, that politicians act on forecasts, implying that bad forecasts will cause bad decisions.

However, politicians often work the other way round. Let’s assume that a politician (or the lobbyist paying him) wants to trigger a war. He produces a forecast by which the citizens’ purpose will support the liberation (politically correct word for war) of some country, somewhere. For example, the politician might predict the existence of unspeakable weapons. The public diaspproves the weapons, thus approves the war.

Of course, when the fabricated prediction fails to materialise, the public purpose is frustrated.

So: Do we chose to call this a forecasting failure? After all the politician’s real intent did work out perfectly fine.

The Riddle’s Answer

The big answer is not “42”. It is “50%”.

Explanation: There is no such thing as a forecast failure (see 1. above). There are right and wrong decisions (see 2. above), and “failure” is but the non-achievement of the decision maker’s purpose (see 3. above).

For economics, let’s assume that the investor wants to make a return in line with benchmark. On the stock exchange, it is obvious by tautology, that 50% of investors will perform above benchmark and 50% below. This very tautology is of course exactly valid for economic and political decisions. So the answer is 50% of forecasts fail.