EXTRA – Ensemble predictions are not real forecasts – EXTRA
Contributed by McFools
Forecasting has since the beginning of time been the process of making statements about events whose actual outcomes (typically) have not yet been observed. Consequently, adding uncertainty to your forecast permitted you to make many statements about the future.
Exposing the uncertainty myth: Who can drive a car using probabilistic traffic lights?
However, theorists claim that you cannot simply add uncertainty as you wish: no, the forecast has to be reliable and skilful! It seems like an awful lot of work!
Previously, we used deterministic, high resolution forecasts and changed them according to our intuition and experience. Most of the time that worked pretty well, and since we showed a single forecast everybody believed that we knew exactly what we were doing.
In those days we could easily derive a simple traffic light system with green (all OK), yellow (take care), red (flooding imminent). With ensemble forecasts everything has changed (how do you explain that there is a 10% chance of getting flooded, it just does not make sense!!!) and we just look like fools, having no clue about the future.
So, following this reasoning, it is clear that ensemble predictions are not real forecasts, they are just an excuse for hydrologists to shift responsibility and avoid getting the blame and being fired.
This might sound a bit harsh, but it is the truth, and it is about time to wake up from this uncertainty nightmare. Other ludicrous claims and assumptions behind ensemble/probabilistic predictions confirm this:
- They never ever give you an exact and unique answer to your problem
- They give you forecast hydrographs, but you cannot recognize your peak flow and they tell you it’s normal because it is a ‘spaghetti plot’!
- They keep telling you that you need to wait some time to get a verification score and finally learn about the quality of the forecasts – I don’t want to wait, I want to know the skill of my forecast today!
- They say they have economic value, but you still have no extra money in your pocket at the end of the month
- They push hydrologists to work with meteorologists in the same room and tell you that it will be okay and nobody will get hurt
- They make your computer crash with low memory space by the time you start believing you can run ensemble predictions and get the first results ready for the next risk prone season (after the season is over of course)
- When you solve your memory problem, they come again with a 20-year reforecast to run!
- They promise you a PhD time of scientific glory and when you publish your results, reviewers tell you that your paper is rejected because confidence intervals around the Brier Score are missing
- They ask you to be post-processed but when you get better accuracy with a state-of-the-art technique, the reliability score is worsened (and vice versa)
- They tell you they are serious predictions made by serious people and recognized scientists, but when you go to their website, you find games, YouTube videos (just look at the xmas one, which proves my point) and a photo Gallery full of amazing people interested in planning another workshop!
Probabilistic forecasts: press the red button
Therefore, we believe it is time to demand that hydrologists and meteorologists are given again a proper education (and re-education for all the lost cases out there), which above all ends the nonsense of uncertainty in forecasting.
Join in and leave a comment in support!
Want to know more? Why not subscribe to our mailing list, follow us on twitter, facebook or LinkdIn. Before citing this article we strongly suggest to check the author’s name and date of publication.
April 1, 2014 at 10:34
Come on Mc Fools! Use the ensemble mean!
April 1, 2014 at 17:00
The mean of green, red and yellow is light brown – so should I cross the road or not??
April 1, 2014 at 17:05
Now you catch me. This is not fair…
April 1, 2014 at 17:15
Maybe you could use a weighted mean, with a weight vector full of either 0 or 1?
April 1, 2014 at 17:57
In Germany, no. If not green, ppl will wait regardless. In the UK, look to the left and right, then go and cross. So user’s decision, based on culture, background and perception.
April 2, 2014 at 16:08
In essence, I think this answer is true:
For my river system, I accept the risk to cross the road on light brown signal, because I can afford for this risk.
But on my neighbor river system, he cant simply accept any risk, so he will not cross unless the signal is green.
Finally, the deterministic system does not allow me for thinking this: – ok! It is light brown… maybe I should wait some more minutes to see if it is going to be yellow or green, even if my girlfriend will be a little upset because I will be late.
April 1, 2014 at 10:37
I laughed.
April 1, 2014 at 13:01
To a long time supporter of the EPS Mr Fool’s critical views are uncomfortable, in particular since they seem to be shared by an ever widening circle of “skeptics”. They now seem to have decided to go from words to action. From an “agency”, which is bugging the internal ECMWF email correspondence I have been told that there are far reaching plans to do something radical with the EPS to make it more popular and market orientated.
The project, which will soon be launched, would distribute the 50 ensemble members among the Member States: every Member State would get at least one member and the biggest countries get 2 or more, in proportion to their budget contribution. So would Germany get members 1-6, France 7-11, UK 12-16, Italy 17-19, Spain 20-22 and Netherlands 23-24.
This arrangement will solve the problem that the deterministic, very high resolution, detailed forecasts the Member States have so far offered the public and paying customers have been IDENTICAL. Now each Member State will be able to offer “their own” high quality deterministic forecasts. It is a common experience that the public regard weather forecasts made abroad as better than domestic ones. The new scheme now offers for example Montenegro the chance to compete with not only middle-sized countries like Sweden and Norway, but also with France, Germany and the UK!
The idea has been welcomed also in the big countries, my confidential source has revealed to me, since it encourages internal competition. Deutscher Wetterdienst can distribute its six deterministic EPS members individually to its regional offices in Hamburg, Potsdam, Leipzig, Essen, Stuttgart and Munich. Metéo France can distribute its four deterministic members to its major regional offices and for example enable Lille to compete with Paris and Toulouse.
This “New Deal” will hopefully contribute to the strengthening of the European economy. The ECMWF is pleased that at last a sensible use of their long time investments in expensive state-of-the-art computers has been justified. Their scientific staff is relieved that they no longer have to bother about probabilities and other queer statistical artifacts.
April 1, 2014 at 17:11
this joke really amuses me, eh…..it is not a joke?
April 2, 2014 at 05:56
We all know that when we run a model once, the resulting forecast has error (it’s wrong!) almost from the get-go, and to paraphrase Rita Mae Brown (1983), “Insanity is doing the same thing over and over and expecting a [better] result”. Thus you cannot argue but that ensemble forecasting is the very definition of insanity.
Shucks, it’s back to the drawing board…
April 2, 2014 at 10:03
The challenge now : spot the FIRST article seriously citing this blog as an argument against using probabilistic forecasting!!!
April 2, 2014 at 11:52
I would rather like to turn the statement around: -Does the ECMWF T1279 very high resolution deterministic output constitute “forecasts”? In my view NO. Nor does the output from the American GFS, DWD or the UKMO deterministic models.
If you want to e s t i m a t e the typical weight of middle aged male Swedes you do not go out and just weight the first middle aged Swede you meet. He is just a random sample. Your measurement might be close to the best estimate, but you have no idea if it is above or below, and no clue about the typical spread and thus the uncertainty in your value.
But when we want to estimate the future weather then it seems to be all right to rely on a random sample in the name of ECMWF deterministic “forecast” although it has no more information value that our first measured Swede. Looking at the GFS, DWD or UKMO deterministic forecasts is like measuring the 2nd, 3rd and 4th middle aged male Swede you happen to encounter.
Consequently, estimations made from ensemble systems – to which are added reliable uncertainty estimates, is the only information that could be called “forecasts”.
April 2, 2014 at 16:22
Anders,
I really like your answer.
Let me add comments about it:
– Now suppose that you did an “ensemble middle aged Swede measurement experiment” and discovered that the range of
heights are between 1.4m and 1.9m. Also, the most common value is about 1.6m.
– You are not going to sell clothes out of this range, because they are not necessary (or, are rarely).
– Maybe you should have more clothes for people with 1.6m, because this client is more likely to occur.
Now your store seems more prepared than it would be if just relied on selling clothes based in one measurement.
April 3, 2014 at 10:34
This article comforts me in my decision to get away from the hydrological ensemble forecasting community. Climate change people look much more serious to me!
/goes back to his Q99 projections for 2083 on his tiny river basin/