What do operational forecasters really want from their systems?
Operational hydrological forecasting systems have seen a quite remarkable development during the last 10-15 years. Ensemble forecasts, improved numerical weather predictions and new features in hydrological models (updating, data assimilation) and decision support tools have made the life of a hydrological forecaster both easier and more difficult at the same time. Easier, since a lot of the information is available online and in real-time, and more difficult, since with the abundance of information, the uncertainty of the modelling system has become much more visible.
Questions then arise: are the operational forecasters happy with their current systems? Is the research done focussed on improving the right things?
Figure 1: Hydrological ensemble systems put more demands on the operational forecaster than previously.
At the EFAS 7th Annual meeting in 2012 in Norrköping, Sweden, we designed an interactive exercise with the EFAS community to get some answers to the question: What is the most important area of development for the forecasters?
The idea was to pick the brains of the more than 30 hydrological foreasters present at the meeting to work out in which direction they thought operational hydrological forecasting should go, in general, and of course EFAS, in particular.
Part 1. Dragon’s Den
The first part of the exercise was modelled after the popular TV show “Dragon’s Den“, where the forecasters were divided into small groups and given the task to discuss what the most important research and development question was, what to prioritise. They were also asked to prepare a 5 min pitch to present in front of the all other groups and a panel of dragon’s, consisting of experienced forecasters and researchers who scrutinized their pitch. The final part of the exercise was to vote for the most popular pitch. In this, the attendants were given 10 SEK (~€1.1) each, which they could divide amongst the pitches as they saw fit. To keep the money was also obviously a choice!
The pitched priorities (in order of popularity) were:
- Multi-model forecasting system SEK 70
- Build a European flood forecasting infrastructure SEK 56
- Forecast verification tool SEK 55
- Improve physical model representations SEK 51
- Improve standardization of hydrological data SEK 41
The total money spent was SEK 273, which meant that less than 10% of the money was kept by the forecasters for themselves.
There was one pitch that really stood out, a multimodel forecasting system for both hydrological models and NWPs. EFAS uses a multimodel NWP approach, but so far only one hydrological model.
No pitch was deemed useless, all received money, and they were also quite different in their character, showing that there are many aspects of the forecasting chain that are in need of developing.
Inspired by this exercise we took all the suggestions that came up during the discussions, added more priorities of our own account and proceeded to the second part of the study.
Part 2. The Questionnaire
Following the meeting, a questionnaire was sent out to the participants, asking them to rank a total of 23 priorities on a scale from 1 to 5 according to what they thought was most important.
The top ranked priorities were (in descending order):
- Forecast verification for hydrological and meteorological forecasts
- Introduce multi-model approach for hydrological modelling
- Increase the average skill of the medium range forecast (>3 days)
- Education and training of how to use and interpret forecasts
- Improve physical model representations
Maybe not so surprising, the most popular priorities from the pitch were also represented in the survey (marked in bold). Education and training can be said to be part of the effort to build a European flood infrastructure. In the survey, the priority of a hydrological multimodel system was separated from NWP multimodel system, and since EFAS already has an NWP multimodel system it is not so surprising that a hydrological multimodel system was ranked higher in this particular study.
The bottom ranked priorities were (in reverse order):
- Replace/expand web forum by social networks
- Distinguish between different flood situations
- Increase the frequency of forecasts
- Increase the temporal resolution of the forecast
- Blending of national and EFAS forecasts
There were some surprises in the least popular priorities. Social networks were not seen as a priority, although there has been examples of how these can be used in emergencies. However, a HEPS system is an expert system and is not designed to be public. This can also be linked to the fact that blending EFAS on national systems were not popular. It is clear that in the current format, the forecasters want a separation from the early warning system of EFAS and the national systems, perhaps to avoid confusion during a crisis situation.
A road map to future developments
The result from the survey was further analysed in terms of their cost and complexity (figure 2). The cost was estimated in terms of how much capital investment is needed to realise the improvement, and complexity was estimated as the expected technical and/or scientifical difficulty of solving the underlying problem.
Figure 2. The top 10 prioirities from the survey organized according to their cost and complexity of realising each of them.
From this diagram, a “roadmap” for future developments was created, where priorities where grouped in terms of their cost/complexity to be able to make priorities for the short and long term.
1. Secure funds for the priorities that yields most benefit to a low cost and with low complexity
- Training and collaboration between forecasters at national and international level
- A “User guide” for hydrological probabilistic forecasting to improve forecast interpretation and decision making
- E-learning tools designed to show the added benefit of using HEPS
Training and collaboration are currently done within the EFAS and HEPEX communities, and it is a relativerly cheap way to make sure that new ideas and best practices spread amongst researchers. Integrated workshops including both researchers and practitioners, like the recent HEPEX workshop in Maryland, is a good example of this.
2. Plan and coordinate activities to deal with intermediate cost/complexity priorities
- Report past performance through forecast verification scores
- Showing calibration and validation results
- Include more NWP outputs in the system
The second category deals with planned or ongoing improvements of the current operational system. In this case, the priorities are EFAS-specific and other systems will have different research priorities in different stages of their development. Also here, systems can learn from each other and exchange ideas and collborate on development projects.
3. Long-term strategy to coordinate research and development for costly and/or complex priorities
- A multimodel hydrological system
- Standardise hydrological data collection
- Improve forecast dissemination
The third category identifies priorities that are not easy to implement and requires either more research or allocation of funds to adress specific problems. Problems in this category would typically be adressed through larger research projects, either through national research funds (e.g., NERC, DFG) or international funding programmes (COST, HORIZON 2020).
4. Collaboration with the broader scientific community on long-term improvements of HEPS
- Improve the physical representations in the used models
- Improve the forecast on lead times > 3 days
The fourth category consists of priorities that are either extremely complex to adress, such as a general wish to improve the forecast on a certain lead time, or are described as problems where more basic research on hydrological processes is needed. These priorities can often not be addressed in clearly defined, local projects, but they can be defined as problems to be addressed by a larger research community. It is therefore very important that the HEPS community clearly demontrates and points to existing knowledge gaps.
The “roadmap” presented above is derived from the needs of the EFAS system, but it also nicely echoes the HEPEX topics in its Science and implementation plan, which shows that the identified research priorities are universal and that the HEPEX community is an excellent platform where to create the next generation of hydrological forecasting systems.
For further reading, see the opinon paper in HESS.
August 16, 2014 at 22:02
These surveys or “Gallup’s are really interesting and worth while. I would like to see similar investigations done on the meteorological side.
However, what surprised me was the lack of interest in statistical systems. So for example, in the figure 2, the wish for “reports on past performs” implies that knowledge about this will be of practical use in the forecasting. That is quite true, but this evaluation of past performances can be made in a way that an automatic system modifies or suggest modifications to the most recent forecast, both in a frequentist and Bayesian way.
In the frequentist approach you evaluate how, for example, the performance the last year could have improved if certain systematic misbehaviours had been corrected for. And then you apply these corrections for the next year.
In the Bayesian approach you evaluate the misbehaviours the last week or so projected against a pre-defined error structure and then correct the forecasts for the coming days. If these corrections “over-shoot” or “under-shoot” you modify, just as you correct the rifle sight.
August 17, 2014 at 11:47
Anders, thanks for your comment.
You are quite quite right, but the performance in this sense refers to verification of the system, not necessarily for post processing, but more to have trust in the system. It relates more to transparency rather than increasing the skill; the forecasters wants to know how good the system is.
Saying that, increased skill is obviously something desirable. In this survey we did not distinguish between post-processing and model development as methods of increasing skill, which perhaps is a wekaness of the survey. could be worth mentioning that EFAS uses a Bayesian postprocessing for many points in which we have real-time discharge observations.