Good practices in forecast uncertainty assessment and communication

Contributed by Lionel Berthet and Olivier Piotte

The French Flood Forecasting Service (SCHAPI) recently carried out a survey to identify good practices in the assessment and communication of forecast uncertainty.

This was motivated by the fact that SCHAPI currently publishes hydrological warning maps (in French, “Carte de vigilance crues”) and makes available in real-time observed river levels and discharges (see here in real-time and post-event examples here), but intends to provide also hydrological forecasts issued by the local flood forecasting centres to the public in the near future.

SCHAPI forecasters acknowledge that, to publish forecasts on its website, uncertainties need to be first assessed and properly communicated. Special attention needs to be paid to the way this is achieved. Since there is no systematic national guidance for uncertainty assessment and communication so far, a framework to be shared by the national forecasters’ community needs to be built. These aspects are of special interest:

• Uncertainty understanding,
• Forecasting uncertainty assessment,
• Communication (in various situations).

To start building this framework, a survey on good practices in the assessment and communication of forecast uncertainty was prepared to be applied to a selection of similar public institutions that also provide warnings and forecasts to civil protection officers.

Results from this survey were presented during the session ‘Hydrology for decision-making: the value of forecasts, predictions, scenarios, outlooks and foresights‘ at the EGU 2014 Assembly in Vienna (see abstract here) and are summarized in this post.

The survey

The survey was a semi-closed frame which included:

• a unique questionnaire,
• open discussions with respondents, and
• a common analysis of the answers.

Various means for acquiring the information needed were adopted, including:

• Literature review,
• Face-to-face meetings,
• Phone interviews (mostly driven in English, French and German),
• Email discussions (questionnaire, answers, further discussions, etc.).

The survey questionnaire contained four main questions:

1. Do you assess forecasting uncertainty? If so, which tools/models do you use?
2. Do you publish water level or discharge forecasts? If so, are the forecasts completed with information about the associated uncertainty? Which representation do you use? (text, forecast interval, other…).
3. Do you specifically communicate uncertainty information to civil authorities? If so, are they used with such information? Is this information welcomed? Used?
4. Do you meet specific pitfalls in your working context? In uncertainty assessment? In uncertainty communication?

The survey was not intended to be a rigorous sociological study, but rather to be a ‘start point’ for better understanding current practices and implementing new tools for the French flood forecasting service.

Main results

• Do many services deal with uncertainties?

• Forecast uncertainty assessment: 89% (16 answers)
• Uncertainty publication (e.g., on the Internet): 94% (15 answers over 16)
• Specific communication to civil authorities: 50% (8 answers)
• A posteriori evaluation of uncertainty assessment at least: 25% (4 answers)

• Which uncertainty sources?

• Meteorology: always
• Hydrology / Hydraulics: often, but not always

• How do they deal with forecast uncertainty?

• Ensemble forecasts: 69% (11 answers)
• Meteorological multi-model: 44% (7 answers)
• Several meteorological scenarios: 38% (6 answers)
• Post-processing: 69% (11 answers)
• Hydrological/Hydraulic multi-model: 13% (2 answers)
• Controlled subjective analysis: 6% (1 answer)

• How is forecast uncertainty communicated?

• Quantiles: 87% (13 answers over 15)
• Text (literal comments): 38% (6 answers)
• Min/Max: 33% (5 answers)
• Over threshold probability: 33% (5 answers)
• Pessimistic scenario: 7% (1 answer)

• How is forecast uncertainty accepted? (open question)

• Highly variable: depending on the ‘uncertainty general culture’ of the countries;
• It is commonly acknowledged that we need a shared language with stakeholders to efficiently deal with uncertainty, with common trainings of forecasters and key stakeholders and formal messages prepared in common agreement (e.g., SMHI in Sweden, EA in UK and JRC at the European scale).

Lessons learnt

• Most forecasting services deal with uncertainty, but there is no consensus on the method(s) to be used by operational services.
• Two methods are more commonly used: a large majority uses ensemble approaches and/or post-processing of deterministic forecasts.
• All services are aware of the need for a reliable assessment of the meteorological
  uncertainty.
• Uncertainty communication is recognized as valuable.

Additionally, this survey was also useful to ourselves: to help convincing our national forecasters colleagues and our stakeholders of the need of implementing pre-operational experiments on uncertainty quantification and of continue on training forecasters on communicating forecast uncertainty.

We thank all the forecasting services that answered our survey.

Additional comments are welcome here!