The Hidden Challenge of Forecasting Futility: Why Some Forecasts Are Useless or Harmful

By F. Pappenberger, M.H. Ramos, A. Wood

[Editor’s note: This is the first of two posts on Hydrological Futility; stay tuned for the second one!]

The problem

“When models mimic, but nature mocks, futility blooms in the gap between predictions and the truth.“

Have you ever felt like you’re shoveling against the rising tide, your best efforts seemingly swallowed by the vastness of the challenge? This sense of futility, a pervasive force in the realm of human endeavor, manifests in fascinating and often frustrating ways across diverse fields, from medicine to hydrology. 

In this post, we ruminate (argumentatively!) on the current state of hydrological forecasting, drawing inspiration from the nuanced concept of futility in palliative psychiatry to reflect on the shadows it cast also on hydrological forecasting (https://www.nytimes.com/2024/01/03/magazine/palliative-psychiatry.html). Our aim: to decipher its complexities, wrestle with its inherent challenges, and ultimately understand how to navigate its treacherous waters.

Why futility in forecasting?

In the domain of medicine, futility takes center stage in the emergent field of palliative psychiatry. This is expertly captured in Katie Engelhart’s poignant article on palliative psychiatry, which embodies a shift in perspective. Rather than clinging to the unattainable ideal of “curing” chronic and severe mental illnesses like schizophrenia or bipolar disorder, accepts the futility of the goal and instead prioritizes acceptance, allowing a meaningful shift of objectives toward reducing suffering and fostering quality of life. This compassionate approach recognizes the limitations of conventional psychiatry and embraces a nuanced understanding of mental illness, focusing on individuals’ strengths, preferences, and values beyond the confines of diagnoses and prognoses.

We can wonder if futility also lurks within the realm of hydrology, which is ideally governed by objective laws of physics and science. Year after year, despite great investment by countries around the world, damaging droughts can begin with little warning, leaving citizens and cities scrambling to safeguard water supplies; examples abound, including California in 2011, Cape Town in 2015, São Paulo in 2021, Northern Italy in 2022. 

Imagine the modeler or a forecaster tirelessly refining hydrological models and forecasting systems only to see that expected benefits fail to materialize, or worse, produce forecasts that can undermine or even mislead decision-making and exacerbate their outcomes, which may include water-related crises. It is not uncommon to see a forecast of expected ‘normal’ or even wet conditions, barely a month before all rainfall ceases and rivers begin to dry up. 

That the current end point of myriad forecast advancement efforts over decades and across many countries still struggles with such extreme event prediction  could be regarded by some as “hydrological forecasting futility”.  It emerges from a multitude of factors: unreliable observational data, overly complex models, misaligned priorities, or simply the inherent unpredictability of natural systems, which can lead to substantial and irreducible forecast uncertainty. 

We define “Hydrological futility” as a term when the expected benefits of improvements or modifications brought to hydrological modeling or forecasting practices, actions or strategies within a hydrometeorological forecasting system cannot be realized, despite the best and rigorous efforts, or efforts perceived as such, or may even reverse into harming forecast quality or usefulness. This is due to various factors that undermine forecast reliability, usability, acceptability, quality, complexity, alignment, validity, clarity, relevance, improvement, prevention, mitigation, enhancement, or achievement for water-related decision making, considering the way we evaluate the forecasts or the system’s attributes today.

Examples of futility

The concept of hydrological forecast futility might seem abstract, but its consequences can be stark and far-reaching. Let’s dive into some concrete examples to illustrate the diverse ways in which we believe this phenomenon can manifest:

1. Overconfidence in Forecasts: Imagine a scenario where meticulously crafted flood forecasts lull communities into a false sense of security. While forecasts may have the benefit of successfully raising awareness and governing behavior in the face of potentially dangerous situations even when a flood event does not materialize, it may also happen that, based on seemingly robust predictions, residents delay evacuations or neglect precautionary measures. Unforeseen factors like intense localized rainfall or rapid snowmelt may then lead to flash floods, exceeding the forecasted levels and causing devastating damage. An example may also be the forecasts of  Hurricane Irma –  it was initially predicted to hit the east coast of Florida, but it shifted westward and made landfall on the west coast instead. This caused a lot of confusion and panic among the residents, who had to evacuate or seek shelter at the last minute.

2. Misaligned Priorities and Unusable Forecasts: Consider a situation where the forecasting enterprise prioritizes accuracy over accessibility. The resulting forecasts, while technically ‘precise’ in description, may be presented in complex jargon or released behind password protected websites (“our survey data reveal that up to 50 % of the warned residents did not know what to do in July 2021”). This can render them unusable for local communities who lack the resources or expertise to decipher them, hindering timely and effective responses to flood warnings.

3. Chasing the Elusive “Perfection”: The quest for ever-more precise or physically explicit (e.g., higher resolution) hydrological models can become a Sisyphean task. Researchers may pour resources into refining models to account for every minute detail, only to find that the inherent unpredictability of natural systems and associated uncertainties fail to translate this refinement into improved prediction skill. Chasing an impossible ideal of resolving process heterogeneity at scales well below those which observations can support, while  promoting hypothetical forecast benefits from such refinements, can divert attention from core challenges such as  observational data gaps, scientific and technical gaps, communication strategies, and community preparedness, ultimately hindering progress in managing water-related risks.

4. The Paradox of Intervention: While reservoirs and other large-scale infrastructure projects, such as dams and levees, play a crucial role in preventing significant flooding and offer numerous benefits including water storage, hydroelectric power generation, and irrigation support, there are instances where they can have unintended consequences. In certain situations, these structures may alter riverine ecosystems and, particularly in cases of mismanagement or design oversights, can redistribute or even exacerbate flooding in downstream areas. This highlights the complexity of interacting with natural systems and underscores the importance of complimenting such infrastructure with sustainable land management and community-based flood preparedness strategies to mitigate potential adverse impacts effectively. An example of this maybe in the disastrous floods in Pakistan that are are not natural disasters, but rather the result of human actions that create vulnerability and poor governance. 

5. The Looming Specter of Climate Change: The specter of climate change adds a layer of complexity to the already intricate challenge of hydrological forecasting. As weather patterns change, traditional forecasting methods struggle to adapt. This necessitates innovative and robust approaches that can factor in the uncertainties associated with a changing climate and provide actionable insights for managing water resources effectively in the face of this unprecedented challenge. Continuing to focus on historical model performance in optimizing systems, without including metrics of their ability to adapt to change, using variations of the same methods from the last two decades, even as the geophysical landscape evolves in the background, has all the hallmarks of ‘futility’.  So how far do our forecasting systems can play a role in climate change adaptation? Are the models and techniques robust enough to forecast the “never seen before”? The recent flash floods in US showed that “unseen” events do happen! Also check out the papers in this special issue.

A useful concept?

Does the concept of Hydrological Futility resonate with you? Please tell us in the comments. We will continue our thoughts on Hydrological Futility by describing the roots of Futility in Hydrological Forecasting in our next blog post.