Challenges of operational weather forecast verification – interview with Tom Pagano

A new study titled “Challenges of Operational Weather Forecast Verification and Evaluation” has been published in the Bulletin of the American Meteorological Society. Led by Thomas Pagano, a long-time HEPEX member and senior scientist at Australia’s Bureau of Meteorology, the research involved over 50 experts from six countries. We caught up with Tom for a chat.

James Bennett: What drove you to perform this study?

Thomas Pagano: Operational forecasting agencies constantly need to demonstrate the quality and improvement of our forecasts, and that we are making best use of our available resources. Researchers are eager to assist, yet there’s often a gap in understanding our key concerns. We aimed to articulate these challenges from an operational perspective and highlight critical research priorities.

Bennett: What were the major hurdles to clear in writing this article?

Pagano: The article had 20 co-authors and I’m sure there’s some formula for how the effort scales (factorially?) as you add more co-authors. But the passion and insights everyone brought were impressive. I’m particularly grateful to my four Bureau co-authors, including Beth Ebert.

Another challenge was balancing the varied perspectives from different agencies, some with dedicated teams using advanced systems and others working with basic tools like spreadsheets. Yet, common themes emerged about what keeps us up at night.

Bennett: Among the five verification challenges identified – verification approaches, observations, evaluation of forecast impact, communication and institutional factors- which is the most critical?

Pagano: For sure this will depend on the agency and their situation. Some agencies even struggle to have easy access to a long historical record of their forecasts!

But for decades I’ve felt that the most under-appreciated issues are on the social science side, rather than the physical science side. I struggle with the jargon in forecast verification- and I work in the field! How am I supposed to translate all these metrics into something that is clear, concise, and relevant to any given user of our forecasts?

We are increasingly doing hazard- and risk-based forecasts and there’s many interesting challenges there as well. For example, what happens when you issue a flood warning and people take protective action to minimize damage. If you use impacts to verify the warning and there were no reports of damage, was this an over-warning? Or a good response to the warning? Or maybe even just missing data? Luckily, other fields like medicine have ways of accounting for this.  

Bennett: Did any challenges surprise you?   

Pagano: The impacts of climate change on forecasting sparked a lot of discussion. We increasingly see headlines like “Climate change will make forecasting harder” but this can mean many things. Will there be more extremes and that extremes (generally) are harder to forecast? Or is it that some major process is going to change?

Bennett: In the context of hydrology, you might get a rainstorm of an intensity and duration that’s way outside the range of the data you’ve used to calibrate your hydrological model, making the response tough to predict.

Pagano: Right. Or consider last year’s Hurricane Otis. The forecast was for category-nothing but then soon after it intensified to category 5 and made landfall. When there’s news articles saying climate change is to blame, are they talking about the event, the bust, or both?

Weather modellers say that they include nearly everything that is important on weather timescales, so it’s hard to know what more they should be doing to account for climate change. Some think it’s a non-issue or a misunderstanding, but people are asking the question, and it would be helpful to have answers.   

Bennett: It’s also quite possible that the ongoing improvement of weather and hydrological prediction systems will outpace any difficulties posed by non-stationarity.

Pagano: That’s true, the steady upward march in skill we’ve seen has been described as “The Quiet Revolution” and no doubt that’s going to continue.

Bennett: It might even be a crescendo these days. Your paper also lists numerous research opportunities. Can you elaborate?

Pagano: We posed about 50 questions, from dealing with observational uncertainties to exploring advanced AI and machine learning applications in forecast verification.

Bennett: What advice do you have for young scientists in our field?

Pagano: Dive into our research questions. These will be a treasure trove for anyone looking for a topic that will help us operationally. And I think most questions aren’t just weather related, they apply to hydrology as well.

For example, there are literally hundreds of publications about effective communication of probabilistic information, but nearly none about communicating verification information to users. The only one we found was in a river forecasting context.  

But more generally, when I was a grad student around 25 years ago some of my professors cautioned me against working in an operational environment. They worried that doing so would take me off the academic career ladder. But I think things are different now, initiatives like HEPEX show that the community values operational perspectives and experience. Plus, being in an operational environment can be tremendously satisfying when you see your work in action, directly affecting peoples’ lives.

Bennett: I agree that operationalisation research is very satisfying.

Pagano: And there’s no way to describe the feelings of being on the operational desk when major events happen.

Bennett: What’s next for you and the team?

Pagano: The article has a whole section on calls to action, to help agencies climb what we call the Verification Information Value Ladder. Among other things, we’re hoping to foster an international verification community. HEPEX has been a very successful model, bringing together natural and social science academics, operational agencies, forecast users and others. There are opportunities for collaboration, testbeds and other focused effort to help us traverse the “Valley of Death” in transitioning research to operations. We hope that HEPEX members get involved! 

I’m personally about to start the Australian equivalent of a sabbatical. I’m looking forward to lots of scuba diving, hiking and who knows what else. I’d love to connect some HEPEX members, especially in South America and South Asia (feel to reach me at Thomas.C.Pagano@gmail.com while I’m away, or tom.pagano@bom.gov.au later).  

Bennett: Enjoy your time off, and thanks for sharing your insights!

Pagano: Thank you, I’m looking forward to seeing how our work helps shape better forecasting practices!