Interview with Beth Ebert: Forecast Verification, a decade of progress and impact

 Contributed by Tom Pagano, a former HEPEX guest columnist 

Ten years ago, we interviewed Bureau of Meteorology scientist Beth Ebert about the meaning of ‘good forecast verification’ and how hydrologists could benefit from innovations happening in meteorology. At the end of 2024, she will retire, completing a tremendously impactful career, and so we were interested in her reflections and predictions for the field.

Tom: What’s new in weather forecast verification? What are the current challenges?

Beth: It has been an incredible decade, no doubt. There have been some recent conferences on progress in verification. Perhaps most interesting to hydrologists, there has been fundamental progress in the verification of extremes.

Tom: That’s true, customers are always asking us to focus on ‘when it matters most’, but historically the number of interesting events are very small, and so it’s hard to generalise from those.

Beth: This is where the work on threshold weighted scoring, FIRM, and Murphy diagrams in the last few years have been a breakthrough. The Murphy diagrams are great for saying ‘I am vulnerable to these particular conditions, so which forecast is best for me?’

There was also exciting work coming from the Polar Prediction Project, which had to work through some very thorny problems- sparse problematic observations, for example, or verifying sea ice objects floating around, or evaluating predictions of the ribbon of the sea ice edge.

But I’d say just generally, it’s exciting to see how seriously operational agencies are taking verification. We recently got together 20 key international personnel to publish a paper on the Challenges of Operational Weather Forecast Verification which was recently featured on a HEPEX blog.

Tom: Why should hydrologists be interested weather forecast verification?

Beth: Although it is about weather, hydrologists would resonate with much of what was discussed in the Challenges paper. And many of the innovative verification techniques are generalisable; in some ways it’s just numbers, be it wind, temperature or river height.

Hydrologists have their unique challenges, like the timing of the flood peak or how the flow travels through the river network. But river forecasts are critically dependent on the quality of the rainfall inputs. The rainfall verification for hydrologists needs to address the questions they’re interested in, such as the accuracy of catchment rainfall rather than at points.

Tom: Speaking of interfaces, you have been a consummate collaborator over the years, particularly when it comes to interdisciplinary work. You’ve collaborated with hydrologists, emergency managers, social scientists, fire forecasters, you helped develop a service for predicting thunderstorm asthma, among other things. What does it take to succeed as an interdisciplinary scientist?

Beth: That’s a brilliant question. I found it tremendously rewarding but also challenging because you’re on your own a little. When you’re surrounded by people in your own field, it’s easy to feel like you have it all figured out and the communication is easy. But when you step outside that, you appreciate the enormous complexity of some other people’s challenges- but also the creative solutions they’ve had to come up with to rise to the problem. I can think of the recently published WMO Value Chain Framework (which traces data, to modelling, to communication, to decisions). The social scientists have some great ways to assess how information positively affects decisions. But overall, an unexpected challenge was just developing a common language/glossary. For example, a word like ‘climate’ means many things to many people.

Tom: And you’ve spent a lot of time at the coal face transitioning research to operations. When you’re inside an agency it can be challenging enough to innovate and get things adopted, never mind if you’re an outside academic. How have you been able to cross this ‘valley of death?’

Beth: It helps to put yourself in the place of someone who is going to be using your product and have all the interested parties involved from the start- that includes scientists and users but also system developers and maintainers. And you must be patient, these things take a long time.

Tom: I find the mix of people totally crucial. You need an enthusiastic champion who can keep up the momentum and clear the path.

Beth: Trust is very important as well. My early managers were fantastic giving me a lot of freedom (within some guardrails).

Tom: And who were your inspirations over the years?

Beth: I’ve got a super long list. I gave a farewell seminar on forecast verification a few weeks ago (video, slides) that named those who have had an impact on the field and my career. Many people in HEPEX will know Barb Brown, who gave me indispensable career advice over the years. For example, I felt some loss transitioning from a scientist to a senior manager. But she reminded me that instead of being a member of the orchestra, now I’m the conductor, which I thought was sage advice. And she has worked hard to spread the gospel of verification to our field and others.

Tom: Speaking of gospel, I was thrilled to see your talk included ‘The Forecaster’s Prayer’.

Beth: Australian scientist Graham Mills was always talking with the operational staff, trying to understand and respond to their needs. But he was also very generous when it came to sharing our understanding of the latest advances. We still do that, but he was outstanding at it, and I always hoped I could be more like him.

Tom: The HEPEX community has many inspiring prominent women but also many exciting up and comers. Can you reflect on your experience as a woman in STEM during a time when the field was so heavily dominated by men?

Beth: I’ve been very fortunate. I was able to go part time when I had my two children. That’s a common thing now but it wasn’t in those days. I loved working part-time- I recommend it for anyone, children or no children. It’s inspiring to see how leadership positions at AGU are being taken up by women. The last time I looked there were more women section heads in AGU than men. And women are helping each other, that is so critically important. This can be informal, but also through organisations like Science in Australia Gender Equity (SAGE).

Tom: Do you have any predictions for the field?

Beth: Obviously we’re in the middle of the AI revolution. How do we verify AI models? How can we use AI to derive insights from our verification? The potential there is stunning.

More generally, I feel what’s coming is a diffusion of verification methods to other fields. Already some of our biology colleagues are enthusiastically taking up some of our methods. Some of this is helped along by community software like METplus or ‘scores’.

Tom: And what’s next for you personally?

Beth: Ha! Retirement, a well-deserved break. But I’d like to stay involved, such as in AMOS, our Australian equivalent to AGU or EGU. And maybe some small bits of work in the Bureau of Meteorology if there’s an opportunity.

Tom: Well, either way, I feel that there’s some giant shoes to fill. Thanks for your time!

Beth: Cheers.