The elusive optimal decision rule and the impact of forecasting

Contributed by Micha Werner, IHE Delft and Deltares

This winter just passed I was faced with the kind of dilemma of the type I am sure many find all too familiar. I was the designated driver one weekend for my daughter’s hockey team away match.

Parents take turns to drive to away matches, and so I was all set to head off with four excitedly chatting thirteen year-olds. At breakfast, I saw on the internet that KNMI, the Dutch national meteorological agency had earlier that morning issued an amber alert. It had been well below zero for a couple of days and the roads were cold. A warm front from the West was now pushing its way through, and there was a high risk of black ice, leading to very slippery roads and potentially dangerous situations.

All non-essential trips were strongly discouraged, particularly on motorways in the west of the country that had been as yet little used that Sunday morning. The warning indicated that the risks would diminish by 9:30. I had to leave by 9 to make it on time and my route included a section of motorway. What was I to do? Should I heed the warning? Was this trip essential or was it non-essential?

Having professionally dabbled with the concept of costs and losses, the decision seemed obvious. The risks were high, and the consequences of a potential accident severe. So the obvious rational decision was not to go. But there were also consequences to not going. The match would need to be called off if the four girls I had with me did not show. Would the opposing team claim victory for a no-show? That could be embarrassing, and moreover could knock our team out of the top of the pool. Would other parents decide not to go; or rather would they go and be let down by my decision to stay at home?

A furtive discussion on our team WhatsApp group ensued. Finally an agreement was reached. Yes, I explained to a concerned mother, yes I had invested in winter tyres. The decision was made. We were off.

Making decisions based on warning information

This little story reflects the difficulty of making decisions based on warning information. Even when provided with quite specific information such as I received that morning, which not only targeted the road type but also provided specific details as to the timing. This is the level of detail that impact based warnings set out to achieve. Information contained in the forecast that is personally relevant to the consumer of the forecast. And yet making the decision that would seem obvious on paper is not easy in practice.

There has been some research on the process of an individual making a decision based on an advisory of flooding, or severe weather, or other (natural) hazards.

• Does the warning apply to me? Do I trust it?
• Am I inherently risk averse, or rather risk acceptant?
• How is my decision influenced by the potential losses?
• And in making a decision are these losses balanced against the cost of taking measures?

A decision making “serious-game”: the Shopkeepers Dilemma Game

The latter question listed above is the reasoning that underlies the rational decision making principle. It was also the question what was asked participants in a decision making “serious-game” we played at the 2016 European Geophysical Union during the Ensemble Hydro-meteorological Forecasting session. Some readers may also recall playing the game at the 2016 HEPEX conference in Quebec City, Canada, or at the Royal Meteorological Society Conference in Manchester, England. It was also played with a group of students in a flood management course at IHE Delft.

This year’s EGU saw a poster presenting the results, and a paper will further explore the details but is still in preparation (I had of course intended to write the paper earlier, and may have found more time to do so if only I had heeded more warnings and stayed at home, instead of going to weekend sports matches with my kids!)

In the game, three different shop-keepers were presented with a series of seven forecasts, each providing participants with the forecast probability that they and their shop could be flooded. They were asked to make one of three decisions for each forecast; choosing between taking no action; raising temporary defences on the embankment between their shop and the river; or moving their inventory. Except for doing nothing, all actions came at a cost. But flooding also caused a loss.

What was also important was that if the shop could stay open for business then a profit could be made. Obviously the shop could not stay open for business if flooded, or if the decision to move the inventory had been taken. However, if the demountable defences were raised, and these were not subsequently overtopped, then it was business as usual and a profit could be made.

The three shops selected in the game had quite different costs and losses:

• One shop sold Ferraris, with significant losses when flooded, as well as high costs when moving the inventory. But profits when the shop stayed open for business were also high.
• The second type of shop was a grocery store, with lower losses when flooded; but also lower costs to move inventory, and lower profits when staying open for business.
• The third shop sold gravestones. Losses due to flooding gravestones were low; and lower or equal to the cost of raising the defences or moving the inventory. Of course when raising the embankments the shop could stay open for business if the embankments were not overtopped, making a modest profit.

At each step, after the decision on the bulletin was made, the actual outcome of the event was shown and participants asked to tally their results. The winner was the participant who had the lowest expected expense at the end of the game, due either to taking measures, which at times were taken in vain when the forecast flood did not materialise, or due to losses incurred when either no action, or inadequate action was taken.

A couple of interesting patterns

Reviewing the results of the 215 participants that played the game across the four sessions revealed a couple of interesting patterns. I will leave the full assessment of those results to the full paper, but some interesting thoughts came forward that may be worth mentioning.

1. Choices made by those selling Ferraris were more or less as expected. They were keen to act on the forecasts; and the most likely to take action in response to the forecasts, with a clear relationship between the probability of forecasting and the inclination to take action. The forecast was of clear value to them.
2. This was also more or less the case for those with grocery stores, though they did not take action quite as often.
3. What was interesting though was the behaviour of those selling gravestones. Though these was little need to respond to the forecasts for them, as the cost of unnecessary action was very high compared to the losses when no action was taken, the gravestone sellers seemed overly keen to respond to the forecasts. Their behaviour was more or less the same as the grocery store participants. It would appear that they were prepared to take the risk of getting it wrong despite the cost, preferring to invest to avoid being flooded.  Such behaviour is often considered to be risk adverse, influencing decisions to be made that stray from the rule that would be prescribed by the rational cost-loss theory.

But not all decisions made in risky situations are necessarily risk adverse. The quite well known prospect theory, first described by Kahneman and Tversky (1979) outlines how decision makers may be risk seeking, or rather how they may be risk averse, depending on how information informing the risk based decision is framed.

Generally, in the game it seemed that the prospect of being flooded caused the decision to take protective action to be taken more often than necessary. This could also be considered loss aversion. People are prepared to invest in taking measures, just to avoid the displeasure of being flooded, which given the disgruntled mutters of participants in the room when flooded after deciding not to take action, clearly gives rise to negative emotions.

The role of emotion in the framing of a decision

Emotion is clearly an important factor in the framing of a decision, as outlined by Druckman and Mcdermott (2008). They go on to discuss how different emotions may influence decisions. Maybe a bit too much to discuss in a blog post that started short but just kept growing; and definitely a topic that is increasingly out of reach of a hydrologist such as myself. However, they also shed some light on my little dilemma. Clearly, the negative emotion connected to not going to the match, as well as my own preference confidence led to the taking of what was a risk-seeking decision.

Emotions may also lead to paralysis in decision making, resulting in no decision being made. That reminded me of a discussion I recently had with a certain Dr Stephen Hussey. He leads an NGO called the Dabane Trust that works with communities in Southern Zimbabwe. What he has found somewhat confounding is that despite clear and acknowledged signs of oncoming drought conditions provided by a seasonal forecast product, that it was very difficult to change what was considered to be normal behaviour by the communities. They could easily plant a less drought sensitive crop than the traditional maize; but they had always planted maize and changing that was not easily done.  Changing behaviour could perhaps be done if a champion for change could be found in the community, but otherwise it proved to be very difficult.

Of course I need not tell the reader that the true value of any forecast will depend on how that forecast is used to influence behaviour and take protective action. Impact based forecasting has recently emerged as a step towards personalising forecasts and through this influencing what an individual should do to get themselves and their property out of harm’s way in the face of natural hazards. That is no doubt a good thing, but the impact of a forecast ultimately depends on the behavioural change that it influences.

This calls to my mind for branching out to research on behavioural change, working with psychologists, social scientists, and economists, as well as many others; on thinking how to frame forecast information. We cannot do this as hydrologists alone. I myself always like to find analogies from my own experiences in decision making in my personal life, wondering how this may help explain the behaviour of recipients of forecasts.

There were no real issues with the roads that Sunday morning. We left a bit earlier than planned, took caution and arrived well on time. That day the team did reasonably; of the two matches they had to play they won one, and lost the other.

References

• Daniel Kahneman and Amos Tversky, 1979, Prospect theory: An analysis of decision under risk, Econometrica, Vol. 47, No. 2, pp. 263-292
• James Druckman and Rose Mcdermott, 2008, Emotion and the Framing of Risky Choice, Political Behavior, Vol. 30, pp. 297–321