How writing an article can come out of the blue (KGE on log-transformed flows: a bad idea?)
Contributed by Léonard Santos (Irstea, France).
It is common to read articles in which the Kling and Gupta Efficiency (KGE, Gupta et al., 2009) or its modified version (KGE’, Kling et al., 2012) are used as a metric to evaluate the quality of streamflow simulations. They are often seen as a solution to substitute the Nash and Sutcliffe Efficiency (NSE, Nash and Sutcliffe, 1970).
However, are these two criterion totally comparable? Can the KGE be used exactly in the same way as the NSE is?
I am currently ending my PhD at Irstea (France) and during this PhD I was faced with the questions above. Let me tell you the story of how the work on a big amount of data can lead to better understanding some hidden scientific questions.
This story began one year ago. I was working on model development and I tried to evaluate my work on a wide set of catchments (i.e., 650 gauged stations over France). In order to analyse the model performances on low-flows, I decided to calculate the KGE’ on the log-transformed flows. This choice seemed logical as the same transformation is used to analyse low-flows with the NSE. It was also used with the KGE by some authors. I thus trusted the ability of this metric to represent the quality of low-flows simulations.
Some doubts came when I noticed some highly negative KGE’ values on log-transformed flows in some catchments. I knew that a good performance can be inferred when the KGE’ value is over 0.8. But what is the difference between a value of -0.1 and a value of -165? In addition, how a value of -165 is even possible?
Starting from this observation, I tried to analyse the mathematical formulation of the KGE’ in order to understand these highly negative values. I then noticed that the KGE’ value is unstable when the mean log-transformed streamflow (observed or simulated) is close to zero. This is more likely when the logarithm of flow is used than when no transformation is applied (because the logarithm of 1 is equal to 0 and an average flow of 1 is more likely than an average flow of 0).
The next step was then to speak about this issue with my colleagues, the members of the Catchment hydrology research group at Irstea in Antony: “Hey, has anybody ever faced this issue?”
And the answer was… they had not!
However, the discussion allowed to understand that the issue was even worse. The discussion with my colleagues can be summarized by the following quotes from different members of the team:
- Alban De Lavenne (in his office): “Yes, it is a problem, but I wonder why we do not use the nth root of the flow, it allows to avoid issues with zero flows.”
- Maria-Helena Ramos (in the metro): “If you use litres per second instead of cubic meters per second, it will solve the issue! :-)”
- Laure Lebecherel (before the metro, in the bus): “Wait a minute… The metric value is not expected to change when the flow dimension is modified???”
As a result of these fruitful discussions, I noticed that, in addition of being unstable, the KGE and the KGE’ criteria are not dimensionless when the logarithm transformation is used.
After that, given that the logarithm transformation is used in some papers, my “senpai” (Dr. Guillaume Thirel) decided that we needed to publish on the findings to warn the hydrological modellers’ community. We chose to make a technical note, which seemed well adapted to this type of manuscript.
We submitted a manuscript to HESS journal. It was quite well received by both editor (Bettina Schaefli) and reviewers (Lieke Melsen and Björn Guse) and the open discussion even provided an interesting solution to replace the logarithm (see, mainly, the comments by John Ding).
However, the lesson to be learnt here is not only that the use of the logarithm transformation to calculate the KGE needs to be used carefully.
The main purpose of this blog post is to underline that this study and the results obtained were possible because I was using a great amount of data (i.e., a large data set of catchments). It thus remind us of the usefulness of using large dataset in hydrology.
Also, it shows that a paper is not necessarily the result of a long-term publication plan. It can also come from a side issue arising during the research work and, most importantly, after openly discussing with colleagues and exchanging ideas.
Gupta, H. V., Kling, H., Yilmaz, K. K., and Martinez, G. F.: Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling, J. Hydrol., 377, 80–91, 2009.
Kling, H., Fuchs, M., and Paulin, M.: Runoff conditions in the upper Danube basin under ensemble of climate change scenarios, J. Hydrol., 424–425, 264–277, 2012.
- Santos, L., G. Thirel, and C. Perrin: Technical note: Pitfalls in using log-transformed flows within the KGE criterion. Hydrol. Earth Syst. Sci., 22, 4583–4591, 2018.