Cinzia Mazzetti, Senior Scientist in EFAS operations at ECMWF
Just before the pandemic struck, the scientific lead for the European Flood Awareness System (EFAS) at the EU Joint Research Centre (JRC) issued ECMWF a major new challenge: to develop a kilometre-scale EFAS, increasing the spatial resolution of the system from the existing 5 km to 1 arcmin (~1.5 km at EFAS latitudes). My role within EFAS operations at ECMWF has put me right at the centre of this major undertaking, which is set to bring improved flood forecasts. Of course, it has only been possible with the collaborative effort of a whole range of experts with whom it has been an absolute pleasure to work.
EFAS provides pan-European flood forecasts to help prepare ahead of major flood events, particularly in large trans-national river basins. Funded by the European Commission through its Copernicus Emergency Management Service (CEMS), EFAS is managed by the JRC. ECMWF is the EFAS computational centre and is in charge of producing forecasts and hosting the EFAS-Information System platform. Around 24 scientists from 12 European countries contributed in some way to the EFAS ‘family’ at ECMWF in 2022.
More than 500 million citizens live in the EFAS domain. Smaller rivers, which are more and more the protagonists of deadly floods, are better represented at kilometre-scale resolution, contributing to an overall improvement of flood forecasts over larger rivers as well. Accuracy increases for both flood peak timing and magnitude.
It also means 14 times more grid cells to calculate, bringing the final count to nearly 5 million, with computational and storage capacity bearing the impact of the new resolution.
After three years of work and an immeasurable number of hours on Microsoft Teams, three critical aspects of the EFAS 2023 upgrade were delivered:
- New geophysical maps
- A new version of the LISFLOOD hydrological model
- A new EFAS calibration
Figure 1: Part of the EFAS domain over northern France, showing the increased level of detail captured by the new 1 arcmin grid (left) compared with the 5 km grid (right).
Upgrading geophysical information and meteorological forcings
Rainfall-runoff modelling is at the heart of EFAS. Its hydrological model LISFLOOD explicitly considers the spatial distribution of physical properties across catchments to provide estimates of river discharge and other hydrological variables. Driven by meteorological forcing data (precipitation, temperature, and evaporation for example), LISFLOOD calculates a complete water balance for every grid cell of the EFAS domain.
LISFLOOD requires a wide range of input data such as topography, soil type and channel geometry. A total of 68 geophysical maps is needed. These data are constantly evolving but had not been updated in EFAS for many years, as generating new geophysical input maps is a daunting and lengthy activity.
We used Earth Engine (EE) and JavaScript to analyse and manipulate global high-resolution (100 m) datasets to generate the maps needed (Figure 2). Although EFAS uses a pan-European domain, the datasets were created globally and for two different spatial resolutions (1 arcmin and 3 arcmin). In the process, the computational resources of EE were pushed to the limit, resulting in frequent emails between ECMWF staff and EE, but at the same time EE and its vast public archive were instrumental in solving issues like storage and processing.
Figure 2: Forest fraction for the LISFLOOD model domain at a resolution of 1 arcmin. Forest fraction is one of the 68 geophysical datasets needed by the model.
Along with geophysical maps, a high-resolution, sub-daily pan-European meteorological dataset spanning from 1990 to 2021 was created by the EFAS Meteorological Data Collection Centre operated by DWD and KISTERS. ECMWF staff helped by enhancing software, workflows, and data quality assurance.
Continental hydrological modelling at kilometre-scale
A kilometre-scale EFAS requires LISFLOOD to carry out calculations at millions of grid cells at each time step, while handling large datasets. This is a challenge that hydrological modellers at ECMWF and the JRC could only tackle with the help of computer scientists. Running a sub-daily 31-year simulation is already intimidating, but it is nothing compared to the calibration of the hydrological model where runs are repeated hundreds of times for each of the nearly 2,000 calibration catchments.
LISFLOOD enhancements for the EFAS 2023 upgrade followed two parallel but interconnected routes.
Hydrological modellers worked on scientific improvements, such as grid-cell by grid-cell computation of water infiltration into the soil and improvements in the modelling of water abstraction for anthropogenic use.
At the same time, computer scientists improved the software efficiency and flexibility. Firstly, profiling and software optimisation techniques were used to identify the parts of the code where larger speed gains were possible. Then, the forcings reader module was completely refactored using modern Python technologies (such as Xarray and Dask) with data chunking and caching approaches, dramatically decreasing the time required to load large inputs.
This joint effort resulted in a version of LISFLOOD 30-times faster than before, in addition to introducing best practice software development, such as continuous integration and unit tests.
Using ATOS computing power for EFAS calibration
Like most rainfall-runoff models, LISFLOOD equations include a range of parameters which require calibration. This is generally done by adjusting parameters to minimise errors between observed and simulated flow over a multi-year period.
During the LISFLOOD calibration, for each river gauge, a population of potential model parameters evolves from one generation (parent) to the next (offspring), and ultimately the best performing parameters are identified according to an objective function (or error quantification).
Each sub-catchment is calibrated separately but using a multi-site cascading calibration approach. The calibration is performed sequentially from upstream to downstream and from the catchments with the smallest upstream area to the largest, as flow routing calculations must be carried out in a serial manner along the mainstream of a river basin.
Model calibration is where hydrologists and computer scientists joined forces again at ECMWF. Nearly 2,000 catchments needed to be calibrated, with very different characteristics. Catchments range from a few hundred grid cells to several thousand, some with just a few years of observations and some with nearly the full 31-year period of meteorological forcing, and with observations at either daily or sub-daily timesteps. And to add to the challenge, many of the calibration catchments were nested into each other, forcing the execution of all upstream catchments to finish before those downstream – a real computational timing nightmare (Figure 3)!
Figure 3: Po river basin, Italy, showing calibration stations (numbered yellow circles), critical path (red) and drainage area for calibration catchments.
The team implemented technical solutions that delivered a timely calibration without compromising on the science:
- River parallelisation allowed multiple rivers to be calibrated at the same time
- Catchment hierarchy to satisfy the physical need for water to run from upstream to downstream catchments
We built dependency trees (Figure 4) to expose catchment dependency and identify the critical path in the calibration process for each river (in red in Figures 3 and 4). The catchments in the dependency tree are processed in layers, starting from head catchments (in orange in Figure 4) to the one furthest downstream (catchment 393 in Figures 3 and 4). Catchments in the same layer have no dependencies between each other and can be calibrated in parallel – computer scientists used the maximum number of jobs available on ECMWF’s ATOS supercomputer to run as many parallel calibrations as possible.
Figure 4: Catchment tree showing the hydrological dependencies for the river Po, Italy. Each number represents a calibration station. Catchments in orange belong to the first layer and calculation can be executed in parallel. Catchments in blue belong to the second calculation layer and can be executed as soon as output from the first calculation layer is available. Catchments on the critical path (in red) are calculated serially. Not all calibration stations shown in Figure 3 are included in the graph to improve readability.
The team used ecFlow (ECMWF client/server workflow package) to optimise the computing time by distributing the workload across ATOS nodes and achieving execution load balancing.
The LISFLOOD 1 arcmin calibration took around 2 weeks to execute, approximately the same time required by LISFLOOD at 5 km, but running on double the number of catchments, some with 14 times more grid cells, and the length of calibration runs increased on average by 42%. A scalability performance that hydrologists would have never achieved without help from the EFAS ‘family’.
The long road to operational kilometre-scale flood forecasting
The road to making the EFAS 2023 upgrade operational is still long, with the team at ECMWF targeting the second quarter of 2023 for the release. There is still a lot of work to do: producing the reanalysis data, generating new re-forecast datasets, evaluating the performance and skill of the new system, just to mention a few.
The EFAS product generation and operational suites will also need some refactoring to meet EFAS key performance indicators, and the system will be thoroughly tested with the help of the EFAS Dissemination Centre and the JRC prior to the official launch.
The EFAS 2023 upgrade is the first kilometre-scale hydrological calibration exercise at ECMWF and although differences exist between LISFLOOD and the land surface model used in the ECMWF Integrated Forecasting System (ECLand), lessons learnt from EFAS will be useful for kilometre-scale Earth system modelling and the Destination Earth digital twins.
Banner image credit: Duncan Cuthbertson/iStock/Getty Images Plus