ECMWF is making big strides in preparing weather forecasting for the exascale era of supercomputing by improving efficiency and scalability in all parts of the production and dissemination workflow.
The challenge is to make an increasingly complex forecasting system running at ever higher resolution work efficiently on tomorrow’s massively parallel supercomputing architectures.
Fundamental constraints are the time-to-delivery of forecasts, which must not increase, and supercomputer energy consumption, which must remain sustainable.
A recent international meeting on High-Performance Computing in Meteorology provided an opportunity to take stock.
“Two years ago we didn’t quite know what the systems we are aiming for would look like,” ECMWF’s Deputy Director of Research Peter Bauer told the workshop, which took place at the Centre from 24 to 28 October.
“Now we are in a much better place: we can ask much more specific questions and we can break the problems down.”
Today ECMWF’s Scalability Programme includes several projects dealing with observational data processing, data assimilation, model development, model data processing and coding.
“In the end we’ll have to deal with the entire numerical weather prediction (NWP) workflow,” Dr Bauer emphasised. “The forecasting model is an important part of that but it’s not everything.”
As the forecasting system becomes more complex and the resolution increases, production and dissemination systems will have to cope with a much greater data load. The diagram shows the expected 10-year growth in data along ECMWF’s forecast production workflow.
From coding to big data
ECMWF is working with research centres, computing scientists and vendors around the globe to tackle the scalability challenge.
Several EU-funded scalability projects in which ECMWF is involved were launched in 2015. They address a range of issues, from coding to big data handling.
- ESCAPE, which is coordinated by ECMWF, aims to define fundamental algorithm building blocks by breaking down NWP legacy codes. These ‘weather and climate dwarfs’ will be tested on different supercomputer architectures. The ‘dwarfs’ are expected to be demonstrated for the first time in spring 2017.
- ESiWACE will run demonstrator simulations at a target resolution of just 1 km to estimate the computability of configurations that can address key scientific challenges in weather and climate prediction.
- NEXTGenIO aims to develop innovative solutions to tackle input/output bottlenecks that will develop as high-performance computing moves towards the exascale. A recent NEXTGenIO workshop held at ECMWF reviewed performance analysis concepts and tools.
- EarthServer-2 aims to establish scalable web-based analysis and processing services for multi-dimensional geo-referenced Earth science data. This will help ECMWF to continue to provide full access to the data it holds as data volumes increase.
In addition to these projects funded under the EU’s Horizon 2020 Research and Innovation programme, scalability is an important part of ECMWF’s in-house research activities.
Tests have, for example, shown that there may be scope to reduce the precision of some computations in ECMWF’s Integrated Forecasting System (IFS) without degrading forecast quality, enabling significant savings in computational cost.
“ECMWF’s new ten-year Strategy defines ambitious goals. Achieving them will rely on making substantial efficiency gains across the forecast generation and product dissemination chain,” says Dr Bauer.
“In the three years since the launch of the Scalability Programme, considerable progress has been made. Over the next couple of years, the results of a number of scalability-related projects will become available to help us achieve the goals set out in the Strategy.”
Top image: blackdovfx/iStock/Thinkstock