Software Strategy and Roadmap 2023-2027

Title
Software Strategy and Roadmap 2023-2027
Technical memorandum
Date Published
01/2023
Secondary Title
ECMWF Technical Memoranda
Number
904
Author
Tiago Quintino
Umberto Modigliani
Florian Pappenberger
Sylvie Lamy-Thepaut
Simon Smart
James Hawkes
Iain Russell
Laurent Gougeon
Cristiano Zanna
Baudouin Raoult
Publisher
ECMWF
Abstract ECMWF develops several software packages to support the primary purpose of the Centre, to develop a capability for medium-range weather forecasting and the provision of medium-range weather forecasts to the Member States, as well as the objectives aligned with this purpose1. This strategy is in support of the “ECMWF Strategy 2021–2030”2 in particular the pillar on Science and Technology which aims to develop and make operational use of cutting-edge science and technology and the pillar on Impact which aims to provide exceptional value for money to ECMWF’s Member States. Therefore, many of the software packages are also used by Member and Co-operating States as well as other users to process and analyse ECMWF archived or disseminated data. These software packages have been developed over many years, sometimes decades, and continue to provide a reliable service to ECMWF operations.

As part of our constant drive for efficiency and improvement, ECMWF is reviewing its strategy and roadmap for these software packages. The general principles that underpin ECMWF’s software strategy are as follows:
  • Efficient use of development resources. We aim to continue supporting, with the available development resources, our current services, whilst providing capability to continue to grow both in breadth and depth. To realise this goal, we need to rationalise and be efficient in all development activities, as well as to minimise software maintenance efforts. This will be achieved by modernisation, reducing technical debt, and consolidating functionality so that it incurs less maintenance cost.
  • Software reusability. To consolidate the development efforts, we will merge common functionality into reusable components with well-defined interfaces that encourage usage in a variety of environments, from HPC workloads to interactive end-user sessions.
  • Software componentisation and integration. To enhance reusability, existing software will be refactored into smaller units of functionality. We aim to make these components smaller, more reusable, and simpler to integrate with each other. We also recognise that some of our software does not integrate well with Python, a common environment for our users and the community in general. We want to make these components more integrated with Python.
  • Further use of external software. We aim to strike a good balance between our own development of domain-specific software that is critical to the delivery of ECMWF’s core mission, with the use of well-maintained and supported community software. This will leverage the community interaction to further strengthen our developments whilst allowing us to focus on the functionality that is our core mission.
  • Adoption of open development. We believe that software developed openly with interaction and feedback from the community leads to increased quality. Our experience has shown that the costbenefit is very favourable and leads to enhanced interaction with Member and Co-operating States.
  • Data scalability. One of the drivers of this software strategy is the required improvements in scalability of data handling, as the forecast data sets grow more than quadratically, with increases in resolution and new scientific parameters. We aim to refactor our software stack to improve its use of data in particular stages of the workflow, to minimise expensive data access operations, such as network or I/O operations; and where possible provide the software tools and services to support operating on the data where it is, i.e., a more data-centric workflow.
  • Modernisation to new standards for higher interoperability. Over the years some new standards have emerged, particularly in the landscape of web technologies, that bring higher levels of interoperability between systems within and outside our data centre. This increased interoperability improves access to ECMWF data and is strongly aligned with efforts such as the move to open data and adoption of FAIR principles.
URL https://www.ecmwf.int/en/elibrary/81334-software-strategy-and-roadmap-2023-2027
DOI 10.21957/c6d7df0322
Download citation