After the earthquake that hit Croatia in 2020 and damaged its headquarters in Zagreb, the Croatian Meteorological and Hydrological Service (DHMZ) started a process to relocate its offices and data centre to new premises. This project reached an important milestone in July 2021, when all the remaining IT infrastructure was successfully moved to its new home. During the relocation of DHMZ’s highperformance computing facility (HPCF) and archive, ECMWF’s own HPCF and the European Weather Cloud were used to bridge the gap. This arrangement ensured DHMZ could carry on with its official duties, providing forecast services without any noticeable disruption to its end users. The solution built on the experience gained after the disaster occurred, when an emergency backup had to be implemented in a matter of days (see ECMWF Newsletter No. 164). This time, however, the schedule for the move was well known, so everything could be appropriately organised, prepared, and tested weeks in advance.
ECMWF’s HPCF and the European Weather Cloud
To ensure that all users continued to receive prognostic products, the backup prognostic system at ECMWF’s HPCF was reactivated. It consists of two ALADIN model configurations: one with a horizontal grid spacing of 4 km across a larger domain, and the other with a horizontal grid spacing of 2 km across a smaller domain, covering the broad area of Croatia. The initial conditions for both configurations are obtained from an unperturbed member of A-LAEF (ALARO - Limited Area Ensemble Forecasting), based on ALADIN’s canonical model configuration ALARO, while coupling files come from the high-resolution system (HRES) of ECMWF’s Integrated Forecasting System (IFS), taking advantage of data locality as both are produced on ECMWF’s HPCF. On this occasion, the list of products generated by these suites could be extended to satisfy the requirements of almost all DHMZ end users.
For the first time at DHMZ, all post-processing routines were implemented using ecFlow running on ECGATE. It was seen as a great opportunity to design, implement and test an ecFlow suite in a production context. The experience has been very positive, and DHMZ plans to extend the use of ecFlow to all operations in the future. Additionally, the suite was promoted to enjoy time-critical priority on ECMWF’s HPCF during the preparation and the active backup period. This ensured minimal waiting times and timely production and delivery of products to the end users.
Several weeks before the relocation of computing infrastructure, a first notice was sent to customers. After that, new products were tested by some of the more advanced customers. Since some products for internal DHMZ users were stored on servers that were about to be moved, it was agreed that they would move to new locations. One location was a server located in the new DHMZ headquarters, and the other was a virtual machine on the European Weather Cloud. For external customers, the switch to the backup solution could be made completely transparent. DHMZ received a lot of appreciation from end users, who did not even feel this change in their weather-related applications, and showed that even in such challenging moments it can provide uninterrupted support.
Successful migration
The magnitude 5.5 earthquake on 22 March 2020, which was followed by eight strong aftershocks above magnitude 3.0 in the following days, caused considerable damage to the early 19th-century building hosting DHMZ in Zagreb at the time. The building was deemed unsafe to work in, but staff managed to carry on with their duties remotely and deliver the service to the public, relevant authorities, and critical end users. Several months after the earthquake, the relocation of employees to a new location was organised. However, most of the IT equipment continued to operate in the old building until it could finally be safely moved to the new premises. The new premises for the supercomputer system and archive system (tapes and disks) are in a modern data centre, while other servers were moved to the new DHMZ headquarters.
Thanks to close liaison and careful planning between DHMZ and ECMWF teams throughout this delicate process, the migration completed successfully without any service interruption. From DHMZ’s side, in addition to the authors of this article, several other colleagues should be acknowledged for their hard work before, during and after the relocation: S. Panežić, A. Šljivić, M. Hrastinski, I. Odak Plenković and I. Muić.