This spring has seen the opening of the new JupyterHub service at ECMWF, a new way of accessing ECMWF resources in an interactive and modern way. With JupyterHub, authorised users can now launch Jupyter sessions on multiple backends, including ECMWF’s high-performance computing facility (HPCF) and the ECGATE Service (ECS). They can thus leverage the computational capacity and data resources available at ECMWF without leaving their browser.
Jupyter is an open-source platform that provides an interactive computing environment where users can manipulate data, run code, and visualise results in real time, fostering a seamless and efficient workflow. It enables users to create and share documents – known as notebooks – that contain live code, equations, visualisations, and narrative text. It supports various programming languages, including Python, R, and Julia, making it a versatile tool for data analysis, machine learning, scientific computing, and more. Jupyter also offers rich visualisation capabilities, allowing users to create interactive plots, charts, and graphs directly within their notebooks.
Use of JupyterHub
The JupyterHub service had been running in beta mode for over six months. Early adopters amongst ECMWF staff identified issues for the new service and refined the final offering before it was made available more generally. The creation of this service addresses a requirement highlighted in a recent internal survey at ECMWF: a great number of scientists expressed their desire for improved and more modern interactive computing, data analysis and plotting capabilities.
In its current configuration, Jupyter users can enjoy one session at a time on either the HPCF or ECS, depending on their permissions, with a fixed duration of up to seven days. Sessions can be tailored to fit any particular requirements for the task at hand. Resources, such as the number of CPUs, memory and temporary storage requirements, can be customised. Once they are active, users can take advantage of the rich software stack readily available for their activities, as well as using their own Conda or Python virtual environments.
Sessions are based on JupyterLab, an evolution of the classic Jupyter Notebook web-based application. It comes with additional features and customisation options to offer a truly interactive remote development environment. Users can leverage JupyterLab's multiple panes to use text editors with syntax highlighting and auto-completion, terminal consoles for direct system interaction and command execution, and image/document displayers for quick inspection of files. Seamlessly integrated with version control systems like Git, JupyterLab promotes collaborative coding endeavours while facilitating reproducible research practices.
The JupyterHub system is designed with flexibility in mind, with a clear separation between the frontend service, responsible for managing user sessions, and the different backends, where those sessions are run. This modular architecture ensures a future-proof solution, with the possibility of expanding our backend catalogue with additional resources, such as those based on cloud technologies.
Overall, JupyterHub offers an intuitive and efficient environment for exploratory data analysis, visualisation, and sharing reproducible research. Its accessibility via a web browser not only enables remote collaboration but also ensures seamless access to computational resources across various devices, underpinning its utility as a cornerstone tool for modern data-driven workflows.
Access to JupyterHub
JupyterHub is currently available to all registered users with access to ECMWF’s HPCF and ECS, and more than 200 of them have already tried it. In order to use those services, non-registered users need to apply for access. People who work for a national meteorological service or scientists/researchers who work for a research organisation within one of our Member States or Co-operating States may follow the standard guidelines to access our computing facilities: https://www.ecmwf.int/en/computing/access-computing-facilities.
You can find more information about the service and how to use it on our JupyterHub documentation page: https://confluence.ecmwf.int/display/UDOC/JupyterHub.