Enhancing HPC Service Management on Alps using FirecREST API

The functionality and user experience of FirecREST have been significantly augmented by the introduction of pyFirecREST [4]. This Python library streamlines the integration of REST API calls within Python classes, thereby enhancing the ease of use and efficiency in managing HPC workflows.

In Listing 1 we can outline a range of methods defined in pyFirecREST to enhance the integration of the REST API within a script.

Throughout this paper, we will notice the important role that pyFirecREST plays in terms of integration with services that expose their SDKs (Software Development Kits) and APIs using scripting languages.

The utilization of the API through a web interface requires the presence of a dedicated gateway at HPC centers, tasked with managing web-based requests. This gateway serves as a singular access point for each system within the CSCS framework. To accurately direct requests to the intended system, they are uniquely identified by incorporating the machine's name within the request headers as described in Fig. 1, step (2).

Additionally, gateways provide features to prevent abuse of the resources behind the API using rate limiters per endpoint; and enhance access management and security with authorization and authentication plugins, monitoring tools, traffic management, and developer portals.

Regarding identity and access management (IAM), FirecREST API employs the Open ID Connect (OIDC) - OAuth2[3] protocol, utilizing an Identity Provider (IdP) for user authentication. In this framework, CSCS users are required to manage their API keys for accessing FirecREST as shown in Fig. 1, steps (0.a) and (0.b). This process involves authenticating through the IdP to obtain an access token (Fig. 1, step (1)), which is defined by a specific validity scope and application access permissions.

IAM Workflow and Gateway request dispatching diagram from FirecREST to various systems.

The generation and renewal of access tokens can be efficiently automated using the IdP's API and the pyFirecREST library as shown in Listing 2 . This automation facilitates the seamless integration of access token management within user applications, enhancing both the security and usability of the API.

Is important to mention that this specific OIDC workflow is called “client credentials” and is meant for machine-to-machine communication without human intervention when creating or refreshing the access token. This workflow enables the usage of FirecREST by web applications, pipelines, and scheduled tasks facilitating its integration on services operated in this fashion.

Continuous Integration (CI) represents an emerging service tailored for the scientific community, designed to facilitate the testing of software releases across various programming environments and hardware systems. The integration of CI services within HPC centers empowers the community to conduct continuous testing following each code enhancement, thereby fostering a more sustainable software development ecosystem [9].

Establishing CI pipelines for the consistent testing and deployment of scientific software on HPC infrastructure presents several challenges for scientific software engineers. These challenges include:

• source code repository access which needs to be cloned into a node of the target machine, which means that SSH credentials (i.e., username/password, private/public key, or certificates) have to be handled inside the application, raising security concerns.
  
• the SSH session management to keep alive the connection during the pipeline execution.
  
• overall lack of interfaces from the SSH libraries to provide consistent outputs from diverse executed commands.
  

With FirecREST's assistance, we can tackle these challenges while also applying the same approach across various technologies such as GitLab CI, GitHub Actions, Jenkins CI, etc. Additionally, leveraging the HPC abstraction layer allows us to develop pipelines for diverse architectures and software stacks, facilitating portability and distribution.

As shown in Fig. 2, there is no requirement for a Git repository or a Runner to be installed on the HPC center to execute a pipeline. Instead, only access to the API from a public repository and a pair of API keys is necessary.

CI pipelines using Git repository and FirecREST

To exemplify the integration of an HPC pipeline using FirecREST, we can showcase a GitHub Actions [5] workflow that submits a job through a step of the pipeline using a Python script as shown in Listing 3 .

In the named script, the intention is to assert that the pipeline will fail if the system_name is not available for utilization or if the job submitted fails in its execution on the workload scheduler. In any other case, if the job is completed successfully the pipeline will succeed.

Following the example, the pipeline will be executed each time a commit is pushed to the main branch. This can be seen on the GitHub workflow specification (Listing  4), where the pipeline is configured to install pyFirecREST, obtains the API keys from the secret store of GitHub, sets them as part of the environment of the runner, and runs the tests for a specific system_name.

As an extension to this use case, we will delve into a complete feature developed by CSCS that allows users to utilize CSCS infrastructure to run integration tests of their scientific application hosted on public repositories. This is achieved through the utilization of a CI GitLab-Runner, which operates in conjunction with FirecREST as described in Figure 3.

CICD Ext service at CSCS

This service facilitates the testing of scientific software applications at CSCS by mirroring the public repository of the application onto a GitLab repository managed by CSCS.

Users set up a Git webhook on the CI/CD Ext Middleware platform exposed by CSCS. This setup automatically mirrors commits done on the application repository to the GitLab instance managed by CSCS and transparently executes tests, ensuring continuous integration with dependencies and libraries installed on CSCS systems.

The approach works as a spinoff of the CI pipeline execution showcased earlier, albeit with a centralized service overseen by CSCS. This centralized service enables users to bypass pipeline configuration, thus facilitating usage.

JupyterHub [6] serves as a multi-user hub designed to initiate, oversee, and proxy multiple Jupyter notebooks. Frequently employed in HPC centers, it enables the launching of notebooks directly from the browser onto compute nodes. Scientists use Jupyter to develop proof of concept code, explore datasets, or as an educational tool to teach their communities.

The integration with workload managers within JupyterHub is facilitated by the batchspawner package. This package implements Jupyter notebooks execution for various workload schedulers, such as Torque, and SLURM. When JupyterHub is used with batchspawner, they're typically deployed on a machine with a workload manager configured for the target system. Such a setup introduces significant maintenance overhead. Configuring a system to interface the SLURM controller requires system administrators to set up a scheduler daemon and share a key between a controller and daemons. This complicates the deployment of JupyterHub and restricts the systems on which it can operate.

To address this at CSCS, we've developed a spawner based on FirecREST which offers the same functionality as batchspawner but employs pyFirecREST as a backend. This approach significantly reduces the requirements for deploying JupyterHub on HPC systems since any system able to connect to the web can target compute nodes through the FirecREST service.

In our FirecREST spawner, following the batchspawner implementation, we use the JupyterHub's Spawner class as base and reimplement the start(), poll(), and stop() methods to interact with the target HPC system via pyFirecREST (Listing 5).

An interesting outcome of this integration is that the same OIDC-OAuth2 client used for JupyterHub's Web UI can be used to authenticate with FirecREST to manage the notebook lifecycle. This way, user authentication for both services is significantly simplified. Additionally, this approach allows the creation of different instances of JupyterHub interfacing with different systems by simply changing the X-Machine-Name header to specify the target system, as described in Figure 4.

Jupyter notebook execution using FirecREST

ReFrame [8] is a framework designed for regression testing on HPC systems. Its primary function is to facilitate the periodic testing of key scientific applications, ensuring their operational integrity and consistent performance as a way to assess quality of services to the scientific communities. A typical scenario where ReFrame is employed involves verifying the optimal performance of scientific applications following maintenance, particularly when libraries undergo updates, in an HPC environment.

The installation and maintenance of this service need to be addressed by the administrators of the data center where the software is being tested. With FirecREST, we can broaden the utilization of ReFrame for testing and integration, allowing administrators, maintainers, developers, and users to carry out these tasks from an instance in a public cloud provider.

To understand how to do this, it should be noticed that the pipeline of ReFrame for each test goes through different stages: setup, compile, run, sanity, performance, and cleanup, and that ReFrame presents a Python class for execution on HPC facilities using the SLURM workload manager: the SlurmJobScheduler class.

By extending this class, we can register a new one for scheduling jobs with the help of pyFirecREST (firecrest-scheduler) and mapping the abstract classes submit(), poll(), cancel(), and wait() methods. This is shown in Listing 6 .

This scheduler must be set on the site configuration of ReFrame along with the target system and environment to test. This setup enables its on-demand execution or through a scheduled pipeline for periodic testing.

Utilizing ReFrame in conjunction with FirecREST offers a key advantage: users are not required to log in to the HPC system to perform Quality of Service (QoS) tests. Instead, this process can be initiated from any machine with access to FirecREST. This design ensures flexibility and independence for users, allowing them to seamlessly execute QoS tests without being tethered to the HPC system itself.

Apache Airflow[1], a popular workflow orchestration tool, offers a robust framework for defining, scheduling, and monitoring diverse workflows in particular in the Machine Learning (ML) domain. However, integrating Airflow with HPC workload managers, such as SLURM, presents certain challenges.

A primary obstacle is the absence of seamless integration between Apache Airflow and workload managers, which rely on custom commands and APIs for job submission and monitoring. Consequently, to interact with the HPC system, Apache Airflow must either run on a machine where the workload manager is installed and configured, or Airflow tasks must establish remote login connections to such a machine.

The flexibility of Airflow's operator API makes it particularly convenient for integration with pyFirecREST, enabling the definition of operators, such as file transfers to and from HPC systems as well as job submission (Listing 7) and monitoring. These operators can then be used to construct Airflow Directed Acyclic Graphs (DAGs) (Listing 8), streamlining the orchestration of complex workflows.

In this DAG, when the FileSensor task indicates that a new file has been created by an external process, it will trigger the DAG execution by uploading a couple of input files, submitting a job, and downloading the output on the job finishes. Finally, some post-processing is done.

The outcome of the DAG execution can be seen graphically on the AirFlow instance (Fig. 5). The screenshot shows how the usage of the RESTAPI provides basic operators that can be arranged in different ways to compose the DAGs which are executed via a web application without the need to provide a workflow orchestrator at the HPC premises.

AirFlow using FirecREST operators

The availability of such pyFirecREST operators will open new possibilities for enhancing the scaling and performance of Airflow DAGs, especially in the machine learning domain, where the tool has been widely adopted, as it often involves complex and resource-intensive computations that demand efficient orchestration and coordination and requires access to large-scale HPC systems like Cray EX with Alps as an example.