Using the Slurm REST API to integrate with distributed architectures on AWS

The Slurm Workload Manager by SchedMD is a popular HPC scheduler and is supported by AWS ParallelCluster, an elastic HPC cluster management service offered by AWS. Traditional HPC workflows involve logging into a head node and running shell commands to submit jobs to a scheduler and check job status. Modern distributed systems often use representational state transfer (REST) API operations to programmatically communicate between system components. This blog post provides details on the high-level design and use of the Slurm REST API with AWS ParallelCluster, and how you can use it to integrate HPC workloads securely and elastically with other AWS services. After reading this blog post, you will be ready to integrate HPC workflows into a distributed cloud-based architecture.

AWS services are built with API operations. Many of these API operations use HTTP as their communication protocol and function as REST API operations. When connecting Slurm to these services in a distributed architecture, the Slurm REST API provides a more native integration than the alternative method of invoking shell commands in a head node. This allows you to design integrations in a scalable and secure way.

SchedMD provides documentation and support for Slurm while maintaining its open source implementation. Part of the Slurm package supported by SchedMD is the REST API. The Slurm REST API was originally implemented as v0.0.35 in Slurm 20.02. As of this writing, the most recent stable Slurm release is 20.11 and includes v0.0.36 of the REST API. Past versions of the REST API are included in each Slurm release as separate endpoints for backward compatibility and are marked deprecated when they intend to be removed in subsequent versions. This makes it a sustainable interface that maintains compatibility while being improved and expanded.

The first section of this blog outlines the functionality of the Slurm REST API. The second section presents a brief overview of integrating custom private API operations into your AWS architecture. The third section provides example solutions that use the Slurm REST API and AWS ParallelCluster to provide HPC capabilities in a distributed application. The example solution architectures are:

Collect metrics from Slurm in Amazon CloudWatch
Cluster monitoring with AWS Amplify
Run HPC jobs with zero-touch using Amazon Simple Storage Service (Amazon S3)

The design patterns used in these examples can easily be extended to other Slurm-based clusters on-premises or in hybrid environments. For assistance building with Slurm, SchedMD offers professional services in AWS Marketplace to help customers create advanced HPC solutions.

Slurm REST API architecture

In this section, we discuss the REST API architecture from the perspective of scalability, flexibility, and security.

Figure 1. Architecture of Slurm and user workflows, demonstrating two methods of interacting with Slurm. In the first method, the user accesses the Head Node via SSH and runs helper scripts like sinfo, squeue, sbatch, and scontrol. In the second method, the user issues REST API calls through HTTP to slurmrestd.

Scalability

The Slurm REST API is provided through a daemon named slurmrestd. It functions adjacent to Slurm command line interface applications (sbatch, sinfo, scontrol, and squeue) so that Slurm can be interacted with by both interfaces. A Slurm cluster is controlled by the Slurm controller daemon running on the head node (slurmctld), while slurmrestd functions only as the REST API interface. slurmrestd functions synchronously with slurmctld. This means that a request is only considered complete after the HTTP response code is sent. slurmrestd is also stateless because after the request is complete any state associated with a request is discarded. These features allow it to function as a highly scalable interface.

Security

Slurm has traditionally functioned with authentication provided by Munge, which is based on authentication by the UID and GID of a process calling Slurm. With the addition of slurmrestd, JSON Web Tokens (JWTs), an open standard RFC, were added as a new means of authentication. In Slurm, JWTs can function as authentication for slurmctld and slurmdbd. Any call to slurmrestd must include a JWT, which is passed to these daemons for authentication.

To use JWTs as Slurm authentication, you must configure Slurm to use them, create a JWT using scontrol, and provide the JWT to Slurm when using the API or CLI. The JWT functions similarly to Munge authentication in that it is associated with a user defined by a UID/GID. Due to this symmetry, both JWT and Munge authentication can be used concurrently and the familiar Slurm user/group definitions are not changed.

The Slurm REST API is intended for use in a distributed architecture, but is not intended to be externally facing. REST API traffic should be TLS wrapped outside of trusted networks as it does not contain HTTPS support by default.

Flexibility

The REST API contains many useful commands that can be used to control the cluster and collect data. The REST API definitions follow the OpenAPI 3.0.2 standard. Up-to-date definitions are available in the SchedMD Slurm documentation or by running a GET request to /openapi. Two main endpoints are available: /slurm and /slurmdb. The /slurm endpoint allows the requestor to submit jobs and perform query functions of the clusters, nodes, jobs, and partitions. It provides the base functionality of the traditional CLI applications sinfo, sbatch, and squeue. The /slurmdb endpoint allows the requestor to access the functionality of slurmdbd, including running queries to the Slurm accounting database to perform tasks including modifying cluster configuration and user accounts. The variety of API functions allow the vast majority of Slurm functionality to be consumed via the REST API. This gives flexibility to solutions when integrating with the API.

Using the API

Let’s look at an example of a common function in Slurm, submitting a job to a cluster. To do so, the submitting entity will issue a POST request to /slurm/v0.0.36/job/submit with an example payload given by:

{
    "job": {
    "name": "test",
    "ntasks":2,
    "nodes": 2,
    "current_working_directory": "/home/ubuntu/test",
    "standard_input": "/dev/null",
    "standard_output": "/home/ubuntu/test/test.out",
    "standard_error": "/home/ubuntu/test/test_error.out",
    "environment": {
        "PATH": "/bin:/usr/bin/:/usr/local/bin/",
        "LD_LIBRARY_PATH": "/lib/:/lib64/:/usr/local/lib"}
    },
    "script": "#!/bin/bash\necho 'I am from the REST API'"
}

This will run the provided script to print “I am from the REST API” in an output file test.out. The definition of the script to run (given in the script field) requires newline characters be used. The standard input, output, error, and environment fields must be set appropriately when submitting jobs via the REST API. The remaining fields of name, ntasks, and nodes are the familiar fields from running sbatch scripts and correspond to the job name, the number of tasks and the number of nodes to run the job on. While this may initially seem more difficult than running an sbatch script, it is important to note that the REST API is not intended as a direct user interface. You will likely not write the API payload, but will instead interact with a user-facing application. This could be in the form of another service or a client library. As of writing, there is no client library available for the Slurm REST API. However, generators like OpenAPI Generator exist which can create client libraries for many languages.

Two AWS services should be highlighted that can be used to interact with the Slurm REST API and easily integrate HPC in a distributed systems architecture. First, Amazon API Gateway can act as the front door for the REST API and can refactor the input or output before sending it upstream or downstream. This effectively customizes the endpoints and payloads. It can also act as a caching layer, preventing repeated API calls to Slurm in the event of network issues. Second, AWS Lambda is a compute service that lets you run code without managing servers. Using Lambda functions, the Slurm REST API can be called by directly using REST or by code written with generated client libraries. User-defined Lambda function can be initiated by an event or set to run on a schedule. With the Slurm REST API, Lambda functions can be written to collect cluster health metrics, submit jobs, or monitor usage.

Use case examples with AWS ParallelCluster

Collect metrics from Slurm in Amazon CloudWatch

In this reference architecture, a Lambda function is used to issue REST commands that collect metrics on the jobs, nodes, and users. It then exports those metrics to CloudWatch. The Lambda function can be set up to run on a schedule like a cron job and export the metrics according to a preset frequency.

Figure 2. An example architecture which uses a Lambda function to interact with slurmrestd on the Head Node in order to publish Slurm metrics to Amazon CloudWatch.

Figure 2. An example architecture which uses an AWS Lambda function to interact with slurmrestd on the Head Node in order to publish Slurm metrics to Amazon CloudWatch.

Cluster monitoring with AWS Amplify

Another way of interfacing with a REST API is through API Gateway. In this example, it is used to refactor the output of data collected from the REST API on job status and node health. This refactored data can be displayed on an Amplify dashboard to access via mobile devices.

Figure 3. An example architecture which uses API Gateway to interface a dashboard constructed with Amplify with slurmrestd in order to control and monitor Slurm via a UI.

Zero-touch HPC jobs through Amazon S3

Beyond collecting cluster data, the Slurm REST API can be used to submit and control jobs. In this example architecture, a batch computing pipeline is created using the Slurm REST API. S3 bucket event triggers are used to call a Lambda function when an sbatch file is uploaded to an S3 bucket. The Lambda function makes an API call to submit a job using the REST API. An Amazon FSx for Lustre filesystem exports the job output data to an output S3 bucket at the completion of the job so that you can access your results easily.

Figure 4. An example architecture which uses a Lambda function to interact with slurmrestd in order to automatically create Slurm jobs as job scripts are added to an input S3 bucket.

Figure 4. An example architecture which uses an AWS Lambda function to interact with slurmrestd in order to automatically create Slurm jobs as job scripts are added to an input S3 bucket.

Conclusion

In this blog post, we have introduced the Slurm REST API, described high-level architecture, explained general usage, and highlighted several example use cases with AWS ParallelCluster. SchedMD, the maintainers of Slurm, provide support and maintenance of the Slurm REST API. SchedMD also offers professional services in AWS Marketplace. AWS customers can obtain SchedMD support for Slurm in their development of cutting edge HPC solutions in distributed systems.

To learn details on how to set up the Slurm REST API in AWS ParallelCluster, interested readers can explore the AWS HPC Workshops and read the Slurm REST API documentation from SchedMD. Learn more about enterprise-level support and training for Slurm, including the Slurm REST API, by reading about support offerings from SchedMD on the AWS Marketplace. With these resources, you will be able to use the Slurm REST API and AWS ParallelCluster to effectively integrate HPC workloads into your cloud solution.

AWS HPC Blog