AWS Storage Blog

Building an HPC cluster with AWS ParallelCluster and Amazon FSx for Lustre

High performance computing (HPC) customers love the breadth of services offered by AWS and the flexibility offered by the cloud to address their computational challenges. AWS provides you with the opportunity to innovate quickly and accelerate your workflow thanks to a virtually unlimited capacity. For more information, see the following posts:

However, building an HPC system can be complex, which is one reason AWS created AWS ParallelCluster. This tool offers a simple method to create an automatically scaling HPC system in the AWS Cloud, utilizing services such as Amazon EC2, AWS Batch, and Amazon FSx for Lustre. AWS Parallel Cluster configures the compute and storage resources necessary to run your HPC workloads.   Settings based on your specific storage, compute, and network requirements all help optimize its functionality.

You can manage an HPC cluster with AWS ParallelCluster through a simple set of commands: create, update, and delete. The compute portion of the cluster automatically scales when jobs have to be processed. As a result, you only pay for what you use. Additionally, to provide shared storage to the cluster, AWS ParallelCluster can use Amazon FSx for Lustre as a high-performance file system that can be accessed by all compute nodes.

In this post, I show how easy it is to spin up a fully functioning compute cluster with a shared, high-performance file system in a matter of minutes using AWS ParallelCluster and Amazon FSx for Lustre. To test file system throughput, which is a common performance metric for sizing HPC clusters, you’ll use the SGE scheduler (pre-installed with AWS ParallelCluster) once your cluster is up to run a quick I/O benchmark job. The I/O benchmark will also verify that the compute cluster can achieve the expected level of file system performance. If you’re new to AWSParallelCluster, review this post detailing how to get started before continuing.

Overview

AWS recently announced AWS ParallelCluster support for FSx for Lustre. AWS ParallelCluster is a cluster-management tool for AWS that makes it easy for scientists, researchers, and IT administrators to deploy and manage HPC clusters in the AWS Cloud. This tool also facilitates both quick-start proofs of concept (POCs) and production deployments.

FSx for Lustre provides a high-performance file system that makes it easy to process data stored on Amazon S3 or on premises. With FSx for Lustre, you can launch and run a file system that provides sub-millisecond access to your data. It allows you to read and write data at speeds of up to hundreds of gigabytes per second of throughput and millions of IOPS.  By integrating AWS ParallelCluster with FSx for Lustre, HPC customers can now spin up clusters with thousands of compute instances, and shared storage that scales performance for the most demanding workloads so that compute instances never have to wait for data and you get to the answers faster.

To test the performance of your FSx for Lustre file system, you can use IOR, a commonly used HPC benchmarking tool. IOR provides a number of options that approximate workload performance. However, I always recommend that you benchmark with your own workload to get full visibility into performance characteristics.

Right-sizing

Use IOR to generate a sequential read/write I/O pattern to test the high-throughput capabilities of FSx for Lustre. This means that you begin your sizing at the storage layer, and then determine how many compute nodes should drive your desired I/O performance.

When sizing FSx for Lustre performance, a good rule of thumb is to plan for 200 MB/s of throughput per 1 TiB allocated. FSx for Lustre allocates file systems in 3.6-TiB increments, which provide around 720 MB/s per increment. A 39.6-TiB file system, should give you around 7.92 GB/s of total file system throughput.

After you know the storage performance target, you can figure out how many compute nodes you need. Each compute node communicates with FSx for Lustre over the network, making the workload throughput-based, and so network bandwidth is the primary sizing factor for compute nodes.

Amazon EC2 instances provide different levels of network bandwidth, depending upon the instance type. In this case, I selected the c4.4xlarge instance type because it is widely available in all Regions and, in testing, provides about 500-MB/s peak network bandwidth. To reach 7.92 GB/s of performance, you want sixteen c4.4xlarge compute nodes in your cluster.

Prerequisites

You must have the AWS CLI installed and configured on a system with Python and PIP. This could be on your laptop, a desktop computer, or an Amazon EC2 bastion host running in AWS. Make sure that you are running the latest version of the AWS CLI by running the following command:

$ pip install --upgrade awscli

If you already have the AWS ParallelCluster CLI installed, run the following command to make sure you are using the latest version:

$ pip install --upgrade aws-parallelcluster

Defining the cluster

AWS ParallelCluster uses a configuration file to define how the cluster should be created and initialized. Use the following configuration file to create the cluster:

[global]
cluster_template = default
update_check = true

[aws]
aws_region_name = us-east-2

[cluster default]
key_name = <EC2-KEY-NAME>
initial_queue_size = 0
max_queue_size = 16
min_queue_size = 0
placement_group = DYNAMIC
cluster_type = spot
master_instance_type = c4.4xlarge
compute_instance_type = c4.4xlarge
vpc_settings = myvpc
ebs_settings = myebs
fsx_settings = myfsx

[vpc myvpc]
vpc_id = vpc-<VPC-ID>
master_subnet_id = subnet-<SUBNET-ID>

[ebs myebs]
shared_dir = /shared
volume_type = gp2
volume_size = 100

[fsx myfsx]
shared_dir = /fsx
storage_capacity = 39600

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}

Global configuration

This section reviews some of the previous parameters. For a complete description of the parameters, see the AWS ParallelCluster post on the AWS Open Source Blog.

  • aws_region_name – Name of the Region where the cluster is created. In this case, I used us-east-2 (Ohio) because it’s one of the Regions that supports FSx for Lustre.
  • key_name – Name of an existing Amazon EC2 key pair, from the specified Region. It is used for Secure Shell (SSH) access to the cluster master node.
  • initial_queue_size – Initial number of compute nodes attached to the cluster when created.
  • max_queue_size – Maximum number of compute nodes created using Amazon EC2 Auto Scaling when jobs are submitted.
  • min_queue_size – Minimum number of compute nodes in your cluster at any given time.
  • cluster_type – Defines if Amazon EC2 On-Demand or Spot Instances are to be used for the compute portion of the cluster. The Spot price is capped at the On-Demand price but you can change that setting.
  • compute_instance_type – EC2 instance type to be used for the compute nodes. In this case, choose c4.4xlarge to get a good balance of CPU performance and network performance. Evaluate different instance types and sizes with your HPC workloads and select the one that makes the most sense.
  • shared_dir – Directory created on the master node and exported via NFS to all other nodes in the cluster. It provides a shared space that can be used to store applications or static files.
  • vpc_id – ID of the VPC, in <aws_region_name>, where the cluster deploys.
  • master_subnet_id – ID of the subnet, in <aws_region_name>, where the cluster deploys.

FSx for Lustre configuration

The new configuration option fsx_settings gives you the following abilities:

  • Create a new FSx for Lustre file system and attach it to the cluster.
  • Attach a pre-existing FSx for Lustre file system.

The [fsx] section in the configuration file specifies the options for the file system. In this case, the system creates a 39.6-TiB file system and mounts it under the /fsx folder on all nodes.

FSx for Lustre can also load datasets from Amazon S3 when it creates a file system. This allows you to store data durably, reliably, and at a low price point in Amazon S3. The file data will be loaded from S3 into FSx for Lustre the first time the files are read.  Subsequent reads of the file will pull data directly from the FSx for Lustre file system. To initialize your FSx for Lustre file system with data from Amazon S3, use the following settings under the [fsx] configuration section:

  • import_path – This specifies an S3 bucket and path to import data from (for example, s3://mybucket/mydata). When FSx for Lustre creates its file system, it scans the objects in this path and creates corresponding file metadata in Lustre. The actual object data is not loaded into the file system until the first time it reads the file.
  • export_path – To export data from the file system back to Amazon S3, you can specify a path, which must be in the same bucket as the import_path.

For more information, see Configuration Options.

Creating the cluster

Now that you’ve defined the configuration file, save it to ~/.parallelcluster/config. Then create the cluster using the following command:

$ pcluster create mycluster
Beginning cluster creation for cluster: mycluster
Creating stack named: parallelcluster-mycluster
Status: parallelcluster-mycluster - CREATE_COMPLETE
MasterPublicIP: 18.221.232.178 
ClusterUser: ec2-user
MasterPrivateIP: 172.31.12.244

Alternatively, place the configuration file anywhere you like and use the following command, where myconfig is the name of the file containing your AWS ParallelCluster configuration:

$ pcluster create mycluster -c myconfig

It takes a few minutes to spin up the cluster, and create and initialize the FSx for Lustre file system. AWS ParallelCluster verifies that the provided configuration file is correct. Then, it generates an AWS CloudFormation template to build the required infrastructure for an HPC system in AWS with automatic scaling capabilities. For more information about the services that AWS ParallelCluster creates, see How AWS ParallelCluster works.

The cluster creation process creates a single master node. When you connect to the cluster using SSH, you land on this node and manage your jobs from it. You can connect to the master node using the following command:

$ pcluster ssh mycluster -i <path to your EC2 SSH private (.pem) key file>

Installing IOR

For your test, install the IOR benchmark to the shared directory on all of the compute nodes in the cluster by running the following script on the master node:

#!/bin/bash

# load MPI wrappers
module load openmpi
mkdir -p /shared/hpc/performance/ior
git clone https://github.com/hpc/ior.git
cd ior
git checkout 3.2.1
./bootstrap
./configure --with-mpiio --with-lustre --prefix=/shared/hpc/performance/ior
make
sudo make install

Running IOR

Use the SGE command line tools to launch your IOR job across the cluster with the following shell script:

#!/bin/bash
#$ -V
#$ -cwd
#$ -N myjob
#$ -j Y
#$ -o output_$JOB_ID.out
#$ -e output_$JOB_ID.err
#$ -pe mpi 256

module load openmpi
mkdir /fsx/ior
OUTPUT=/fsx/ior/test
export PATH=/shared/hpc/performance/ior/bin:$PATH

mpirun -np ${NSLOTS} ior -w -r -B -C -o $OUTPUT -b 5g -a POSIX -i 1 -F -t 4m

The script option -pe mpi 256 tells SGE to use the mpi parallel environment and to allocate 256 slots across the cluster. Because there can be up to 16 compute nodes and each c4.4xlarge has 16 vCPUs, there are 16 IOR processes allocated per node. Each process writes and then reads a 5-GiB file (using 4-MiB I/O transfer sizes) using the following IOR flags:

  • -B  Use O_DIRECT to bypass system library caching and directly access the file system. This provides raw file system performance metrics.
  • -a  Define the I/O access interface to use. In this case, select POSIX as the method.
  • -F  Create one file per process.
  • -C Change the task ordering to n+1 ordering for read-back. Use to avoid read cache effects on client processes.
  • -b Define the size of the file to create, per process.
  • -t  Define the I/O size to use for each read/write operation.

Run the following command on the master node to submit your job to the cluster, where myjob.sh is the name of the earlier shell script:

$ qsub myjob.sh
Your job 1 (myjob") has been submitted

You can monitor the status of the job using the qstat command:

$ qstat
job-ID prior   name  user     state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
1      0.55500 myjob ec2-user r     03/15/2019 21:57:27 all.q@ip-172-31-5-228.us-east- 64

After a few minutes, compute instances are available in your cluster through automatic scaling and the IOR job runs automatically. For a description of the workflow, see AWS ParallelCluster processes.

Example of results

Using the shell script, the output of the run goes to a file called output_$JOB_ID.out, where $JOB_ID is the job ID as assigned by the job scheduler. From the qstat output shown earlier, your job ID was “1”. The output from your test run follows:

[ec2-user@ip-172-31-23-169 ~]$ cat output_1.out
IOR-3.2.1: MPI Coordinated Test of Parallel I/O
Began               : Tue Jul 16 14:27:19 2019
Command line        : ior -w -r -B -C -o /fsx/ior/test -b 5g -a POSIX -i 1 -F -t 4m
Machine             : Linux ip-172-31-23-3
TestID              : 0
StartTime           : Tue Jul 16 14:27:19 2019
Path                : /fsx/ior
FS                  : 36.3 TiB   Used FS: 0.0%   Inodes: 167.7 Mi   Used Inodes: 0.0%
Options:
api                 : POSIX
apiVersion          :
test filename       : /fsx/ior/test
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 256
clients per node    : 16
repetitions         : 1
xfersize            : 4 MiB
blocksize           : 5 GiB
aggregate filesize  : 1.25 TiB

Results:
access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ---------- ---------  --------   --------   --------   --------   ----
write     9115       5242880    4096       0.039549   143.79     85.61      143.79     0
read      8636       5242880    4096       0.048285   151.77     80.74      151.77     0
remove    -          -          -          -          -          -          0.029827   0
Max Write: 9115.26 MiB/sec (9558.04 MB/sec)
Max Read:  8636.00 MiB/sec (9055.50 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        9115.26    9115.26    9115.26       0.00    2278.82    2278.82    2278.82       0.00  143.79401     0    256  16    1   1     1        1         0    0      1 5368709120  4194304 1310720.0 POSIX      0
read         8636.00    8636.00    8636.00       0.00    2159.00    2159.00    2159.00       0.00  151.77395     0    256  16    1   1     1        1         0    0      1 5368709120  4194304 1310720.0 POSIX      0
Finished            : Tue Jul 16 14:32:15 2019

To summarize:

  • You launched 256 processes total across the cluster, 16 per node.
  • Each process created a 5-GiB file on the FSx for Lustre file system, reading and writing from the file sequentially in 4-MiB chunks.
  • The aggregate throughput was 9.5 GB/s on writes and 9.0 GB/s on reads.

Cleaning up

When you have your results, you can shut down and delete the cluster using the following command:

$ pcluster delete mycluster

This terminates both the cluster nodes and the FSx for Lustre file system.

Conclusion

In this post, you learned how to use AWS ParallelCluster to create a fully functional HPC cluster. After reviewing the AWS ParallelCluster settings for using FSx for Lustre, you created a file system as part of the overall cluster-creation process. Then, you used IOR to benchmark the file system to show how quick and easy it is to run a job on the cluster.

For more information, see High Performance Computing (HPC). Please let us know what you think in the comments.

Happy building!

Pierre-Yves Aquilanti

Pierre-Yves Aquilanti

Pierre-Yves Aquilanti is a Sr. HPC Specialized Solutions Architect at Amazon Web Services. He is working with Oil & Gas and Autonomous Vehicle customers to run large scale computational workloads in the Cloud. Prior to joining AWS, Pierre-Yves was working in the Oil & Gas industry.

Jeff Bartley

Jeff Bartley

Jeff Bartley is a Storage Solutions Architect at AWS. After 20 years of writing software and leading storage engineering teams, he decided that helping customers have great experiences in the cloud was a lot more fun.