How do I mount an existing Amazon FSx for Lustre file system to AWS Batch in a managed compute environment?

Last updated: 2020-09-11

I want to mount an existing Amazon FSx for Lustre file system to AWS Batch in a managed compute environment.

Short description

To mount an existing Amazon FSx for Lustre file system to your containers without creating custom Amazon Machine Images (AMIs), you can use an Amazon Elastic Compute Cloud (Amazon EC2) launch template with AWS Batch.

Important: When you create an Amazon FSx for Lustre file system, use the same Amazon Virtual Private Cloud (Amazon VPC) and subnets that are assigned to your compute environment.

Resolution

1.    Create an Amazon Fsx for Lustre file system, and then choose either a persistent or scratch file system deployment type.

Tip: Use persistent file systems for longer-term storage and workloads. Use scratch file systems for temporary storage and shorter-term data processing.

2.    Note the file system ID (for example, fs-12345678) of your file system.

3.    In your launch template, create a user data section using the MIME multi-part file format. For more information, see Mime Multi Part Archive on the Cloud-init website.

The following example MIME multi-part file configures the compute resource to install the Lustre 2.10 package from the Extras library and mounts an existing Amazon FSx for Lustre file system at /scratch. The following example is for Amazon Linux 2. For installation instructions for other Linux distributions, see Installing the Lustre client.

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
--==MYBOUNDARY==
Content-Type: text/cloud-config; charset="us-ascii"
runcmd:
- file_system_id_01=fs-12345678
- region=us-east-1
- fsx_directory=/scratch
- fsx_mount_name=xxxxxxxx
- amazon-linux-extras install -y lustre2.10
- mkdir -p ${fsx_directory}
- mount -t lustre -o noatime,flock ${file_system_id_01}.fsx.${region}.amazonaws.com@tcp:/${fsx_mount_name} ${fsx_directory}
--==MYBOUNDARY==--
\=

Note: Replace fs-12345678, us-east-1, and xxxxxxxx with your file system ID, AWS Region, and 8-character mount name.

4.    Launch the template to invoke the user data.

5.    Create a file called mount-fsx-lustre.json, and the adjust the size of the volume based on your needs. See the following examples.

Example of a launch template based on Amazon Linux 2:

{
  "LaunchTemplateName":"user-data",
  "LaunchTemplateData": {
      "BlockDeviceMappings": [
        {
          "Ebs": {
            "DeleteOnTermination": true,
            "VolumeSize": 30,
            "VolumeType": "gp2"
          },
          "DeviceName": "/dev/xvda"
        } 
      ],
           "UserData": "TUlNRS1WZXJzaW9uOiAxLjAKQ29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSI9PU1ZQk9VTkRBUlk9PSIKCi0tPT1NWUJPVU5EQVJZPT0KQ29udGVudC1UeXBlOiB0ZXh0L2Nsb3VkLWNvbmZpZzsgY2hhcnNldD0idXMtYXNjaWkiCgpwYWNrYWdlczoKLSBhbWF6b24tZWZzLXV0aWxzCgpydW5jbWQ6Ci0gZmlsZV9zeXN0ZW1faWRfMDE9ZnMtODc0MTc4MDYgICAgIAotIGVmc19kaXJlY3Rvcnk9L21udC9lZnMKCi0gbWtkaXIgLXAgJHtlZnNfZGlyZWN0b3J5fQotIGVjaG8gIiR7ZmlsZV9zeXN0ZW1faWRfMDF9Oi8gJHtlZnNfZGlyZWN0b3J5fSBlZnMgdGxzLF9uZXRkZXYiID4+IC9ldGMvZnN0YWIKLSBtb3VudCAtYSAtdCBlZnMgZGVmYXVsdHMKCi0tPT1NWUJPVU5EQVJZPT0tLQ=="
}
}

Example of a launch template based on Amazon Linux 1:

{
  "LaunchTemplateName":"userdata",
  "LaunchTemplateData": {
      "BlockDeviceMappings": [
        {
          "Ebs": {
            "DeleteOnTermination": true,
            "VolumeSize": 8,
            "VolumeType": "gp2"
          },
          "DeviceName": "/dev/xvda"
        },
        {
          "Ebs": {
            "DeleteOnTermination": true,
            "VolumeSize": 22,
            "VolumeType": "gp2"
          },
          "DeviceName": "/dev/xvdcz"
        }          
      ],
           "UserData": "TUlNRS1WZXJzaW9uOiAxLjAKQ29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSI9PU1ZQk9VTkRBUlk9PSIKCi0tPT1NWUJPVU5EQVJZPT0KQ29udGVudC1UeXBlOiB0ZXh0L2Nsb3VkLWNvbmZpZzsgY2hhcnNldD0idXMtYXNjaWkiCgpwYWNrYWdlczoKLSBhbWF6b24tZWZzLXV0aWxzCgpydW5jbWQ6Ci0gZmlsZV9zeXN0ZW1faWRfMDE9ZnMtODc0MTc4MDYgICAgIAotIGVmc19kaXJlY3Rvcnk9L21udC9lZnMKCi0gbWtkaXIgLXAgJHtlZnNfZGlyZWN0b3J5fQotIGVjaG8gIiR7ZmlsZV9zeXN0ZW1faWRfMDF9Oi8gJHtlZnNfZGlyZWN0b3J5fSBlZnMgdGxzLF9uZXRkZXYiID4+IC9ldGMvZnN0YWIKLSBtb3VudCAtYSAtdCBlZnMgZGVmYXVsdHMKCi0tPT1NWUJPVU5EQVJZPT0tLQ=="
}
}

Note: If you add user data to a launch template in the Amazon EC2 console, paste in the user data as plaintext, or upload the user data from a file. If you use the AWS Command Line Interface (AWS CLI) or an AWS SDK, you must first base64-encode the user data. Then, submit that string as the value of the UserData parameter when you call CreateLaunchTemplate.

6.    Create a launch template based on the mount-fsx-lustre.json file that you created in step 5.

aws ec2 --region us-east-1 create-launch-template --cli-input-json file://mount-fsx-lustre.json

Note: Replace us-east-1 with your AWS Region.

The output is similar to the following:

{
    "LaunchTemplate": {
        "LaunchTemplateId": "lt-08cb09d54bcf551f3",
        "LaunchTemplateName": "fsx-test",
        "CreateTime": "2020-06-30T17:13:22.000Z",
        "CreatedBy": "arn:aws:iam::12345678999:user/test",
        "DefaultVersionNumber": 1,
        "LatestVersionNumber": 1
    }
}

7.    Create a new compute environment and associate that environment with your launch template.

Important: By default, AWS Batch managed compute environments use an approved version of the Amazon Elastic Container Service (Amazon ECS)-optimized AMI for compute resources. You must explicitly set the AMI ID for Amazon ECS-optimized Amazon Linux 2 AMIs and other Linux distributions.

Now, when AWS Batch spins up instances, the Amazon FSx Lustre file system is mounted on the container instances.

8.    To check if the Amazon FSx Lustre file system is mounted with the container instance, use SSH to connect to the instance launched by AWS Batch, and then run the following command:

$ df -h

The output is similar to the following:

Filesystem                                  Size      Used    Avail    Used%   Mounted on
devtmpfs                                    3.9G        0       3.9G     0%       /dev
tmpfs                                       3.9G        0       3.9G     0%       /dev/shm
tmpfs                                       3.9G     448K       3.9G     1%       /run
tmpfs                                       3.9G        0       3.9G     0%       /sys/fs/cgroup
/dev/xvda1                                   30G     4.2G        25G    15%       /
172.31.79.79@tcp:/xxxxxxxx                  1.1T     4.5M       1.1T     1%       /scratch
tmpfs                                       798M       0        798M     0%       /run/user/1000

Note: /scratch is mounted automatically.

9.    Create a job definition that includes the volume and mount point. For example:

{
    "jobDefinitionName": "Fsx-sample",
    "jobDefinitionArn": "arn:aws:batch:us-east-1:12345678999:job-definition/userdata:1",
    "revision": 1,
    "status": "ACTIVE",
    "type": "container",
    "parameters": {},
    "containerProperties": {
        "image": "busybox",
        "vcpus": 1,
        "memory": 1024,
        "command": [],
        "volumes": [
            {
                "host": {
                    "sourcePath": "/scratch"
                },
                "name": "Scratch"
            }
        ],
        "environment": [],
        "mountPoints": [
            {
                "containerPath": "/scratch",
                "sourceVolume": "Scratch"
            }
        ],
        "ulimits": [],
        "resourceRequirements": []
    }
}

10.    Submit an AWS Batch job using the job definition in step 8.


Did this article help?


Do you need billing or technical support?