How do I mount an Amazon EFS to AWS Batch in a managed compute environment?

Last updated: 2020-02-17

I want to mount an Amazon Elastic File System (Amazon EFS) volume in AWS Batch.

Short Description

You can use a launch template to mount an Amazon EFS volume to your containers through AWS Batch without creating custom Amazon Machine Images (AMIs).

Important: When you create an Amazon EFS volume, use the same VPC and subnets that are assigned to your compute environment.

Resolution

1.    Create an Amazon EFS volume.

2.    Note the file system ID (for example, fs-12345678) of your elastic file system.

3.    In your launch template, create a user data section using the mime multi-part file format.

The following example mime multi-part file configures the compute resource to install the amazon-efs-utils package and then mount an existing Amazon EFS file system at /mnt/efs:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
--==MYBOUNDARY==
Content-Type: text/cloud-config; charset="us-ascii"
packages:
- amazon-efs-utils
runcmd:
- file_system_id_01=fs-12345678
- efs_directory=/mnt/efs
- mkdir -p ${efs_directory}
- echo "${file_system_id_01}:/ ${efs_directory} efs tls,_netdev" >> /etc/fstab
- mount -a -t efs defaults
--==MYBOUNDARY==--

Note: Replace fs-12345678 with your elastic file system ID.

4.    Launch the template to invoke the user data, and then create a file called mount-efs.json.

Note: Adjust the size of your volume based on your needs.

Launch template example based on Amazon Linux 2:

{
  "LaunchTemplateName":"user-data",
  "LaunchTemplateData": {
      "BlockDeviceMappings": [
        {
          "Ebs": {
            "DeleteOnTermination": true,
            "VolumeSize": 30,
            "VolumeType": "gp2"
          },
          "DeviceName": "/dev/xvda"
        } 
      ],
           "UserData": "TUlNRS1WZXJzaW9uOiAxLjAKQ29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSI9PU1ZQk9VTkRBUlk9PSIKCi0tPT1NWUJPVU5EQVJZPT0KQ29udGVudC1UeXBlOiB0ZXh0L2Nsb3VkLWNvbmZpZzsgY2hhcnNldD0idXMtYXNjaWkiCgpwYWNrYWdlczoKLSBhbWF6b24tZWZzLXV0aWxzCgpydW5jbWQ6Ci0gZmlsZV9zeXN0ZW1faWRfMDE9ZnMtODc0MTc4MDYgICAgIAotIGVmc19kaXJlY3Rvcnk9L21udC9lZnMKCi0gbWtkaXIgLXAgJHtlZnNfZGlyZWN0b3J5fQotIGVjaG8gIiR7ZmlsZV9zeXN0ZW1faWRfMDF9Oi8gJHtlZnNfZGlyZWN0b3J5fSBlZnMgdGxzLF9uZXRkZXYiID4+IC9ldGMvZnN0YWIKLSBtb3VudCAtYSAtdCBlZnMgZGVmYXVsdHMKCi0tPT1NWUJPVU5EQVJZPT0tLQ=="
}
}

Launch template example based on Amazon Linux 1:

{ "LaunchTemplateName":"userdata", "LaunchTemplateData": { "BlockDeviceMappings": [ { "Ebs": { "DeleteOnTermination": true, "VolumeSize": 8, "VolumeType": "gp2" }, "DeviceName": "/dev/xvda" }, { "Ebs": { "DeleteOnTermination": true, "VolumeSize": 22,
    "VolumeType": "gp2" }, "DeviceName": "/dev/xvdcz" } ], "UserData": "TUlNRS1WZXJzaW9uOiAxLjAKQ29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSI9PU1ZQk9VTkRBUlk9PSIKCi0tPT1NWUJPVU5EQVJZPT0KQ29udGVudC1UeXBlOiB0ZXh0L2Nsb3VkLWNvbmZpZzsgY2hhcnNldD0idXMtYXNjaWkiCgpwYWNrYWdlczoKLSBhbWF6b24tZWZzLXV0aWxzCgpydW5jbWQ6Ci0gZmlsZV9zeXN0ZW1faWRfMDE9ZnMtODc0MTc4MDYgICAgIAotIGVmc19kaXJlY3Rvcnk9L21udC9lZnMKCi0gbWtkaXIgLXAgJHtlZnNfZGlyZWN0b3J5fQotIGVjaG8gIiR7ZmlsZV9zeXN0ZW1faWRfMDF9Oi8gJHtlZnNfZGlyZWN0b3J5fSBlZnMgdGxzLF9uZXRkZXYiID4+IC9ldGMvZnN0YWIKLSBtb3VudCAtYSAtdCBlZnMgZGVmYXVsdHMKCi0tPT1NWUJPVU5EQVJZPT0tLQ=="
    } }    

Note: If you add user data to a launch template in the Amazon Elastic Compute Cloud (Amazon EC2) console, paste in the user data as plaintext, or upload it from a file. If you use the AWS Command Line Interface (AWS CLI) or an AWS SDK, you must first base64-encode the user data. Then, submit that string as the value of the UserData parameter when you call CreateLaunchTemplate, as shown in the preceding JSON template.

5.    Create a launch template based on the mount-efs.json file that you created in step 4.

aws ec2 --region <Region> create-launch-template --cli-input-json file://mount-efs.json

The output should be similar to the following:

{
    "LaunchTemplate": {
        "LaunchTemplateId": "lt-06935eb650e40f886",
        "LaunchTemplateName": "user-data",
        "CreateTime": "2019-12-26T09:40:46.000Z",
        "CreatedBy": "arn:aws:iam::12345678999:user/alice",
        "DefaultVersionNumber": 1,
        "LatestVersionNumber": 1
    }
}

6.    Create a new compute environment and associate that environment with your launch template.

Now, when AWS Batch spins up instances, the Amazon EFS volume is mounted on the instances.

7.    To check if the Amazon EFS volume is mounted with the container instance, use SSH to connect to the instance launched by AWS Batch, and then run the following command:

$ df -h

The output should be similar to the following:

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G   92K  3.9G   1% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
/dev/xvda1       50G  854M   49G   2% /
127.0.0.1:/     8.0E     0  8.0E   0% /mnt/efs

Note: /mnt/efs is mounted automatically.

8.    Create a job definition that includes the volume and mount point. See the following example:

{
    "jobDefinitionName": "userdata",
    "jobDefinitionArn": "arn:aws:batch:us-east-1:12345678999:job-definition/userdata:1",
    "revision": 1,
    "status": "ACTIVE",
    "type": "container",
    "parameters": {},
    "containerProperties": {
        "image": "busybox",
        "vcpus": 1,
        "memory": 1024,
        "command": [],
        "volumes": [
            {
                "host": {
                    "sourcePath": "/efs"
                },
                "name": "efs"
            }
        ],
        "environment": [],
        "mountPoints": [
            {
                "containerPath": "/mnt/efs",
                "sourceVolume": "efs"
            }
        ],
        "ulimits": [],
        "resourceRequirements": []
    }
}

9.    Submit an AWS Batch job using the job definition in step 8.


Did this article help you?

Anything we could improve?


Need more help?