How do I resolve the "No space left on device" error in AWS Batch?

Last updated: 2020-02-03

I receive the following error message when my AWS Batch job fails: "no space left on device."

Short Description

The following common scenarios can cause this error:

  • You don't have enough disk space on your Amazon Elastic Block Store (Amazon EBS) volume (dev/xvdcz).
  • You have reached the default Docker storage limit of 10 GiB.

Note: The following resolution applies to Amazon Linux 1 Amazon Machine Images (AMIs) optimized for Amazon Elastic Container Service (Amazon ECS). An Amazon ECS-optimized AMI is the default image used when you create a managed compute environment in AWS Batch. To understand how Docker uses the Logical Volume Management (LVM) device, see Version 2015.09.d and Later.

Resolution

To see how much storage you have available for Docker (dev/xvdcz), see the "To determine the available storage for Docker" section of AMI Storage Configuration.

If you must extend the Amazon EBS volumes for your instances launched by AWS Batch, then increase Amazon EBS volumes in AWS Batch using a launch template.

If extending Amazon EBS volumes isn't the best solution for your scenario and you must go beyond the default storage limit of 10 GiB, then increase your Docker container storage volume limit for Amazon ECS instances launched from Amazon Linux 1 AMIs.

Note: You can use a launch template to build a configuration template that applies to all your Amazon Elastic Compute Cloud (Amazon EC2) instances launched by AWS Batch. A launch template allows you to customize the default ECS-optimized AMI used by AWS Batch without using a custom AMI.

If you add user data to a launch template in the Amazon EC2 console, paste in the user data as plain text, or upload it from a file.

If you use the AWS Command Line Interface (AWS CLI) or an AWS SDK, you must first base64 encode the user data. Then, submit that string as the value of the UserData parameter when you call CreateLaunchTemplate, as shown in the following JSON template:

{
    "LaunchTemplateName": "base64-user-data",
    "LaunchTemplateData": {
        "UserData": "ewogICAgIkxhdW5jaFRlbXBsYXRlTmFtZSI6ICJpbmNyZWFzZS1jb250YWluZXItdm9sdW..."
    }
}

The following example MIME multi-part file overrides the default Docker image settings for a compute resource:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#cloud-boothook
#!/bin/bash
cloud-init-per once docker_options echo 'OPTIONS="${OPTIONS} --storage-opt dm.basesize=20G"' >> /etc/sysconfig/docker

echo ECS_CLUSTER=default>>/etc/ecs/ecs.config
echo ECS_IMAGE_CLEANUP_INTERVAL=60m >> /etc/ecs/ecs.config
echo ECS_IMAGE_MINIMUM_CLEANUP_AGE=60m >> /etc/ecs/ecs.config

--==MYBOUNDARY==--

Did this article help you?

Anything we could improve?


Need more help?