How do I resolve the "No space left on device" error in AWS Batch?

Last updated: 2020-04-23

I receive the "No space left on device" error message when my AWS Batch job fails.

Short description

The following common scenarios can cause this error:

  • You don't have enough disk space on your Amazon Elastic Block Store (Amazon EBS) volume (dev/xvdcz).
  • You have reached the default Docker storage limit of 10 GiB.

Note: The following resolution applies to Amazon Linux 1 Amazon Machine Images (AMIs) optimized for Amazon Elastic Container Service (Amazon ECS). An Amazon ECS-optimized AMI is the default image used when you create a managed compute environment in AWS Batch. To understand how Docker uses the Logical Volume Management (LVM) device, see Version 2015.09.d and later.


Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.

To see how much storage you have available for Docker (dev/xvdcz), see AMI storage configuration and follow the instructions in To determine the available storage for Docker.

If you must extend the Amazon EBS volumes for your instances launched by AWS Batch, then increase Amazon EBS volumes in AWS Batch using a launch template.

Note: You can use a launch template to build a configuration template that applies to all your Amazon Elastic Compute Cloud (Amazon EC2) instances launched by AWS Batch. A launch template allows you to customize the default Amazon ECS-optimized AMI used by AWS Batch without using a custom AMI.

If you add user data to a launch template in the Amazon EC2 console, paste in the user data as plaintext, or upload the user data from a file.

If you use the AWS CLI or an AWS SDK, you must first base64 encode the user data. Then, submit that string as the value of the UserData parameter when you call CreateLaunchTemplate. For example, see the following JSON template:

    "LaunchTemplateName": "base64-user-data",
    "LaunchTemplateData": {
        "UserData": "ewogICAgIkxhdW5jaFRlbXBsYXRlTmFtZSI6ICJpbmNyZWFzZS1jb250YWluZXItdm9sdW..."

The following example MIME multi-part file overrides the default Docker image settings for a compute resource:

Content-Type: multipart/mixed; boundary="==BOUNDARY=="
MIME-Version: 1.0

Content-Type: text/cloud-boothook; charset="us-ascii"
cloud-init-per once docker_options echo 'OPTIONS="${OPTIONS} --storage-opt dm.basesize=20G"' >> /etc/sysconfig/docker

Content-Type: text/x-shellscript; charset="us-ascii"

# Set any ECS agent configuration options
echo ECS_CLUSTER=default>>/etc/ecs/ecs.config
echo ECS_IMAGE_CLEANUP_INTERVAL=60m >> /etc/ecs/ecs.config
echo ECS_IMAGE_MINIMUM_CLEANUP_AGE=60m >> /etc/ecs/ecs.config

Did this article help?

Do you need billing or technical support?