How do I troubleshoot Amazon S3 Batch Operations issues?

Last updated: 2021-12-16

I'm trying to create an Amazon Simple Storage Service (Amazon S3) Batch Operations job for objects stored in my bucket. However, Amazon S3 keeps returning an error or my batch job keeps failing. How do I troubleshoot this?

Short description

If an Amazon S3 Batch Operations job encounters an issue that prevents it from running successfully, then the job fails. For example, if S3 is unable to read the specified manifest, or objects in your manifest don't exist in the specified bucket, then the job fails. A failed job generates one or more failure codes and reasons. S3 Batch Operations stores the failure codes and reasons with the job so that you can view them by requesting the job's details. You can also review your failure codes and reasons in the completion report for the job.

To prevent jobs from running a large number of unsuccessful operations, Amazon S3 also imposes a task-failure threshold on every Batch Operations job. Amazon S3 monitors the task failure rate after at least 1,000 tasks have been run. If a job exceeds the failure rate of 50%, the job fails. To resolve this failure, review the causes of the failures and correct them before resubmitting the job.

Here are some common reasons that Amazon S3 Batch Operations fails or returns an error:

  • Manifest file format (CSV or JSON)
  • Manifest file specifies multiple bucket names or contains multiple header rows
  • Permissions to read the manifest file
  • Batch job Region
  • Target bucket for your S3 Inventory report
  • AWS Identity Access Management (IAM) role's trust policy
  • IAM role permissions for creating a batch job
  • IAM role access to source bucket, S3 Inventory report, and destination bucket
  • AWS Organizations service control policy (SCP)

Resolution

Manifest file format (CSV or JSON)

Amazon S3 Batch Operations supports CSV and JSON (S3 Inventory report) manifest files. When you create a new batch job in Amazon S3, select or specify the correct manifest format for your manifest file:

  • For the Amazon S3 Inventory report, make sure to use a CSV-formatted report and specify the manifest.json file associated with the inventory report.
  • For CSV files, each row in your manifest file must include the manifest object key, ETag, and optional version ID. Object keys must be URL encoded. The manifest must either include version IDs for all objects or omit version IDs for all objects. Note: If the objects in your manifest are in a versioned bucket, you must specify the version IDs for the objects. Otherwise, the batch job might fail. Or, the batch job might be applied to the latest version of the object (instead of the object that existed when the job was created).

For more information about manifest files and formats, see Specifying a manifest.

Manifest file specifies multiple bucket names or contains multiple header rows

With S3 Batch Operations, you can copy objects, modify the object lock retention date of objects, or modify the object lock legal hold status. These three batch job operations require that all objects listed in the manifest file also exist in the same bucket. Otherwise, you receive the following error:

Reasons for failure:
Cannot have more than 1 bucket per Job. <Job ID>

If you're performing one of these three batch job operations, make sure that your manifest file specifies only one bucket name. Additionally, the manifest file must not contain any header rows. For example, if your manifest file looks like this (where there are multiple header rows), then Amazon S3 will return an error:

bucket,key
my-batch-bucket,object001.txt
my-batch-bucket,object002.txt
my-batch-bucket,object003.txt
my-batch-bucket,object004.txt

Permissions to read the manifest file

Verify that the IAM role that you use to create the S3 Batch Operations job has GetObject permissions to allow it to read the manifest file. You can do this by checking the object's metadata. Look for any mismatches in access with S3 Object Ownership or any unsupported AWS KMS keys that are being used to encrypt the manifest file.

If you don't have permission to read the manifest file, then you get the following errors when you try to create an S3 Batch Operations job.

AWS CLI:

Reason for failure
Reading the manifest is forbidden: AccessDenied

Amazon S3 console:

Warning: Unable to get the manifest object's ETag. Specify a different object to continue

Note: S3 Batch Operations supports CSV inventory reports that are AWS KMS-encrypted. S3 Batch Operations doesn't support CSV manifest files that are AWS KMS-encrypted. For more information, see Configuring inventory or Specifying a manifest.

Batch job Region

S3 Batch Operations copy jobs must be created in the same AWS Region as the destination bucket where you want to copy your objects to. Therefore, make sure to select the same Region as your destination bucket when you create your batch job. For example, if your destination bucket resides in the us-west-2 Region, then select us-west-2 as the Region for your batch job.

Target bucket for your S3 Inventory report

Confirm that the target bucket for your S3 Inventory report exists. Also, confirm that the S3 bucket policy doesn't deny the s3:PutObject action. If the report is delivered to another AWS account, then confirm whether the target bucket allows the IAM role to perform the s3:PutObject action.

IAM role's trust policy

Note: Make sure that you're specifying an IAM role and not an IAM user.

Unlike an IAM user, an IAM role has a trust policy that defines which conditions must be met for other principals to assume it. To allow the S3 Batch Operations service principal to assume the IAM role, attach a trust policy to the role.

The following example trust policy delegates access to Amazon S3, while reducing any risks associated with privilege escalation:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Principal":{
            "Service":"batchoperations.s3.amazonaws.com"
         },
         "Action":"sts:AssumeRole"
      }
   ]
}

IAM permissions for creating a batch job

Before creating and running S3 Batch Operations jobs, grant the required permissions. If your IAM role is missing the required permissions to perform the S3 Batch Operations job, then the batch job fail.

To create an S3 Batch Operations job, s3:CreateJob permissions are required. The same entity that creates the job must also have iam:PassRole permissions to pass the IAM role that's specified for the batch job. For more information about specifying IAM resources, see IAM JSON policy, Resource elements.

IAM role access to source bucket, S3 Inventory report, and destination bucket

Check to make sure that the IAM role that you're using for S3 Batch Operations has the required permissions to perform the batch job. For example, the IAM policy for the copy operation looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectTagging"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::{{DestinationBucket}}/*"
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectTagging",
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::{{SourceBucket}}",
                "arn:aws:s3:::{{SourceBucket}}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::{{ManifestBucket}}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::{{ReportBucket}}/*"
            ]
        }
    ]
}

AWS Organizations service control policy (SCP)

If you're using AWS Organizations, then confirm that there aren't any deny statements that might deny access to Amazon S3. For example, if your service control policy is explicitly denying all S3 actions, you might get an Access Denied error when you create a batch job.

Here's an example policy that explicitly denies all S3 actions:

{
    "Version": "2012-10-17",
    "Statement": [
        {   
            "Principal":"*",
            "Effect": "Deny",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}

If you intend to apply a restrictive policy, you can allowlist the IAM role that S3 Batch Operations will use to perform the operation. For example:

{
    "Version": "2012-10-17",
    "Statement": [
        {   
            "Principal":"*",
            "Effect": "Deny",
            "Action": "s3:*",
            "Resource": "*",
            "Condition": {
            "StringNotLike": {
                "aws:userId": [
                    "AROAEXAMPLEID:*",
                    "AIDAEXAMPLEID",
                    "111111111111"
                ]
            }
        }
        }
    ]
}

Did this article help?


Do you need billing or technical support?