How can I verify an Amazon S3 Lifecycle configuration rule for cleaning up incomplete multipart uploads?

Last updated: 2022-11-22

I set an Amazon Simple Storage Service (Amazon S3) Lifecycle configuration rule to clean up incomplete multipart uploads. How can I confirm that the rule is working?

Short description

You can verify the Amazon S3 Lifecycle configuration rule in each of the following ways:

  • Query server access logs.
  • Test the rule by uploading some parts of a multipart upload using the AWS Command Line Interface (AWS CLI).
    Note: When you use the AWS CLI to set the rule, the rule is called AbortIncompleteMultipartUpload.

Resolution

Query server access logs

To query server access logs, you must have server access logging turned on in your bucket before the S3 Lifecycle rule is set to run. After the logs are available, review the logs to verify if the rule cleaned up an incomplete multipart upload. For more information, see S3 Lifecycle and logging.

Test the rule by uploading some parts of a multipart upload using the AWS CLI

If you don't have server access logging turned on, then you can test the rule by running an incomplete multipart upload:

Note: The following example uses the AWS CLI to perform a multipart upload. To use an AWS SDK to perform a multipart upload, see Uploading objects using multipart upload API.

1.    Split the file that you want to upload into multiple parts. For example, if you're using a Linux operating system, run the split command:

split /path/to/filetoupload

2.    Run the create-multipart-upload command using the AWS CLI to initiate a multipart upload:

aws s3api create-multipart-upload --bucket awsexamplebucket --key large_test_file

Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version.

The command returns an output that contains the UploadID.

3.    Copy the UploadID value as a reference for later steps. The command output looks like this:

{
    "AbortDate": "Mon, 03 Jun 2019 00:00:00 GMT",
    "AbortRuleId": "multipartcleanup",
    "Bucket": "awsexamplebucket",
    "Key": "objectname",
    "UploadId": "exampleTUVGeKAk3Ob7qMynRKqe3ROcavPRwg92eA6JPD4ybIGRxJx9R0VbgkrnOVphZFK59KCYJAO1PXlrBSW7vcH7ANHZwTTf0ovqe6XPYHwsSp7eTRnXB1qjx40Tk"
}

4.    Run the upload-part command to upload the first part of the file:

aws s3api upload-part --bucket awsexamplebucket --key large_test_file --part-number 1 --body large_test_file.001 --upload-id exampleTUVGeKAk3Ob7qMynRKqe3ROcavPRwg92eA6JPD4ybIGRxJx9R0VbgkrnOVphZFK59KCYJAO1PXlrBSW7vcH7ANHZwTTf0ovqe6XPYHwsSp7eTRnXB1qjx40Tk

Note: Replace all values with your Amazon S3 bucket, file, and multipart upload values.

The command returns an output that contains an ETag value for the part of the file that you uploaded.

5.    Copy the ETag value as a reference for later steps. The command output looks like the following:

{
    "ETag": "\"example8be9a0268ebfb8b115d4c1fd3\""
}

6.    Repeat steps 3 and 4 for some parts of the file. For this test, don't upload all parts to complete the file.

Note: Increase the part number with each new part that you upload.

7.    For this test, don't complete or abandon the multipart upload operation. Instead, note the parts that were uploaded by running the list-parts command:

aws s3api list-parts --bucket awsexamplebucket --key large_test_file --upload-id exampleTUVGeKAk3Ob7qMynRKqe3ROcavPRwg92eA6JPD4ybIGRxJx9R0VbgkrnOVphZFK59KCYJAO1PXlrBSW7vcH7ANHZwTTf0ovqe6XPYHwsSp7eTRnXB1qjx40Tk

The command returns a list of the parts:

{
    "Parts": [
        {
            "PartNumber": 1,
            "LastModified": "2019-06-01T18:17:39.000Z",
            "ETag": "\"example8be9a0268ebfb8b115d4c1fd3\"",
            "Size": 162641
        },
        {
            "PartNumber": 2,
            "LastModified": "2019-06-01T18:18:19.000Z",
            "ETag": "\"example246e31ab807da6f62802c1ae8\"",
            "Size": 3961
        }
    ],
    "Initiator": {
        "ID": "arn:aws:iam::111122223333:user/jane",
        "DisplayName": "jane"
    },
    "Owner": {
        "DisplayName": "bucketowner",
        "ID": "examplea2395fe1985ffabfe0c17d3522e5bc7fa1a2d048f8fc764d7709da80d"
    },
    "StorageClass": "STANDARD"
}

8.    Identify the multipart uploads that are in progress for your bucket by running the list-multipart-uploads command:

aws s3api list-multipart-uploads --bucket awsexamplebucket

The command returns a list of in-progress multipart uploads:

{
    "Uploads": [
        {
            "UploadId": "exampleTUVGeKAk3Ob7qMynRKqe3ROcavPRwg92eA6JPD4ybIGRxJx9R0VbgkrnOVphZFK59KCYJAO1PXlrBSW7vcH7ANHZwTTf0ovqe6XPYHwsSp7eTRnXB1qjx40Tk",
            "Key": "large_test_file",
            "Initiated": "2019-06-01T17:08:33.000Z",
            "StorageClass": "STANDARD",
            "Owner": {
                "DisplayName": "bucketowner",
                "ID": "examplea2395fe1985ffabfe0c17d3522e5bc7fa1a2d048f8fc764d7709da80d"
            },
            "Initiator": {
                "ID": "arn:aws:iam::111122223333:user/jane",
                "DisplayName": "jane"
            }
        }
    ]
}

9.    Wait the number of days that you set for your S3 Lifecycle configuration rule. (In your S3 Lifecycle configuration rule, you specified how many days after the start of a multipart upload the cleanup starts.)

10.    Run the list-parts command again to see if the parts of the incomplete multipart upload were deleted. After the parts are deleted by the rule, the command returns the following response:

"An error occurred (NoSuchUpload) when calling the ListParts operation: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed."

11.    Run the list-multipart-uploads command again to see if the multipart operation was abandoned. After the multipart operation is abandoned by the rule, the command returns no output.