When I run a list command for the objects in my Amazon S3 bucket, the command is unresponsive. How can I troubleshoot this?
Last updated: 2019-12-19
I'm trying to run the aws s3 ls command or the aws s3api list-objects command on my Amazon Simple Storage Service (Amazon S3) bucket. However, the command doesn't return any output and eventually times out. How can I fix this?
Try the following ways to troubleshoot an unresponsive list command:
- Remove expired object delete markers.
- Try the list-objects-v2 command or the latest version of the AWS Command Line Interface (AWS CLI).
- Use an Amazon S3 inventory report.
Remove expired object delete markers
In a versioning-enabled bucket, when a delete operation is made for a versioned object (without specifying a version ID), then a delete marker is created. When previous object versions are set to expire using lifecycle configuration rules, delete markers that are associated with the expired objects are in the bucket.
If there are hundreds of thousands of delete markers, the list operation might timeout. This is why removing the expired object delete markers from the bucket can improve the performance of a list operation.
You can use a lifecycle rule to clean up expired object delete markers:
- Open the Amazon S3 console.
- From the list of buckets, choose the bucket that has the expired object delete markers.
- Choose the Management tab.
- Choose Add lifecycle rule and enter a rule name. Then, choose Next.
- For Storage class transition, you can skip this section by choosing Next.
- For Configure expiration, select Clean up expired object delete markers. Then, choose Next.
- Review the lifecycle rule settings, and then choose Save.
Try the list-objects-v2 command or the latest version of the AWS CLI
The list-objects-v2 command (or ListObjectsV2 API call) is an updated version of the list operation that includes some performance improvements. The following example command requests a list of objects in awsexamplebucket:
aws s3api list-objects-v2 --bucket awsexamplebucket
Additionally, confirm that you're using the latest version of the AWS CLI. Earlier AWS CLI versions might still be using previous ListObjects API call.
Note: For backward compatibility, Amazon S3 continues to support the ListObjects API call.
Use an Amazon S3 inventory report
Instead of running a list operation, you can use an Amazon S3 inventory report to get a list of your objects. When a bucket has hundreds of millions of objects, running multiple list operations (listing 1,000 objects at a time) can be challenging.
You can get daily or weekly Amazon S3 inventory reports for a bucket or a certain prefix, and you can include additional object metadata in your reports. Additionally, you can pass an inventory report as an input for a batch operation, and you can use Amazon Athena to query a report.