How can I specify multiple Amazon S3 object prefixes for my Snowball export job?
Last updated: 2021-09-29
I want to specify multiple Amazon Simple Storage Service (Amazon S3) object key name prefixes for my AWS Snowball export job. However, when I create a Snowball export job with multiple prefixes, I see that only some files in the prefix range are copied.
Snowball export jobs support only one prefix range per S3 bucket. To copy all objects across multiple prefixes, you must specify the range to cover all the object key name prefixes that you want to copy. If you're using the AWS Command Line Interface (AWS CLI) to create an export job, then you must confirm that the BeginMarker and EndMarker that you specify within the KeyRange include all the prefixes that you want to copy.
Note: The AWS CLI doesn't return an error when you specify multiple values for BeginMarker or EndMarker. However, doing so results in an incomplete copy operation.
If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.
If the prefix range (key range) that you specify results in an incomplete copy operation, you can do either of the following:
- Copy the remaining objects to another S3 bucket. You can choose to use the whole bucket to export the remaining data. Or, you can choose to export the remaining objects to a prefix range using a single value for BeginMarker or EndMarker. You're charged for the PUT, COPY, and LIST requests.
- If you can't copy the remaining objects to another S3 bucket, then you must cancel the incomplete export job. Then, create another export job to copy the remaining objects or prefixes. For example, create two export jobs (or more as needed) to include the desired objects or prefixes.
Warning: This option costs more than copying the remaining objects to another bucket.