How can I specify multiple Amazon S3 object prefixes for my Snowball export job?

Last updated: 2020-03-12

I want to specify multiple Amazon Simple Storage Service (Amazon S3) object key name prefixes for my AWS Snowball export job. However, when I create a Snowball export job with multiple prefixes, I see that only some files in the prefix range are copied. Why is this happening?

Resolution

Snowball export jobs support only one prefix range per S3 bucket. To copy all objects across multiple prefixes, you must specify the range so that it covers all the object key name prefixes that you want to copy. If you're using the AWS Command Line Interface (AWS CLI) to create an export job, then you must confirm that the BeginMarker and EndMarker that you specify within the KeyRange include all the prefixes that you want to copy.

Note: The AWS CLI doesn't return an error if you specify multiple values for BeginMarker or EndMarker, but doing so results in an incomplete copy operation.

If the prefix range (key range) that you specify results in an incomplete copy operation, you can either:

  • Copy the remaining objects to another S3 bucket. You can choose to use the whole bucket to export the remaining data. Or, you can choose to export the remaining objects to a prefix range using a single BeginMarker or EndMarker. You're charged for the PUT, COPY, and LIST requests.
  • If you can't copy the remaining objects to another S3 bucket, then you must cancel the incomplete export job. Then, create another export job to copy the remaining objects or prefixes.
    Warning: This option costs more than copying the remaining objects to another bucket.

Did this article help you?

Anything we could improve?


Need more help?