Why aren't changes on my Amazon S3 bucket showing on the Storage Gateway file share?
Last updated: 2020-02-04
I made changes to my Amazon Simple Storage Service (Amazon S3) bucket, such as uploading a new file or removing an existing file. However, the changes aren't propagating to my AWS Storage Gateway file share. How can I fix this?
Storage Gateway updates the file share cache automatically when you write files to the cache locally using the file share. However, Storage Gateway doesn't automatically update the cache when you upload a file directly to Amazon S3. When you do this, you must perform a RefreshCache operation to see the changes on the file share. If you have more than one file share, then you must run the RefreshCache operation on each file share.
You can refresh the cache using the Storage Gateway console, the AWS Command Line Interface (AWS CLI), or through an AWS Lambda function.
Using the Storage Gateway console
To refresh the cache using the Storage Gateway console, see Refreshing Objects in Your Amazon S3 Bucket.
Using the AWS CLI
1. Run the list-file-shares command:
$ aws storagegateway list-file-shares
2. Copy the Amazon Resource Number (ARN) of the file share with the cache that you want to refresh. Then, run the refresh-cache command and enter the ARN as the value for --file-share-arn:
$ aws storagegateway refresh-cache --file-share-arn arn:aws:storagegateway:eu-west-1:12345678910:share/share-FFDEE12
As another option, you can create a script or scheduled cron job to run the refresh-cache command at certain intervals. For example, the following cron job runs the command every 12 hours:
0 */12 * * * root aws storagegateway refresh-cache --file-share-arn arn:aws:storagegateway:eu-west-1:12345678910:share/share-FFDEE12
Note: You must schedule the cron job using the credentials of an AWS Identity and Access Management (IAM) user or role that has permissions to run the refresh-cache command.
Using a Lambda function
You can create a Lambda function that runs the RefreshCache API on a certain schedule. The following example function uses boto3:
import json import boto3 client = boto3.client('storagegateway') def lambda_handler(event, context): print(event) response = client.refresh_cache( FileShareARN='arn:aws:storagegateway:ap-southeast-2:123456789012:share/share-EEFFBD9C' ) print(response) return 'Your FileShare Cache has been Refreshed'
Note the following considerations for automating the refresh cache operation:
- If there are millions of files in the S3 bucket, then the refresh cache operation can take several hours. By default, the objects and folders at the root of the bucket are refreshed. You can choose to run RefreshCache on a specific folder or list of folders instead of refreshing the entire bucket.
- It's not a best practice to invoke the refresh cache operation based on an event-driven operation, such as an Amazon S3 event notification, unless the events are spaced well apart. Instead, run the refresh cache operation on a timer. For example, you can use an Amazon CloudWatch Event rule to run an AWS Lambda function on a schedule. This configuration can space the refresh cache calls apart by a few hours.
- It's a best practice to set up notifications for when a refresh cache operation completes. To do this, create an Amazon Simple Notification Service (Amazon SNS) topic and subscription. Then, create an Amazon CloudWatch Event that triggers on a Storage Gateway Refresh Cache Event and set the SNS topic as the target.