AWS Storage Gateway for Files

The file gateway allows you to cost-effectively and durably store large files and media assets on AWS. Local applications also benefit from a low-latency local cache of frequently used content. The result is tiered, hybrid cloud content storage, which can be accessed easily by on-premises applications via NFS or SMB, from wherever you deploy gateway appliances - including in Amazon EC2. Storage Gateway automatically preserves the file metadata as object metadata, and also preserves the directory structure by including it in the object name. Content stored in Amazon S3 can be manipulated by in-cloud services via API, for example to do automatic image resizing with an AWS Lambda function, or to index the files with Amazon Elasticsearch Service.

Many organizations start their cloud journey by moving secondary and tertiary data, such as backups, to the cloud. The file gateway’s SMB and NFS interfaces provide an ideal way for IT groups to simply transition their backup jobs from existing on-premises backup systems to the cloud. Backup applications, native database tools or scripts that can write to SMB or NFS can write to the file gateway, which will store the backups as Amazon S3 objects of up to 5TiB in size. With an adequately sized local cache, recent backups can be used for fast on-site recoveries, while long-term retention needs are addressed by tiering backups to low-cost S3 Standard, S3 Infrequent Access and Amazon Glacier cloud storage tiers.

The file gateway makes it easy for Business Intelligence, Analytics or other teams that use Machine Learning to easily move file-based data into Amazon S3. They can then use that data for analytics, either with in-place queries via services such as Amazon Athena, Amazon Redshift Spectrum, or load it into other cloud tools such Amazon EMR for Hadoop-based processing. Post-analysis, result sets can be stored back in the same bucket, and the storage gateway service can make those new results files (objects) visible to on-premises applications wherever you have deployed a file gateway.

Additionally, you can apply simple compute functions with AWS Lambda to process data files stored in S3 with the file gateway, or even apply Machine Learning services to the data, for instance using Amazon Rekognition to perform image recognition or flag objectionable content.

Industries including Oil & Gas, Media & Entertainment, Design & Architecture, and Manufacturing have domain-specific applications that generate large specialized files. These files often need to be distributed, or at least accessible, from multiple sites. Over time, most of these files become infrequently accessed, and can be stored on lower cost, but durable online cloud storage, if not fully archived. File gateway allows such on-premises applications, across multiple locations, to use Amazon S3 and Amazon Glacier to store the files. It also enables migration of such file-based applications to Amazon EC2, by providing the central, globally accessible repository, based on Amazon S3 object storage.