AWS Storage Gateway in 2020 – year in review

As 2020 draws to a close, we’ve shared several new launches and use cases with our customers and partners during the recent AWS Storage Day and re:Invent 2020 events. Tens of thousands of customers around the world are using AWS Storage Gateway to bridge between their on-premises applications and AWS Storage. Based on ongoing feedback and roadmap discussions, we have been busy enhancing the service to meet customer requirements.

Since January, we’ve launched 20 new features across all AWS Storage Gateway types (File Gateway, Tape Gateway, Volume Gateway), improving performance, scalability, security, and ease of management. While backup and archiving remain popular use cases for Storage Gateway, customers are increasingly using Storage Gateway for additional hybrid use cases. These include providing on-premises file shares backed by cloud storage, and distributed access to data in AWS for data pipelines. We look forward to learning more about customers’ hybrid cloud needs as we continue to expand the Storage Gateway service in 2021.

Looking back at 2020, I wanted to review all the new features we’ve launched, and share examples of how these features are helping our customers access AWS Storage in their data centers, remote office locations, and at the edge using Storage Gateway.

What is AWS Storage Gateway?

AWS Storage Gateway provides on-premises access to virtually unlimited cloud storage. The service supports standard storage protocols such as NFS, SMB, iSCSI, and iSCSI-VTL, so existing applications can use AWS Storage without any changes.

Storage Gateway provides a local, on-premises cache that delivers low latencies for frequently accessed data and optimizes data transfers to AWS. Gateways are managed through AWS using the AWS Management Console, CLI, and SDK, in the same way you manage other AWS services. Storage Gateway integrates with AWS services such as AWS Identity and Access Management (IAM), Amazon CloudWatch, and AWS CloudTrail, enabling secure access to AWS services, easy management and monitoring, and tracking of user activity on AWS resources.

AWS Storage Gateway enhancements in 2020

Storage Gateway already provides capabilities to meet the hybrid cloud storage needs of the most demanding enterprise customers across all industries. These include access to virtually unlimited storage in AWS through an on-premises storage cache, high availability for VMware vSphere deployments, data encryption in transit and at rest, and 99.999999999% durability of data in Amazon Simple Storage Service (Amazon S3).

In 2020, we focused on addressing customers’ requirements for larger-scale, more demanding, and compliant hybrid workloads, in addition to further simplifying management and ease of use for customers at all stages of cloud adoption:

Increased performance and scalability

• Increased the maximum supported cache size by 4x from 16 TB to 64 TB for Tape, Volume and File Gateways. This enables low latency access to larger working datasets on premises for backups and analytic workloads.
• Increased File Gateway performance by 4x from 1 Gbps to 4 Gbps for large file reads from the cloud.
• Increased Tape Gateway performance by 2x from 2.6 Gbps to 5.2 Gbps for writes, and from 4 Gbps to 8 Gbps for cache reads. This accelerates backup and restore times between backup applications and Tape Gateway.

Enhanced security and compliance

• Added support for FIPS 140-2 compliant service endpoints in AWS GovCloud (US) and all North America AWS Regions to protect sensitive information for regulated workloads.
• Achieved FedRAMP High authorization in AWS GovCloud (US) to store and manage your critical workloads in the AWS GovCloud (US) Region’s authorization boundary with data up to the high impact level.
• File Gateway added audit logging for SMB file shares enabling compliance with internal security policies and meeting external compliance requirements.
• File Gateway added access-based enumeration, enabling you to protect sensitive information. It allows you to prevent users from viewing SMB file shares, folders, and files that they would not be able to open based on their access permissions.
• File Gateway introduced file-level upload notification, enabling you to trigger automated workflows in the cloud and easily process on-premises file data stored with AWS as Amazon S3 objects.
• Tape Gateway added WORM support for virtual tapes ensuring that data on active tapes in your virtual tape library cannot be overwritten or erased.
• Tape Gateway enabled Tape Retention Lock, allowing you to specify a retention lock type and period on archived virtual tapes. This prevents deletion for a fixed amount of time, or even indefinitely. Both WORM and Tape Retention Lock capabilities combine to help you protect your data from accidental or malicious deletion.

Simplified management and ease-of-use

• Launched updated AWS Storage Gateway console for simplified gateway creation and management.
• Consolidated monitoring of a gateway’s CloudWatch alarms directly from the Storage Gateway console to provide you continuous visibility into gateway health and performance, all in one place.
• File Gateway introduced automated cache refresh. This ensures that frequently accessed directories are automatically refreshed to synchronize with the S3 bucket and infrequently accessed directories are refreshed only when needed.
• File Gateway enabled you to define multiple S3 prefixes for a S3 bucket and map a single S3 prefix to a single gateway file share.
• File Gateway introduced support for S3 Intelligent-Tiering, enabling you to optimize storage costs automatically when data access patterns change, without impacting performance or incurring operational overhead.
• Tape Gateway added support for IBM Spectrum Protect 8.1.10, enabling you to back up and archive data from IBM Spectrum Protect to AWS without changing backup workflows.
• Tape Gateway introduced automatic tape creation to reduce manual management of new tapes, enabling you to easily manage large deployments.
• Tape and Volume Gateways added schedule-based bandwidth throttling enabling you to optimize network use between your data center and AWS for data synchronization.

Expanded deployment options and availability to new AWS Regions

• Expanded international availability of the Storage Gateway Hardware Appliance to AWS Regions in Asia Pacific, South America, and Canada, and streamlined the quoting and ordering processes.
• Launched support for deploying Storage Gateway on Linux Kernel-based Virtual Machine (KVM) Hypervisor.
• Launched Storage Gateway in 4 new AWS Regions: AWS GovCloud (US-East), China (Ningxia), Europe (Milan), and Africa (Cape Town). Storage Gateway is now available in all 24 AWS Regions, including AWS GovCloud (US).

How are customers using AWS Storage Gateway?

Customers in all stages of their AWS journey are using AWS Storage Gateway. Storage Gateway accelerates their cloud adoption, whether they are just starting to use the cloud, are in the process of migrating to the cloud with applications and data both on premises and in the cloud, or have moved to AWS but need on-premises access to data in the cloud.

Storage Gateway offers an easy way for customers to start using AWS, since they can use it to move their on-premises backups and archival data to AWS. This allows them to free up on-premises storage, while storing data durably in AWS. Often Storage Gateway is the first AWS service that customers use, as we highlighted in our video and blog, “Cloud storage in minutes with AWS Storage Gateway.”

Customers using Storage Gateway for backup and archival use cases include Ryanair using Tape Gateway, Kellogg’s using File Gateway for database backups, and STEMCELL using Volume Gateway.

Then there are customers that are moving to the cloud and want to minimize their on-premises storage footprint, but often need on-premises access to storage for their existing applications. These customers use Storage Gateway as a way to replace on-premises storage with cloud-backed storage, which allows their existing applications to operate without changes, while still getting the benefits of storing and processing this data in AWS.

Customers using Storage Gateway for on-premises access to cloud storage in AWS include CME Group and Moderna Therapeutics. Both are using File Gateway to move data to AWS for processing by their cloud-based applications and provide their on-premises teams with low latency access to data stored in AWS.

Finally, customers that have data in the cloud, such as content repositories or media archives, often want to distribute it to multiple office and edge locations. They need to create business workflows that capture data from source systems at the edge, perform in-cloud processing and analytics, and provide access to end users in distributed locations. Customers using Storage Gateway as part of their data pipelines for processing data include Bristol Myers Squibb and Gritstone Oncology. These customers use File Gateway as a way to distribute and share data in AWS.

To learn more, watch the recent Storage Gateway sessions from this year’s AWS Storage Day and re:Invent virtual events:

• AWS Storage Day: What’s new in hybrid cloud, edge computing, and data transfer
• re:Invent 2020: Get on-premises access to cloud storage in minutes with AWS Storage Gateway

To get started using AWS Storage Gateway today, visit the Storage Gateway console. The Storage Gateway service team looks forward to working with you in 2021 to continue to enhance the service, and address your existing and emerging hybrid and edge cloud storage needs. Thank you for reading and let us know if you have any comments on the blog or requests for new features.