Cloud storage in minutes with AWS Storage Gateway (updated)

Moving workloads to the cloud is one of the primary ways for CIOs to address their top strategic priorities: increasing agility, accelerating innovation, strengthening security, and reducing cost. Many companies have existing infrastructure in data centers, remote offices, and edge locations, but want to take advantage of cloud services. AWS helps these customers accelerate their journey to the cloud by providing hybrid cloud storage services to create a seamless extension of their on-premises infrastructure in AWS. As companies evaluate their applications, some are relatively straightforward to move to the cloud, while others are more complex. Applications may need to remain on-premises for performance or compliance reasons, or because they require tight integration with local IT infrastructure. For these reasons, organizations want to explore hybrid cloud storage solutions that provide on-premises access to data stored in AWS and extend the benefits of the AWS Cloud to their data centers and edge locations.

AWS Storage Gateway is a hybrid cloud storage solution that helps customers overcome hybrid cloud storage challenges, and bridge the gap between their on-premises environments and the cloud. Storage Gateway enables on-premises applications to use cloud storage by providing low-latency data access over standard storage protocols. A local cache stores your most recently used data on premises, and the cloud provides scalability in addition to industry-leading data protection, durability, availability, security, and performance. Designed to be easy to deploy, easy to activate, and easy for your applications to access, Storage Gateway provides a simple first step into the cloud.

Since we published the original version of this blog almost a year ago, we have added a new gateway type, the Amazon FSx File Gateway, launched a new management console, and added a variety of other enhancements to Storage Gateway. In this blog post, we cover these updates as we describe the core features and functionality of Storage Gateway and review hybrid storage architectures from a high level. I also define the four types of Storage Gateway (Amazon S3 File, Amazon FSx File, Tape, and Volume Gateways), and reference examples of common customer use cases for Storage Gateway. Lastly, I show you how to get started by walking through the three main steps when deploying a Storage Gateway in your environment.

Figure 1: This outlines common challenges between on-premises environments and the AWS Cloud

Introduction to Storage Gateway

A Storage Gateway can be deployed directly in your on-premises environment as a virtual machine (VMware ESXi, Microsoft Hyper-V, Linux KVM), in AWS as an Amazon EC2 instance, or as a pre-configured standalone hardware appliance. Moreover, Storage Gateway requires no special networking or additional hardware, while providing:

• Support for standard storage protocols such as NFS, SMB, iSCSI, and iSCSI VTL, so existing applications can use AWS Cloud storage without changes.
• A local cache for low-latency access for your applications.
• Optimized and secure data transfers between on premises and the AWS Cloud.
• Interoperability with other AWS Cloud storage services such as Amazon S3, Amazon S3 Glacier, Amazon FSx for Windows File Server, Amazon EBS, and AWS Backup.
• Integration with other AWS services such as AWS Key Management Service (KMS), AWS Identity and Access Management (IAM), AWS CloudTrail, and Amazon CloudWatch.

Storage Gateway combines the benefits of on-premises infrastructure with all the benefits of the cloud.

Figure 2: High-level architecture of Storage Gateway

Three common use cases for Storage Gateway include moving backups to the cloud, presenting on-premises file shares backed by cloud storage, and providing on-premises applications low latency access to data stored in the cloud. By leveraging the cloud for these workloads, customers can reduce the on-premises capacity of their primary storage systems, reduce their physical storage footprint, and in some cases, consolidate data centers.

Some examples include storing SQL Server backups in Amazon S3 using AWS Storage Gateway, and replacing physical tape-based backups with Tape Gateway.

Figure 3: Three common use cases of Storage Gateway, which can be deployed across various stages of the cloud adoption journey

To support these use cases AWS provides four different types of Gateways: Amazon S3 File Gateway, Amazon FSx File Gateway, Tape Gateway, and Volume Gateway.

Figure 4: Storage Gateway is made up of four gateway types: Amazon S3 File Gateway, Amazon FSx File Gateway, Tape Gateway, Volume Gateway.

Amazon S3 File Gateway enables you to store file data as objects in Amazon S3 cloud storage for data lakes, backups, and Machine Learning workflows. With Amazon S3 File Gateway, each file is stored as an object in Amazon S3 with a one-to-one mapping between a file and an object.

Amazon FSx File Gateway provides low-latency, on-premises access to fully managed file shares for team file shares, home directories, and file-based business applications.

Tape Gateway provides a virtual tape library, with each virtual tape being stored in Amazon S3, and supports all major backup applications. Tape Gateway is designed to be a seamless drop-in replacement to your existing physical tape infrastructure. When you no longer require immediate or frequent access to data contained on a virtual tape, you can have your backup application eject it. Then, your tape is automatically archived from the virtual tape library into Amazon S3 Glacier or Amazon S3 Glacier Deep Archive, further reducing storage costs.

Volume Gateway provides block storage volumes over iSCSI, backed by Amazon S3, and provides point-in-time backups as Amazon EBS snapshots. Volume Gateway integrates with AWS Backup, an automated and centralized backup service, to protect Storage Gateway volumes.

Getting started with AWS Storage Gateway

When getting started with Storage Gateway, customers typically identify an initial use case. Many customers begin their cloud journey by identifying low risk, high value use cases such as moving backups and archives to the cloud.

Once you’ve identified your use case, deploying Storage Gateway can be done in a matter of minutes. There are three main steps in the deployment process.

For a step-by-step demo and tutorial on how to get cloud storage in minutes with AWS Storage Gateway, check out this video:

Step 1: Deploy Storage Gateway

To get started, you must first log in to the AWS Management Console to create a new Storage Gateway. When creating a new Storage Gateway, you are prompted to select a type (Amazon S3 File Gateway, Amazon FSx File Gateway, Volume Gateway, or Tape Gateway). You are then prompted to download a virtual appliance to deploy in your environment. Select the type of virtual machine based on your environment, download the image, and proceed with importing to your environment. The following screenshot depicts all currently available deployment options.

You must allocate at least 150 GB of local disk storage to the virtual machine (VM). This is where Storage Gateway caches data locally, providing low-latency access to your most active data, with optimized data transfers occurring to and from AWS Cloud storage in the background. The amount of cache required depends on your workload. For details on determining the size of your cache, see the documentation on managing local disks for your AWS Storage Gateway.

The Storage Gateway automatically connects to the AWS Cloud over an encrypted secure connection. For additional security, customers have the option to route the encrypted traffic over a private connection through a Virtual Private Cloud (VPC) endpoint. For more information, see the documentation on activating a gateway in a VPC. In order to protect sensitive information for regulated workloads, customers can also use Federal Information Processing Standards (FIPS) 140-2 compliant endpoints in AWS GovCloud (US) Regions.

To connect your newly deployed Storage Gateway and the AWS Cloud, you’re going to activate the gateway. When completing your activation, you have the option to enable logging by configuring an Amazon CloudWatch log group to notify you about the health of your Amazon S3 File Gateway and its resources. For more information, visit getting S3 File Gateway health logs with CloudWatch Log Groups.

If you are running a VMware environment on-premises, you can also set up and test VMware High Availability. For information on setting up VMware High Availability in your environment, read the blog about deploying a highly available AWS Storage Gateway on a VMware vSphere cluster. You can monitor the status of VMware HA from the AWS Storage Gateway console, and run an ad hoc test manually at any time. The following screenshot is an example of an Amazon S3 File Gateway deployed in a VMware Cluster on-premises with High Availability enabled. The status, as well as the time and date of the last successful HA test for the gateway, are captured and displayed.

Step 2: Provision storage

Once activated, you can immediately start provisioning storage from the AWS Storage Gateway console. The protocol and underlying AWS Storage service depend on the gateway type you deployed:

• With the Amazon S3 File Gateway, you can provision file shares backed by Amazon S3 and access them via NFS or SMB.
• With the Amazon FSx File Gateway, you can access file shares managed in Amazon FSx for Windows File Server via SMB.
• If you deployed a Tape Gateway, you create virtual tapes backed by S3 that you can access via an iSCSI-VTL interface.
• With the Volume Gateway, you provision volumes backed by S3 that you can access via iSCSI.

Step 3: Connect clients

Your on-premises applications can now seamlessly access data in AWS Storage by connecting to the local gateway.

It’s that simple. By deploying Storage Gateway, customers gain the benefits of simplified storage management, and reduced costs and complexity for key hybrid cloud storage use cases.

Conclusion

AWS Storage Gateway is a hybrid cloud storage service designed to help customers who want to consume cloud storage in their data centers, remote offices, and edge locations for any stage in their cloud journey. Getting started is simple, as you can deploy Storage Gateway on premises within minutes without needing to make any changes to your existing applications.

Backed by the AWS Cloud, Storage Gateway provides virtually unlimited cloud storage, while providing low latency access to frequently accessed cached data. Storage Gateway provides four different types of gateways: Amazon S3 File Gateway, Amazon FSx File Gateway, Tape Gateway, and Volume Gateway. These gateways address use cases such as moving backups to the cloud, using on-premises file shares backed by cloud storage, and providing low latency access to data in AWS for on-premises applications.

For more information, including use cases, customer stories, and helpful videos, check out the AWS Storage Gateway product page. To get started, visit the AWS Storage Gateway console.

Thanks for reading this blog post, please leave any questions or comments in the comments section!