Cloud storage in minutes with AWS Storage Gateway

UPDATE: An updated version of this blog post was published on April 27, 2021. Please refer to that post for the most up-to-date content.

Moving workloads to the cloud is one of the primary ways for CIOs to address their top strategic priorities: increasing agility, accelerating innovation, strengthening security, and reducing cost. Many companies have existing infrastructure in data centers and remote offices but want to take advantage of cloud services. AWS helps these customers accelerate their journey to the cloud by providing hybrid cloud storage services to create a seamless extension of their on-premises infrastructure in AWS. As companies evaluate their applications, some applications are relatively straightforward to move to the cloud, while others are more complex. Applications may need to remain on-premises for performance or compliance reasons, or because they require tight integration with local IT infrastructure. For these reasons, organizations want to explore hybrid cloud storage solutions that provide on-premises access to data stored in AWS.

AWS Storage Gateway is a hybrid cloud storage solution that helps customers overcome these challenges and bridge the gap between their on-premises environments and the cloud. Storage Gateway enables on-premises applications to use cloud storage by providing low-latency data access over standard storage protocols. A local cache stores your most recently used data on-premises, and the cloud provides scalability as well as industry-leading data protection, durability, availability, security, and performance. Storage Gateway provides a simple first step into the cloud and is designed to be easy to deploy, easy to activate, and easy for your applications to access. In this blog post, I describe the core features and functionality of Storage Gateway and review hybrid storage architectures from a high-level. I also define the three types of Storage Gateway (File, Volume, Tape), reference examples of common customer use cases, and show you how to get started by walking through the 4 main steps when deploying a Storage Gateway in your environment.

Figure 1: This outlines common challenges between on-premises environments and the AWS Cloud

Introduction to Storage Gateway

A Storage Gateway can be deployed directly in your on-premises environment as a virtual machine (VMware ESXi, Microsoft Hyper-V, Linux KVM) or as a pre-configured standalone hardware appliance. Moreover, Storage Gateway requires no special networking or additional hardware and provides:

• Support for standard storage protocols such as NFS, SMB, iSCSI, and iSCSI VTL, so existing applications can use AWS Cloud storage without changes.
• A local cache for low-latency access for your applications.
• Optimized and secure data transfers between on-premises and the AWS Cloud.
• Interoperability with other AWS Cloud storage services such as Amazon S3, Amazon S3 Glacier, and Amazon Elastic Block Store (Amazon EBS).
• Integration with other AWS services such as AWS Key Management Service (KMS), AWS Identity and Access Management (IAM), AWS CloudTrail, and Amazon CloudWatch.

Storage Gateway combines the benefits of on-premises infrastructure with all the benefits of the cloud.

Figure 2: High-level architecture of Storage Gateway

Three common use cases for Storage Gateway include moving backups to the cloud, using on-premises file shares backed by cloud storage, and providing low latency access for on-premises applications to cloud data.

Figure 3: Three common use cases of Storage Gateway, which can be deployed across various stages of the cloud adoption journey

To support these use cases AWS provides three different types of gateways: File Gateway, Volume Gateway, and Tape Gateway.

Figure 4: Storage Gateway is made up of three gateway types: File Gateway, Volume Gateway, Tape Gateway.

File Gateway provides access to Amazon S3 over standard storage protocols, SMB, and NFS. Each file is stored as an object in Amazon S3 with a one-to-one mapping.

Volume Gateway provides block storage volumes over iSCSI, backed by Amazon S3, and provides point-in-time backups as Amazon EBS snapshots. Volume Gateway integrates with AWS Backup, an automated and centralized backup service, to protect Storage Gateway volumes.

Tape Gateway provides a virtual tape library, with each virtual tape being stored in Amazon S3, and supports all major backup applications. Tape Gateway is designed to be a seamless drop-in replacement to your existing physical tape infrastructure. When you no longer require immediate or frequent access to data contained on a virtual tape, you can have your backup application archive it. Your data goes from the virtual tape library into Amazon S3 Glacier or Amazon S3 Glacier Deep Archive, further reducing storage costs.

Getting started with AWS Storage Gateway

When getting started with Storage Gateway, customers typically identify an initial use case and then deploy a Storage Gateway on-premises for testing until moving to production. Many customers begin their cloud journey by identifying low risk, high value use cases such as moving backups and archives to the cloud. By leveraging the cloud for these workloads, customers can reduce the on-premises capacity of their primary storage systems, reduce their physical storage footprint, and in some cases, consolidate data centers.

Some examples include: storing SQL Server backups in Amazon S3 using AWS Storage Gateway, replacing physical tape-based backups with Tape Gateway, and migrating archive file data to AWS with File Gateway.

Once you’ve identified your use case, deploying Storage Gateway can be done in a matter of minutes. There are four main steps in the deployment process.

For a step-by-step demo and tutorial on how to get cloud storage in minutes with AWS Storage Gateway, check out this video:

Step 1: Deploy Storage Gateway

To get started, you must first log in to the AWS Management Console to create a new Storage Gateway. When creating a new Storage Gateway, you are prompted to select a type (file, volume, or tape). You are then prompted to download a pre-configured virtual appliance to deploy in your environment. Select the type of virtual machine, download the image, and proceed with importing to your environment. The following screenshot depicts the deployment options for your environment.

You must allocate at least 150 GB of storage for the local cache. The amount of cache required depends on the workload. For details on determining the size of your cache, see the documentation on managing local disks for your AWS Storage Gateway.

The Storage Gateway automatically connects over the internet to the AWS Cloud over an encrypted Secure Sockets Layer (SSL) connection. For additional security, customers have the option to route the encrypted traffic over a private connection through a VPC endpoint. For more information, see the documentation on activating a gateway in a Virtual Private Cloud.

Step 2: Activate Storage Gateway

Activation ensures proper connectivity between your newly deployed Storage Gateway and the AWS Cloud.

When completing your activation, you have the option to enable logging by configuring an Amazon CloudWatch log group to notify you about the health of your File Gateway and its resources. For more information, see the documentation on getting File Gateway health logs with CloudWatch Log Groups.

For additional monitoring, Storage Gateway generates metrics that include: CacheHitPercent (percent of read operations served from the cache), CachePercentDirty (percentage of the cache that hasn’t yet uploaded to AWS), and resource utilization of the gateway (memory and CPU). For more information on monitoring see the documentation on monitoring Storage Gateway. The following screenshot is an example of the main dashboard for your storage gateway, which includes visibility into the metrics that have been generated over a customizable period of time.

If you are running a VMware environment on-premises, you can also set up and test VMware High Availability. For information on setting up VMware High Availability in your environment, read the blog about deploying a highly available AWS Storage Gateway on a VMware vSphere cluster. You can monitor the status of VMware HA from the AWS Management Console, and run an ad hoc test manually at any time. The following screenshot is an example of a File Gateway that was deployed in a VMware Cluster on-premises with High Availability enabled. The status, as well as the time and date of the last successful HA test for the gateway, are capture and displayed.

Step 3: Create storage

Once activated, you can immediately begin creating storage from the AWS Management Console. Your storage type depends on the type of Storage Gateway you are deploying, File Gateway file shares (SMB or NFS), Volume Gateway volumes (iSCSI), or Tape Gateway Tapes (iSCSI VTL).

Step 4: Connect clients

Your AWS Cloud storage is now accessible on-premises over your local network, and you can begin to connect clients and applications as you would with a traditional storage system.

It’s that simple. By deploying Storage Gateway, customers gain the benefits of simplified storage management and reduced costs and complexity for key hybrid cloud storage use cases.

Conclusion

AWS Storage Gateway is a hybrid cloud storage solution designed to help customers at any stage in their cloud journey. Getting started is simple, as Storage Gateway can be deployed on-premises within minutes and requires no changes to your existing applications. Because it is backed by the AWS Cloud, Storage Gateway provides virtually unlimited cloud storage, while providing low latency access to frequently accessed cached data. Storage Gateway provides three different types of gateways: File Gateway, Volume Gateway, and Tape Gateway. For more information, including use cases, customer stories, and helpful videos, check out the AWS Storage Gateway product page.

Thanks for reading this blog post, please leave any questions or comments in the comments section!