Q: What is the AWS Storage Gateway service?
The AWS Storage Gateway service enables hybrid storage between on-premises environments and the AWS Cloud. It seamlessly integrates on-premises enterprise applications and workflows with Amazon’s block and object cloud storage services through industry standard storage protocols. It provides low-latency performance by caching frequently accessed data on premises, while storing data securely and durably in Amazon cloud storage services. It provides an optimized data transfer mechanism and bandwidth management, which tolerates unreliable networks and minimizes the amount of data being transferred. It brings the security, manageability, durability, and scalability of AWS to existing enterprise environments through native integration with AWS encryption, identity management, monitoring, and storage services. Typical use cases include backup and archiving, disaster recovery, moving data to S3 for in-cloud workloads, and tiered storage.
AWS Storage Gateway supports three storage interfaces: file, volume, and tape. Each gateway you have can provide one type of interface.
The file gateway enables you to store and retrieve objects in Amazon S3 using file protocols, such as NFS. Objects written through file gateway can be directly accessed in S3.
The volume gateway provides block storage to your applications using the iSCSI protocol. Data on the volumes is stored in Amazon S3. To access your iSCSI volumes in AWS, you can take EBS snapshots which can be used to create EBS volumes.
The tape gateway provides your backup application with an iSCSI virtual tape library (VTL) interface, consisting of a virtual media changer, virtual tape drives, and virtual tapes. Virtual tape data is stored in Amazon S3 or can be archived to Amazon Glacier.
Q: How do I use the AWS Storage Gateway service?
You can have two touchpoints to use the service: the AWS Management Console and a gateway that is available as a virtual machine (VM) or as a physical hardware appliance.
You use the AWS Management Console to download the virtual appliance gateway or purchase the hardware appliance, configure storage, and manage and monitor the service. The gateway connects your applications to AWS storage by providing standard storage interfaces. It provides transparent caching, efficient data transfer, and integration with AWS monitoring and security services.
To get started, sign up for an AWS account and visit the AWS Storage Gateway Management Console to download a gateway VM appliance, or purchase the hardware appliance. Once you’ve installed your gateway, you associate it with your AWS Account through our activation process. After activation, you configure the gateway to connect to the appropriate storage type. For file gateway, you configure file shares that are mapped to selected S3 buckets, using IAM roles. For volume gateway, you create and mount volumes as iSCSI devices. For tape gateway, you connect your backup application to create and manage tapes. Once configured, you start using the gateway to write and read data to and from AWS storage. You can monitor the status of your data transfer and your storage interfaces through the AWS Management Console. Additionally, you can use the API or SDK to programmatically manage your application’s interaction with the gateway.
Q: What is file gateway?
File gateway presents a file-based interface to Amazon S3, which appears as a network file share. It enables you to store and retrieve Amazon S3 objects through standard file storage protocols. File gateway allows your existing file-based applications or devices to use secure and durable cloud storage without needing to be modified. With file gateway, your configured S3 buckets will be available as Network File System (NFS) mount points or Server Message Block (SMB) file shares. Your applications read and write files and directories over NFS or SMB, interfacing to the gateway as a file server. In turn, the gateway translates these file operations into object requests on your S3 buckets. Your most recently used data is cached on the gateway for low-latency access, and data transfer between your data center and AWS is fully managed and optimized by the gateway. Once in S3, you can access the objects directly or manage them using features such as S3 Lifecycle Policies, object versioning, and cross-region replication. You can run file gateway on-premises or in EC2.
Q: What is volume gateway?
Volume gateway provides an iSCSI target, which enables you to create block storage volumes and mount them as iSCSI devices from your on-premises or EC2 application servers. The volume gateway runs in either a cached or stored mode.
- In the cached mode, your primary data is written to S3, while retaining your frequently accessed data locally in a cache for low-latency access.
- In the stored mode, your primary data is stored locally and your entire dataset is available for low-latency access while asynchronously backed up to AWS.
In either mode, you can take point-in-time snapshots of your volumes, which are stored as Amazon EBS Snapshots in AWS, enabling you to make space-efficient versioned copies of your volumes for data protection, recovery, migration and various other copy data needs.
Q: What is tape gateway?
Tape gateway is a cloud-based Virtual Tape Library (VTL). It presents your backup application with a VTL interface, consisting of a media changer and tape drives. You can create virtual tapes in your virtual tape library using the AWS Management Console. Your backup application can read data from or write data to virtual tapes by mounting them to virtual tape drives using the virtual media changer. Virtual tapes are discovered by your backup application using its standard media inventory procedure. Virtual tapes are available for immediate access and are backed by Amazon S3. You can also archive tapes. Archived tapes are stored in Amazon S3 Glacier or Amazon S3 Glacier Deep Archive.
Q: What benefits does AWS Storage Gateway provide?
AWS Storage Gateway provides a set of features that enable you to effectively leverage AWS storage within your existing applications and workflows. It provides a standard set of protocols such as iSCSI, SMB and NFS, which allow you to use your existing applications without any changes. Through its local cache, the gateway provides low-latency access to recently used data. The gateway optimizes data transfer to AWS storage, such as optimization of transfer through intelligent buffering, upload management to address network variations, and bandwidth management. The gateway provides you an effective mechanism to store data in AWS across the range of storage services most suitable for your use cases. The gateway is easy to deploy and can use your existing virtual infrastructure and hypervisor investments, or can be installed in your data center or remote offices as a hardware appliance. The gateway software running as a VM or on the hardware appliance is stateless, allowing you to easily create and manage new instances of your gateway as your storage needs evolve. Finally, the service integrates natively into AWS management services such as Amazon CloudWatch, AWS CloudTrail, AWS Key Management Service (KMS), and AWS Identity and Access Management (IAM).
Q: What sort of encryption does AWS Storage Gateway use to protect my data?
All data transferred between any type of gateway appliance and AWS storage is encrypted using SSL. By default, all data stored by AWS Storage Gateway in S3 is encrypted server-side with Amazon S3-Managed Encryption Keys (SSE-S3). Also, you can optionally configure different gateway types to encrypt stored data with AWS Key Management Service (KMS) via the Storage Gateway API. See below for specifics on KMS support by File Gateway, Volume Gateway, and Tape Gateway.
Q: Is AWS Storage Gateway HIPAA eligible?
Yes. AWS Storage Gateway is HIPAA eligible. If you have an executed Business Associate Agreement (BAA) with AWS, you can use Storage Gateway to store, backup and archive protected health information (PHI) on scalable, cost-effective, and secure AWS storage services, including Amazon S3, Amazon Glacier and Amazon EBS, which are also HIPAA eligible.
Information on HIPAA eligible services on AWS can be found at our HIPAA Compliance page, and you can enter into a BAA with AWS here. HIPAA eligibility for Storage Gateway applies to all gateway types (File, Volume and Tape).
Q: Is AWS Storage Gateway PCI compliant?
Yes, the AWS Storage Gateway in compliance with the Payment Card Industry Data Security Standard (PCI DSS) based on recent assessments. Existing customers can download the Attestation of Compliance (AOC) and PCI Responsibility Summary reports in the AWS Management Console with AWS Artifact, and prospective customers can request them through AWS sales.
Q: What AWS Storage Gateway types can I manage through AWS Backup?
You can manage backup and retention policies for cached and stored volume modes of Volume Gateway through AWS Backup.
Q: What is file gateway?
File Gateway is a configuration of the AWS Storage Gateway service that provides your applications a file interface to seamlessly store files as objects in Amazon S3, and access them using industry standard file protocols.
Q: What can I do with file gateway?
Use cases for file gateway include: (a) migrating on-premises file data to Amazon S3, while maintaining fast local access to recently accessed data, (b) Backing up on-premises file data as objects in Amazon S3 (including Microsoft SQL Server and Oracle databases and logs), with the ability to use S3 capabilities such as lifecycle management, versioning and cross region replication, and, (c) Hybrid cloud workflows using data generated by on-premises applications for processing by AWS services such as machine learning, big data analytics or serverless functions.
Q: What are the benefits of using file gateway to store data in S3?
File gateway enables your existing file-based applications, devices, and workflows to use Amazon S3, without modification. File gateway securely and durably stores both file contents and metadata as objects, while providing your on-premises applications low-latency access to cached data.
Q: Which Amazon S3 storage classes does File Gateway support?
File gateway supports Amazon S3 Standard, S3 Standard - Infrequent Access (S3 Standard - IA) and S3 One Zone - IA. For details on storage classes, refer to the Amazon S3 documentation. You configure the initial storage class for objects that the gateway creates, and then you can use bucket lifecycle policies to move files from Amazon S3 to Amazon Glacier. If an application attempts to access a file/object stored through File Gateway that is now in Amazon Glacier, you will receive a generic I/O error.
Q: What protocols does file gateway support?
File gateway supports Linux clients connecting to the gateway using Network File System (NFS) versions 3 and 4.1 for Linux clients, and supports Windows clients connecting to the gateway using Server Message Block (SMB) versions 2 and 3.
Q: How can I create and use a file share?
You can create an NFS or SMB file share using the AWS Management Console or service API and associate the file share with a new or existing Amazon S3 bucket. To access the file share from your applications, you mount it from your application using standard UNIX or Windows commands. For convenience, example command lines for each environment are shown in the management console.
Q: What options do I have to configure an NFS file share?
You can configure your NFS file share with administrative controls such as limiting access to specific NFS clients or networks, read-only or read-write, or enabling user permission squashing.
Q: What options do I have to configure an SMB file share?
You can configure your SMB file share to be accessed by Active Directory (AD) users only or provide authenticated guest access to users in your organization. You can further limit access to the file share as read-only or read-write, or to specific AD users and groups.
Q: Does file gateway support integration with on-premises Microsoft Active Directory (AD)?
Yes, file gateway integrates with Microsoft Active Directory on-premises as well as with in-cloud Active Directory solutions such as Managed Microsoft AD.
Q: Can I export an SMB file share without Active Directory?
Yes. You can export an SMB file shares using a guest username and password. You will need to change the default password using the Console or service API before setting up your file share for guest access.
Q: Can I export a mix of NFS and SMB file shares on the same gateway?
Q: Can I export an NFS and SMB file share on the same bucket?
No, currently file metadata, such as ownership, stored as S3 object metadata cannot be mapped across different protocols.
Q: How does file gateway access my S3 bucket?
File gateway uses an AWS Identity and Access Management (IAM) role to access your S3 bucket. You can set up an IAM role yourself or have it automatically setup by the AWS Storage Gateway Management Console. For automatic setup, AWS Storage Gateway will create a new IAM role in your account and associate it with an IAM Access Policy to access your S3 bucket. The IAM role and IAM access policy are created in your account and you can fully manage them yourself.
Q: How does my application access my file share?
To use the file share, you mount it from your application using standard UNIX or Windows commands. For convenience, example command lines are shown in the management console
Q: What is the relationship between files and objects?
Files are stored as objects in your S3 buckets and you can configure the initial storage class for objects that file gateway creates. There is a one-to-one relationship between files and objects, and you can configure the initial storage class for objects that file gateway creates.
The object key is derived from the file path within the file system. For example, if you have a gateway with hostname file.amazon.com and have mapped my-bucket, then file gateway will expose a mount point called file.amazon.com:/export/my-bucket. If you then mount this locally on /mnt/my-bucket and create a file named file.html in a directory /mnt/my-bucket/dir this file will be stored as an object in the bucket my-bucket with a key of dir/file.html. Creating sparse files will result in a non-sparse zero-filled object in S3.
Q: What file system operations are supported by file gateway?
Your clients can create, read, update, and delete, files and directories. Files are stored as individual objects in Amazon S3. Directories are managed as folder objects in S3, using the same syntax as the S3 console. Symbolic links and hard links are not supported. Attempting to create a link will result in an error.
Rename operations will appear atomic to your clients, but S3 does not support renaming of objects. When you rename a file or directory the gateway performs copy-put requests to create a copy of the objects in S3 under the new keys and then deletes the originals. This avoids having to re-send large files over the network. Renaming directories containing a large number of files is not instantaneous and will result in 2 copies of your data being stored in S3 until the rename operation completes.
Q: What file system metadata can my client access and where is the metadata stored?
Your clients can access POSIX-style metadata including ownership, permissions, and timestamps that are durably stored in S3 in the user metadata of the object associated with the file. When you create a file share on an existing bucket, the stored metadata will be restored and made accessible to your clients.
Q: How do I set the Content-Type for files uploaded to S3?
For each file share, you can enable guessing of MIME types for uploaded objects upon creation or enable the feature later. If enabled, file gateway will use the filename extension to determine the MIME type for the file and set the S3 objects Content-Type accordingly. This is beneficial if you are using file gateway to manage objects in S3 which you access directly via URL or distribute through Amazon CloudFront.
Q: Can I directly access objects stored in S3 by using file gateway?
Yes. Once objects are stored in S3, you can access them directly in AWS for in-cloud workloads without requiring file gateway. Your objects inherit the properties of the S3 bucket in which they are stored, such as lifecycle management, and cross-region replication.
An object that needs to be accessed by using a file share should only be managed by the gateway. If you directly overwrite or update an object previously written by file gateway, it results in undefined behavior when the object is accessed through the file share.
Q: What if my bucket already contains objects?
If your bucket already contains objects when you configure it for use with file gateway, object keys will be used to present the objects as files to the NFS and SMB clients. The files are given default file system metadata.
To reduce latency and number of S3 requests, file gateway only scans the headers for file metadata associated with the objects when you explicitly list the files or directories. File metadata is collected as a part of that scan, file contents are downloaded only when the object is read.
Q: How are buckets accessed by the gateway? Are entire bucket or file contents downloaded?
The gateway does not automatically download full objects or all the data that exists in your bucket; data is only downloaded when it is explicitly accessed by your clients. Additionally, to reduce data transfer overhead, file gateway uses multipart uploads and copy put, so only changed data in your files is uploaded to S3.
Q: What metadata can my NFS client access for objects created outside of the gateway?
For objects uploaded to the S3 bucket directly, i.e. not using file gateway and an NFS share, you can configure default ownership and permissions.
Q: What metadata can my SMB client access for objects created outside of the gateway?
For objects uploaded to the S3 bucket directly, i.e. without using file gateway and an SMB share, metadata such as ownership and permissions will be inherited from the object’s parent folder. Permissions at the root of the share are fixed and objects created directly under the root folder will inherit these fixed permissions. Refer to the documentation on metadata settings of objects created outside the gateway.
Q: Can I use multiple NFS clients with a single file gateway?
You can have multiple NFS clients accessing a single file gateway. However, as with any NFS server, concurrent modification from multiple NFS clients can lead to unpredictable behavior. Application level coordination is required to do this in a safe way.
Q: Can I have multiple writers to my S3 bucket?
No. We recommend a single writer to objects in your S3 bucket. If you directly overwrite or update an object previously written by file gateway, it results in undefined behavior when the object is accessed through the file share. Concurrent modification of the same object (e.g. via the S3 API and the file gateway) can lead to unpredictable results and we recommend against this configuration.
Q: Can I have two gateways writing independent data to the same bucket?
We do not recommend configuring multiple writers to a single bucket because it can lead to unpredictable results. You could enforce unique object names or prefixes through your application workflow. File gateway doesn’t monitor or report on conflicts in such a setup.
Q: Can I have multiple gateways reading data from the same bucket?
Yes, you can have multiple readers on a bucket managed through a file gateway. You can configure a file share as read-only, and allow multiple gateways to read objects from the same bucket. Additionally, you can refresh the inventory of objects that your gateway knows about using the Storage Gateway Console or the RefreshCache API.
Note however that the if you do not configure a file share as read-only, file gateway does not monitor or restrict these readers from inadvertently writing to the bucket. It is up to you to maintain a single writer/multi reader configuration from your application.
Q: Can I monitor my file share using Amazon CloudWatch?
Yes, you can monitor usage of your file share using Amazon CloudWatch metrics and get notified on completion of file operations through CloudWatch Events. To learn more, visit Monitoring your File Share.
Q: How do I know when my file is uploaded?
When you write files to your file share with file gateway, the data is stored locally first and then asynchronously uploaded to your S3 bucket. You can request notifications through AWS CloudWatch Events when this upload completes. These notifications can be used to trigger additional workflows, such as invoking an AWS Lambda function or Amazon EC2 Systems Manager Automation, which is dependent upon the data that is now available in S3. To learn more, please refer to the documentation for File Upload Notification.
Q: Can I update my file gateway’s view of a bucket to see objects created from an object-based workload or another file gateway?
Yes, you can refresh the inventory of objects that your file gateway knows about using the Console or the RefreshCache API. You will receive notifications through AWS CloudWatch Events when the refresh cache operation has completed. These notifications can be used to send emails using Amazon SNS, or trigger local processing using the updated contents. To learn more, please refer to the documentation.
Q: Can I use the gateway to update data in a bucket that belongs to another AWS account?
Yes, you can use the gateway for cross-account access to buckets. To learn more, please refer to the documentation for Using File Share for Cross-Account access.
Q: Can I use the gateway to access data in Requester Pays S3 buckets?
Yes, when creating your file share you can enable access to Requester Pays S3 buckets. As a requester, you will incur the charges associated with accessing data from Requester Pays buckets.
Q: How many files shares can I create per bucket?
There is a one-to-one mapping between a file share and a bucket. We do not limit the number of file shares per bucket. However, we recommend having a single writer to the bucket, either a file gateway or client accessing S3 directly.
Q: How many file shares can I create per gateway?
You can create up to 10 file shares per gateway.
Q: What is the maximum size of an individual file?
The maximum size of an individual file is 5 TB, which is the maximum size of an individual object in S3. If you write a file larger than 5 TB, you will get a "file too large" error message and only the first 5 TB of the file will be uploaded.
Q: My application checks storage size before copying data. What storage size does the gateway return?
The gateway returns a large number (8 EB) as your total capacity. Amazon S3 does not limit total storage.
Q: Can I use versioning, lifecycle, cross-region replication, and S3 event notification?
Yes. Your bucket policies for versioning, lifecycle management, cross-region replication, and S3 event notification, apply directly to objects stored in your bucket through AWS Storage Gateway.
You can use S3 lifecycle policies to change an object's storage tier or delete old objects or object versions. In the case of objects deleted by lifecycle policy, you will need to call the RefreshCache API to reflect these changes to your NFS clients.
When using an S3 bucket which is the target for cross-region replication, you may need to use the RefreshCache API to ensure the gateway cache and S3 bucket are in sync.
If using S3 event notifications you may receive events for partial files created by the gateway to ensure your data is durably stored in S3. Partial files may occur for a number of reasons, such as the gateway needing to free up cache space, or a high rate of writes to a file. These partial files may not be application consistent.
Q: Can I use file gateway with my backup application?
File gateway supports SMB versions 2 and 3 as well as NFS versions 3 and 4.1. We are continuing to do on-going testing with common backup apps. Please let us know of any specific apps with which you'd like to see compatibility tested.
Q: Can I use file gateway to write files to EFS?
No. File gateway allows you to store files as objects in S3.
Q: When should I use file gateway vs. the S3 API?
You can use file gateway when you want to access objects in S3 as files using standard filesystem operations. File gateway additionally provides low-latency local access and efficient data transfer. You can use the S3 API when your application doesn’t require file system operations and can manage data transfer directly.
Q: How does file gateway manage the local cache? What data gets stored locally?
Local disk storage on the gateway is used to temporarily hold changed data that needs to be transferred to AWS, and to locally cache data for low-latency read access. File gateway automatically manages the cache maintaining the most recently accessed data based on client read and write operations. Data is evicted from the cache only when space is needed to store more recently used data.
To maximize write performance, the gateway uses a write-back mechanism where data is first persisted to disk and then asynchronously uploaded to S3. The gateway serves data from the local cache to maximize read performance. If not present, data is efficiently synchronously fetched from Amazon S3 using byte-range gets.
The local cache should generally be sized for the working set of data that you need low-latency access to. If the cache is too small then read latencies will increase as data being requested must be fetched from S3, and writes could fail if there is no free cache space to store data locally pending upload to S3.
Q: What guidance should I use to provision size of the gateway’s cache disk? What happens if I provision a smaller cache disk?
You should provision your cache based on:
1/ The size of your working dataset to which you need low-latency access, so you can reduce read latencies by decreasing the frequency with which data is requested from S3, and
2/ The size of files written to the gateway by your applications.
Smaller cache disks can result in poor performance and failures during writes if there is no free cache space to store data locally when pending upload to S3. To learn more about monitoring your cache usage, refer to Monitoring Your File Share in the documentation.
Q: When does data in the cache get evicted?
Data written to the cache from your applications or through retrieval from Amazon S3 is evicted from the cache only when space is needed to store more recently accessed data.
Q: Does file gateway perform data reduction (deduplication or compression)?
No. Files are mapped to objects one-to-one in your bucket without modification, enabling you to access your data directly in S3 without needing to use the gateway or deploy additional software to rehydrate your data.
File gateway uses multipart uploads and copy put, so only changed data is uploaded to S3 which can reduce data transfer. The gateway does not automatically download full objects or all the data that exists in your bucket; data is only downloaded when explicitly accessed by your NFS client.
Q: Can I use file gateway with Amazon S3 Transfer Acceleration?
File gateway will not use the accelerated endpoints even if your bucket is configured for S3 Transfer Acceleration.
Q: What sort of encryption does file gateway use to protect my data?
All data transferred between the gateway and AWS storage is encrypted using SSL. By default, all data stored in S3 is encrypted server-side with Amazon S3-Managed Encryption Keys (SSE-S3). For each file share you can optionally configure to have your objects encrypted with AWS KMS-Managed Keys using SSE-KMS. To learn more, please see “Encrypting Your Data Using AWS Key Management System,” in the Storage Gateway User Guide, which includes critical details about usage of the feature.
Q: How much volume data can I manage per gateway? What is the maximum size of a volume?
Each volume gateway can support up to 32 volumes. In cached mode, each volume can be up to 32 TB for a maximum of 1 PB of data per gateway (32 volumes, each 32 TB in size). In stored mode, each volume can be up to 16 TB for a maximum of 512 TB of data per gateway (32 volumes, each 16 TB in size). For more information, please refer to our documentation on Storage Gateway limits.
Volume gateways compress data before that data is transferred to AWS and while stored in AWS. This compression can reduce both data transfer and storage charges. Volume storage is not pre-provisioned; you will be billed for only the amount of data stored on the volume, not the size of the volume you create.
Q: When I look in Amazon S3 why can’t I see my volume data?
Your volumes are stored in an Amazon S3 bucket maintained by the AWS Storage Gateway service. Your volumes are accessible for I/O operations through AWS Storage Gateway. You cannot directly access them using Amazon S3 API actions. You can take point-in-time snapshots of gateway volumes that are made available in the form of Amazon EBS snapshots, which can be turned into either Storage Gateway Volumes or EBS Volumes. Use the file gateway to work with your data natively in S3.
Q: What sort of encryption does volume gateway use to protect my data?
All data transferred between the gateway and AWS storage is encrypted using SSL. By default, all data stored by volume gateway in S3 is encrypted server-side with Amazon S3-Managed Encryption Keys (SSE-S3).
You can optionally configure encryption for data stored in AWS on volumes using AWS KMS managed keys via the Storage Gateway API. You will be able to specify one of the managed Customer Master Keys (CMKs) as the KMS key. The configured CMK used to encrypt a volume cannot be changed after creation. To learn more, please see “Encrypting Your Data Using AWS Key Management System,” in the Storage Gateway User Guide, which includes critical details about usage of the feature.
Q: Can I create an EBS Snapshot from KMS-encrypted volume?
Yes. You can create an EBS snapshot from an AWS KMS-encrypted volume using the API. The EBS snapshot will be encrypted using the same key as the one used for volume encryption
Q: Can I create a volume from KMS-encrypted EBS snapshot?
Yes. You can create an encrypted volume from KMS-encrypted EBS snapshot using the API. The encrypted volume can use the same key that was used to encrypt the EBS snapshot, or you can specify a different encryption key for encrypting the volume.
Q: Why would I use snapshots?
You can take point-in-time snapshots of your volume gateway volumes in the form of Amazon EBS snapshots.You can use a snapshot of your volume as the starting point for a new Amazon EBS volume, which you can then attach to an Amazon EC2 instance. Using this approach, you can easily supply data from your on-premises applications to your applications running on Amazon EC2 if you require additional on-demand compute capacity for data processing or replacement capacity for disaster recovery purposes.
For cached volumes, where your volume data is already stored in Amazon S3, you can use snapshots to preserve versions of your data. Using this approach, you can revert to a prior version when required or repurpose a point-in-time version as a new volume. You can initiate snapshots on a scheduled or ad hoc basis. When taking a new snapshot, only the data that has changed since your last snapshot is stored. If you have a volume with 100 GB of data, but only 5 GB of data have changed since your last snapshot, only the 5 additional GB of snapshot data will be stored in Amazon S3. When you delete a snapshot, only the data not needed for any other snapshot is removed.
For stored volumes, where your volume data is stored on-premises, snapshots provide durable, off-site backups in Amazon S3. You can create a new volume from a snapshot if you need to recover a backup. You can also use a snapshot of your volume as the starting point for a new Amazon EBS volume which you can then attach to an Amazon EC2 instance.
Q: What data will my snapshot contain? How do I know when to take a snapshot to ensure my data is backed up?
Snapshots represent a point-in-time copy of the volume at the time the snapshot is requested. They contain all of the information needed to restore your data (from the time the snapshot was taken) to a new volume. Data written to the volume by your application prior to taking the snapshot, but not yet been uploaded to AWS, will be included in the snapshot.
In practical terms, the snapshot will be assigned an ID and visible in the AWS Management Console and AWS Command Line Interface (AWS CLI) immediately, but will initially be in a PENDING status. When all data written to the volume prior to the snapshot request has been uploaded from the gateway and into EBS, the status will change to AVAILABLE. At this point you can use the snapshot as the base for a new gateway or EBS volume.
Q: How do I restore a snapshot to a gateway?
Each snapshot is given a unique identifier that you can view using the AWS Management Console. You can create AWS Storage Gateway or Amazon EBS volumes based on any of your existing snapshots by specifying this unique identifier.
Using the AWS Management Console, you can create a new volume from a snapshot you’ve stored in Amazon S3. You can then mount this volume as an iSCSI device to your on-premises application server.
Because cached volumes store your primary data in Amazon S3, when creating a new volume from a snapshot, your gateway keeps the snapshot data in Amazon S3 where it becomes the primary data for your new volume.
Because stored volumes store your primary data locally, when creating a new volume from a snapshot, your gateway downloads the data contained within the snapshot to your local hardware. There it becomes the primary data for your new volume.
Q: Do the AWS Storage Gateway’s volumes need to be un-mounted in order to take a snapshot? Does the snapshot need to complete before the volume can be used again?
No, taking snapshots does not require you to un-mount your volumes, nor does it impact your application’s performance. However, snapshots only capture data that has been written to your AWS Storage Gateway volume, which may exclude any data that has been locally buffered by your application or OS.
Q: Can I schedule snapshots of my AWS Storage Gateway volumes?
Yes, you can create a snapshot schedule for each of your volumes. You can modify both the time the snapshot occurs each day, as well as the frequency (every 1, 2, 4, 8, 12, or 24 hours).
Q: How long does it take to complete a snapshot?
The time it takes to complete a snapshot is largely dependent upon the size of your volume and the speed of your Internet connection to AWS. The AWS Storage Gateway compresses all data prior to upload, reducing the time to take a snapshot.
Q: Will I be able to access my snapshot data using Amazon S3’s APIs?
No, snapshots are only accessible from the AWS Storage Gateway and Amazon EBS and cannot be directly accessed using Amazon S3 APIs.
Q: What are the snapshot limits per gateway?
There are no limits to the number of snapshots or the amount of snapshot data a single gateway can produce.
Q: What are the benefits of using AWS Backup to protect my Volume Gateway volumes?
Using AWS Backup to backup Volume Gateway volumes simplifies and centralizes backup management, thus reducing operational burden and making it easier to meet compliance requirements across all your AWS resources. AWS Backup allows you to set customizable scheduled backup policies that meet your backup requirements. Using AWS Backup, you can set backup retention and expiration rules so you no longer need to develop custom scripts or manually manage the point-in-time backups of your Volume Gateway volumes. Finally, you can manage and monitor backups across multiple Volume Gateways, and other AWS resources such as EBS volumes and RDS databases, from a central view.
Q: How do I protect volumes on Volume Gateway using AWS Backup?
You can use AWS Backup to either take a one-time backup or define a backup schedule for Volume Gateway volumes. The volume backups are stored in Amazon S3 as Amazon EBS snapshots and visible in AWS Backup console or Amazon EBS console. The volume backups created by AWS Backup can manually or automatically be deleted from AWS Backup console.
Q: How do I use AWS Backup to manage backup and retention of my Volume Gateway volumes?
You can start from either the Storage Gateway console or the AWS Backup console to manage your backups. If you start from the Storage Gateway console, you have the ability to navigate to the AWS Backup console to complete your backup plan configuration or initiate an on-demand backup. Alternatively, you can start from the AWS Backup console to configure your backup plan or initiate an on-demand backup of Volume Gateway volumes.
Q: Does anything change with how I have been using Volume Gateway volumes today?
No. All existing Volume Gateway snapshot functionality and your existing Amazon EBS Snapshots remain available and unchanged. You can continue to use the Storage Gateway console to create volumes from your EBS Snapshots and use the Amazon EBS console to view or delete your snapshots.
Q: If I use AWS Backup, can I also continue to use Volume Gateway snapshot schedules and existing snapshots?
Yes. You can continue to use Volume Gateway’s existing snapshot capabilities to create Amazon EBS snapshots and use your previously created snapshots for restore purposes. AWS Backup’s backup schedule operates independently from the Volume Gateway scheduled snapshots, and provides you an additional way to centrally manage all your backup and retention policies.
Q: If I have a KMS-encrypted volume on Volume Gateway, will AWS Backup be able to backup that volume?
Yes. AWS Backup will backup KMS-encrypted volumes Volume Gateway with the same key as the one used for volume encryption.
Q: Can I use AWS Backup to create a backup of my Volume Gateway volume in a different region (e.g. cross region)?
AWS Backup supports backup of Volume Gateway volumes within the same region in which AWS Backup operates.
Q: What are the benefits of storing virtual tapes in AWS compared to warehousing tapes offsite?
You get 11 9s of data durability, fixity checks by AWS on a regular basis, data encryption, right data when you restore, and cost savings, when storing virtual tapes in AWS using Tape Gateway with S3 Glacier Deep Archive compared to warehousing physical tapes offsite. First, all virtual tapes stored in S3 Glacier Deep Archive are replicated and stored across at least three geographically-dispersed Availability Zones, protected by 11 9s of durability. Second, AWS performs fixity checks on a regular basis to confirm your data can be read and no errors have been introduced. Third, all tapes stored in S3 Glacier Deep Archive are protected by S3 Server Side Encryption using default keys or your KMS keys. In addition, you also avoid physical security risk associated with tape portability. Fourth, compared to the experience of warehousing tapes offsite where you may receive an incorrect or broken tape during restore, with Tape Gateway, you always get correct data. Finally, you can save in monthly storage costs when storing your data in S3 Glacier Deep Archive compared to warehousing tapes offsite.
Q: What Amazon S3 storage classes does Tape Gateway support?
Tape Gateway supports S3 Standard, S3 Glacier, and S3 Glacier Deep Archive storage classes. When backup application is backing up data to tapes, your data is in virtual tape library in Amazon S3. You can archive tapes in virtual tape library to S3 Glacier (supported today) and S3 Glacier Deep Archive.
Q: How much data can I store on a virtual tape?
When creating virtual tapes, you select one of the following sizes: 100 GB, 200 GB, 400 GB, 800 GB, 1.5 TB, and 2.5 TB. Please note, you only pay for the amount of data stored on each tape, and not for the size of the tape.
Q: How many tapes can the virtual tape library (VTL) hold?
A single tape gateway can have up to 1,500 virtual tapes in the VTL with a maximum aggregate capacity of 1 PB, however there is no limit to the amount of data or number of virtual tapes you can archive. You can also deploy additional tape gateways to scale storage for virtual tapes that are not archived. For more information, please see our documentation on Storage Gateway limits.
Q: How much data can I store in tape archives?
There is no limit to the amount or size or virtual tapes that you can archive.
Q: Which S3 storage classes can I retrieve my archived virtual tape to?
You can retrieve a virtual tape archived in S3 Glacier or S3 Glacier Deep Archive to S3. A tape archived in S3 Glacier is retrieved to S3 using standard retrieval method in 3-5 hours. A tape archived in S3 Glacier Deep Archive is retrieved to S3 using standard retrieval method within 12 hours.
Q: How do I access my data on virtual tapes?
The virtual tape containing your data must be stored in a virtual tape library before it can be accessed. Access to virtual tapes in your virtual tape library is instantaneous. If the virtual tape containing your data is archived, you can retrieve the virtual tape using the AWS Management Console or API. First select the virtual tape, then choose the virtual tape library into which you want the virtual tape to be loaded. It takes 3-5 hours to retrieve a tape from S3 Glacier and within 12 hours from S3 Glacier Deep Archive in the selected virtual tape library. Once the virtual tape is available in the virtual tape library, you can use your backup application to make use of the virtual tape to restore data.
Q: Will I be able to access the virtual tapes in my virtual tape library using Amazon S3 or Amazon S3 Glacier APIs?
No. You cannot access virtual tape data using Amazon S3 or Amazon S3 Glacier APIs. However, you can use the tape gateway APIs to manage your virtual tape library and your virtual tape shelf.
Q: How do I use Tape Gateway with S3 Glacier Deep Archive storage class?
When creating new tapes through the Storage Gateway console or API, you can set archival storage target to S3 Glacier Deep Archive. When your backup software ejects the tapes, they will be archived to S3 Glacier Deep Archive. You can retrieve virtual tapes archived in S3 Glacier Deep Archive to S3 using standard retrieval method within 12 hours.
Q: Can I move my existing virtual tapes in S3 Glacier to S3 Glacier Deep Archive?
No. Tape Gateway supports archiving only new tapes directly to S3 Glacier Deep Archive. We will provide customers the ability to move their existing tapes in S3 Glacier to S3 Glacier Deep Archive in Q2’19.
Q: What backup applications can I use with tape gateway?
The VTL interface is compatible with backup and archival applications that use the industry-standard iSCSI-based tape library interface. For a full list of the supported backup applications see the requirements section of the AWS Storage Gateway user guide.
Q: What sort of encryption does tape gateway use to protect my data?
All data transferred between the gateway and AWS storage is encrypted using SSL. By default, all data stored by tape gateway in S3 is encrypted server-side with Amazon S3-Managed Encryption Keys (SSE-S3).
You can optionally configure encryption on tapes using AWS KMS-Managed Keys via the Storage Gateway API. You will be able to specify one of the managed Customer Master Keys (CMKs) as the KMS key. The configured CMK used to encrypt tape data cannot be changed after creation. To learn more, please see “Encrypting Your Data Using AWS Key Management System,” in the Storage Gateway User Guide, which includes critical details about usage of the feature.
Q: What is the Storage Gateway hardware appliance?
AWS Storage Gateway is available as a hardware appliance, which has Storage Gateway software pre-installed on a Dell EMC PowerEdge R640 server with a validated configuration. You manage the appliance from the AWS Management Console or API.
Q: What gateway types and storage interfaces are supported on the hardware appliance?
The hardware appliance supports File Gateway with NFS and SMB interfaces, Volume Gateway cached volumes with iSCSI, and Tape Gateway with iSCSI-VTL.
Q: Why might I need a hardware appliance?
The hardware appliance further simplifies procurement, deployment, and management of AWS Storage Gateway on-premises for IT environments such as remote offices and departments which lack existing virtual server infrastructure, adequate disk and memory resources, or staff with hypervisor management skills. It avoids having to procure additional infrastructure necessary for a virtual environment in order to operate the local Storage Gateway VM appliance.
Q: How many models of hardware appliances are available?
We are offering a single model at this time.
Q: What are the specifications of the hardware appliance?
The hardware appliance is based on a Dell EMC PowerEdge R640 server. Please refer to Storage Gateway hardware appliance for specifications.
Q: Where is the hardware appliance available? With which AWS regions does it work?
The hardware appliance can be shipped to US and Europe addresses. It can be used with the US East (Northern Virginia), US East (Ohio), US West (Oregon), US West (Northern California), EU (Frankfurt), EU (Ireland), EU (London), and EU (Paris) AWS regions. File Gateway file shares may be added against any global partition Amazon S3 bucket.
Q: Where do I buy the hardware appliance?
We recommend you start at AWS Storage Gateway management console to purchase the hardware appliance. In the Storage Gateway console when you create a new gateway, you have the option to use a hardware appliance in addition to using virtual platforms VMware ESXi, Microsoft Hyper-V, and Amazon EC2. If you don’t already own a hardware appliance, you are directed to Amazon US, Amazon UK, or Amazon Germany to purchase it. You can choose to purchase the appliance from these websites based on your delivery location preference using your Amazon or Amazon Business account.
Q: Who owns the hardware appliance?
After purchase, you own the hardware appliance.
Q: How do I use the hardware appliance?
Once you receive the hardware appliance, you configure IP address through the local hardware console, and use this IP address in the AWS Storage Gateway management console to activate your appliance. This associates your hardware appliance with your AWS account. Once the hardware appliance is activated, you select your desired gateway type from the console, either file, volume (cached), or tape. The selected type of gateway is then enabled on the appliance. Once activated, you manage and use your new hardware gateway appliance with the AWS Console, CLI, or SDK, similar to how you would with the virtual appliance today. For more information, please see the hardware appliance documentation.
Q: Can I run multiple gateways on a single hardware appliance?
No. Currently, the hardware appliance supports running only one gateway at a time.
Q: Can I change the type of gateway once it is installed on a hardware appliance?
Yes. To change the gateway type after it is installed on a hardware appliance, you choose Remove Gateway from the Storage Gateway console, which deletes the gateway and all associated resources. At that point, you are free to launch a new gateway on the hardware appliance.
Q: How can I purchase and use additional storage on the Storage Gateway hardware appliance?
You will have the option to purchase additional storage on Amazon US, Amazon UK, or Amazon Germany. You can purchase the base appliance which offers 5 TB usable storage and choose to add a package of 5 x 1.92 TB SSDs to bring the appliance’s usable storage capacity to 12 TB.
To expand your storage, simply insert the SSDs into the pre-configured appliance. The SSDs are hot pluggable, and the appliance will automatically recognize the extra storage upon adding SSDs to the appliance. See the documentation here for instructions.
Q: How can I purchase a fiber optic network card for the Storage Gateway hardware appliance?
You will have the option to purchase an Intel X710 4-port 10 Gigabit fiber optic network card on Amazon US, Amazon UK, or Amazon Germany for Storage Gateway hardware appliance. You can select the fiber optic network card option in addition to the base appliance when ordering the appliance. Upon receipt of the appliance and the fiber optic network card, you will swap out the 10 Gigabit copper network card with the fiber optic network card using instructions here.
Q: Can I add more storage to a Storage Gateway hardware appliance I had purchased before?
Yes. You can buy 5 x 1.92 TB SSDs available on Amazon US, Amazon UK, or Amazon Germany and add them to the appliance. If you have already activated the appliance and associated it with your AWS account, you will need to factory reset it before adding more storage.
Q: Can I add any SSD or hard drive to increase storage capacity for my Storage Gateway hardware appliance?
No. At this time you can only add the SSDs that are available for purchase on Amazon US, Amazon UK, or Amazon Germany. The SSDs available for purchase are qualified by Dell and AWS for the Storage Gateway hardware appliance.
Q: Does the Storage Gateway hardware appliance support RAID?
Yes. The hardware appliance uses software-based ZFS RAID and provides protection against storage drive failure. The base appliance offering 5 TB usable storage tolerates failure of 1 SSD and the 12 TB usable storage configuration tolerates failure of 2 SSDs.
Performance, monitoring, and maintenance
Q: What performance can I expect?
The AWS Storage Gateway sits between your applications and Amazon storage services. The performance you experience depends on what host platform (hardware appliance, virtual machine, Amazon EC2 instance) you are using to run Storage Gateway software and a number of other factors. These include the network bandwidth between your iSCSI initiator or NFS client and gateway, the speed and configuration of your underlying local disks, the configuration of your VM, the amount of local storage allocated to your gateway, and the bandwidth between your gateway and Amazon storage.
Our technical documentation provides guidance on how to optimize your AWS Storage Gateway environment for best performance.
Q: What are the minimum hardware and software requirements for the AWS Storage Gateway?
For running AWS Storage Gateway on a virtual machine or an Amazon EC2 instance, see the requirements section in the AWS Storage Gateway User Guide. AWS Storage Gateway is also available as a Hardware Appliance with pre-validated specifications.
Q: Can I use the AWS Storage Gateway with AWS Direct Connect?
Yes, you can use AWS Direct Connect to increase throughput and reduce your network costs by establishing a dedicated network connection between your on-premises gateway and AWS. Note that the AWS Storage Gateway efficiently uses your Internet bandwidth to help speed up the upload of your on-premises application data to AWS.
Q: Can I route my AWS Storage Gateway Internet traffic through a local proxy server?
Yes. Volume and tape gateways support configuration of a Socket Secure version 5 (SOCKS5) proxy between your on-premises gateway and AWS. File gateways support configuration of an HyperText Transfer Protocol (HTTP) proxy.
Q: What type of data reduction does AWS Storage Gateway perform?
Volume and tape gateways perform compression of data in-transit and at-rest which can reduce both data transfer and storage charges. The AWS Storage Gateway only uploads data that has changed, minimizing the amount of data sent over the Internet.
Q: Does the AWS Storage Gateway support bandwidth throttling?
Yes, using the AWS Management Console you can restrict the bandwidth between your tape and volume gateway and AWS based on a rate that you provide. You can specify individual rates for inbound and outbound traffic.
Q: How do I monitor my gateway?
You can use Amazon CloudWatch to monitor the performance metrics for your gateway, giving you insight into storage, bandwidth, throughput, and latency. These metrics are accessible directly from CloudWatch; or by following links in the AWS Storage Gateway Console, which take you directly to the CloudWatch metrics for the resource being viewed. Please refer to the CloudWatch details and pricing pages for additional information.
Q: How can I measure the cache performance of my gateway?
You can use Amazon CloudWatch metrics including CachePercentDirty, CacheHitPercent, CacheFree, and CachePercentUsed. These can be viewed by following the Monitoring link on the gateway details tab in the AWS Storage Gateway Console.
Q: How can I measure the bandwidth used by my gateway?
You can use Amazon CloudWatch metrics including CloudBytesUploaded and CloudBytesDownloaded.
Q: How does the AWS Storage Gateway manage updates?
AWS Storage Gateway periodically deploys important updates and software patches to your gateway virtual machine (VM). You can configure a weekly maintenance schedule allowing you to control when these updates will be applied to your gateway. Alternatively, you can apply updates manually when they are made available, either through the AWS Storage Gateway Console or API. Updates should take only a few minutes to complete. For more information, please visit the Managing Gateway Updates section of our documentation.
Q: How will I be billed for my use of AWS Storage Gateway?
There are 3 elements to how you will be billed for AWS Storage Gateway: Storage, requests, and data transfer. For detailed pricing information, please visit the AWS Storage Gateway Pricing page.
Q: How will I be charged for file storage when using a file gateway?
File gateways stores data directly in Amazon S3. You are billed by Amazon S3 for the objects stored and requests made by your file gateway. For more information, please visit the Amazon S3 Pricing page.
Q: How will I be charged for volume or virtual tape storage when using a volume or tape gateway?
You are billed for the amount of volume and virtual tape data you store in AWS. This fee is prorated daily and prices vary by region. You are only billed for the portion of volume or virtual tape capacity that you use, not for the provisioned size of the resource. All volume and virtual tape data is compressed before it is transferred to AWS by the gateway, which can reduce your storage charges. For detailed pricing information, please visit the AWS Storage Gateway Pricing page.
Q: How will I be charged for EBS snapshots taken from my AWS Storage Gateway volumes?
EBS snapshots taken from your Storage Gateway volumes are stored and billed by Amazon EBS. When taking a new snapshot only the data that has changed since your last snapshot is stored to reduce your storage charges. For more information, please visit the Amazon EBS Pricing page.
Q: How will I be charged for reading and writing data?
When your gateway writes data to AWS you will be charged at a flat rate of $0.01 per GB of data written to AWS up to a monthly maximum of no more than $125 per gateway. There is no charge for reading data from AWS. Since the gateway performs caching, bandwidth optimization, and, for volume and tape gateways, compression, the amount of data written to AWS may be less than the amount of data written to the gateway by your application. You can monitor the amount of data written by your gateway to AWS through the provided Amazon CloudWatch metrics and you can configure bandwidth limits on your gateway to manage your costs.
Q: How will I be charged when retrieving data on an archived virtual tape?
You are charged, when retrieving a virtual tape that has been archived in S3 Glacier, at a flat rate of $0.01 per GB of data stored on the tape. For example, retrieving 5 tapes that contain 100 GB each would cost 5 x 100GB x $0.01 = $5.00.
Q: How will I be charged for deleting an archived virtual tape?
If a virtual tape is deleted within three months of being archived in S3 Glacier or within six months of being archived S3 Glacier Deep Archive, you will be charged an early deletion fee. If the virtual tape has been stored for three months or longer in S3 Glacier or for six months or longer in S3 Glacier Deep Archive, there is no charge for deletion.
In the US East (Northern Virginia) Region, you would be charged a prorated early deletion fee of $0.012 per GB deleted within three months. For example, if you delete 1 virtual tape containing 1 GB of data 1 month after uploading archiving it in S3 Glacier, you would be charged a $0.008 early deletion fee. If, instead you delete the same virtual tape after 2 months, you would be charged a $0.004 early deletion fee.
Q: How am I charged for virtual tapes I store in S3 Glacier Deep Archive?
Virtual tapes stored in S3 Glacier Deep Archive will be charged S3 Glacier Deep Archive storage class rate. You can visit Storage Gateway pricing webpage to review Tape Gateway pricing.
Q: How will I be charged for network data transfer to and from AWS when using AWS Storage Gateway?
You are billed for Internet data transfer for each GB downloaded from AWS to your gateway. All data transfer for uploading to AWS is free.
Q: How can I tell how much storage I am going to be billed for?
The Billing and Cost Management console shows an estimate of month-to-date usage for each service, including AWS Storage Gateway volumes and virtual tapes. For a breakdown of usage by individual volume or virtual tape Detailed Billing Reports enables you to see usage for each resource on a daily basis.
Q: When using file gateway, will I incur S3 request charges?
You will pay for the S3 requests made by file gateway on your behalf to store and retrieve your files in S3 as objects. The gateway caches data up to the capacity of the local disks you allocate, which can help reduce costs for data retrieval.
Q: When does each monthly billing cycle begin?
The billing system follows Coordinated Universal Time (UTC). The calendar month begins midnight UTC on the first day of every month.
Q: Do your prices include taxes?
Except as otherwise noted, our prices are exclusive of applicable taxes and duties, including VAT and applicable sales tax. For customers with a Japanese billing address, use of the Asia Pacific (Tokyo) Region is subject to Japanese Consumption Tax.
Q: How much does the hardware appliance cost?
Q: How do I pay for the hardware appliance?
Q: Can I lease or rent the hardware appliance?
No. You pay the full price at the time of purchase.
Q: Does AWS Premium Support cover the AWS Storage Gateway?
Yes, AWS Premium Support covers issues related to your use of the AWS Storage Gateway. Please see the AWS Premium Support detail page for further information and pricing.
Q: What other support options are available?
You can tap into the breadth of existing AWS community knowledge through the AWS Storage Gateway discussion forum.
Q: Who do I call for support related to hardware appliance?
You contact AWS Support, who provides AWS Storage Gateway software and service support. AWS Support also coordinates and hands over any cases related to Dell EMC hardware to a fully trained Dell EMC support team. We recommend that you purchase AWS Premium Support.
Q: Where do I find the Dell EMC service tag for the hardware appliance (also known as serial number)?
The Dell EMC service tag for the hardware appliance can be found in the Hardware view of AWS Storage Gateway console.
Q: What if there is a hardware problem with the hardware appliance?
AWS Support works with Dell EMC for hardware support. Hardware support is included with your appliance purchase and includes 36 months of 7x24 phone support and next-business-day, on-site service for parts replacement.
Q: What are the warranty terms of the hardware appliance?
The hardware appliance comes with 3 years of warranty from Dell with next business day onsite service for parts replacement. You can find warranty information under Product description section of the hardware appliance listing on Amazon US, Amazon UK, or Amazon Germany.