How to create SMB file shares with AWS Storage Gateway using Hyper-V

Businesses across different industries can face several challenges when managing their storage, including new redundancy requirements across locations, hardware renewals, and maximum capacity limits. To purchase additional hardware, companies typically have to take an investment that involves capacity planning for the next three to five years, as well as a potentially lengthy procurement process for the final purchase and delivery. In addition, they must deal with the heavy lifting that comes with the installation and most of the used storage is reserved for archives and backups.

In this post, we show you how to easily create a hybrid-cloud setup to move non-frequently accessed files to the cloud, all while maintaining known protocols. Using a hybrid-cloud storage setup helps you to free up space in your data centers. It can provide an alternative to purchasing additional hardware. This post builds on the blog post Cloud storage in minutes with AWS Storage Gateway and elaborates on extending your storage for infrequently accessed data in an on-premises Windows environment in four steps. We walk through the setup of an AWS Storage Gateway as a Hyper-V virtual machine (VM) and using the AWS Management Console to create Server Message Block (SMB) file shares. It’s possible to apply Active Directory (AD) permissions the same way as with any other SMB, and AD compatible file share that might already be used on-premises.

Before beginning with the walkthrough, let’s have a look on the relevant components and architecture.

Types of AWS Storage Gateway gateways

The Storage Gateway offers multiple types of gateways that serve different purposes and is backed by different types of cloud storage.

1. Amazon S3 File Gateway
2. Amazon FSx File Gateway
3. Volume Gateway
4. Tape Gateway

Volume and Tape Gateway are not used in this solution. To learn more about them, visit the preceding links.

Amazon S3 File Gateway

With the S3 File Gateway, it’s possible to create NFS or SMB files shares that are accessible from the on-premises network. Files stored on those shares are uploaded to Amazon S3 using Hypertext Transfer Protocol Secure (HTTPS). To improve performance and latency of the shares, Amazon S3 File Gateways are equipped with local cache disks that hold most recently accessed files.

Typical, but not exclusive, use cases for this setup are:

1. Backup or archive files that are not frequently accessed
2. Having the need to store files in a physically separated environment
3. Ingesting locally generated data into a data lake for downstream processing in the cloud

Amazon FSx File Gateway

The Amazon FSx File Gateway works similar to the S3 File Gateway. Instead of storing files on Amazon S3, it stores the files on Amazon FSx for Windows File Server, which provides fully managed file storage built on Windows Server. If you are looking for a multi-user interactive file sharing workload, for example an end-user facing application like Microsoft Office, this would be the preferred option. In addition to Getting started with Amazon FSx File Gateway you can also find a discussion on use cases that are best served by Amazon FSx File Gateway.

Solution overview

For cloud storage, we use Amazon S3. Amazon S3 is a fully managed object storage that automatically stores your objects across multiple devices spanning a minimum of three Availability Zones, each separated by miles across the same AWS Region. Hence, we achieve high availability and durability and files are protected against hardware failures.

To upload files to Amazon S3, we use an Amazon S3 File Gateway as an on-premises component. It’s used to create SMB and Network File System (NFS) file shares and map them to an Amazon S3 Bucket. Files stored on the file shared are then cached and asynchronously uploaded to Amazon S3 using encryption in transit and at rest. Most recently accessed files remain in a configurable local cache, up to 64 TB, on the Amazon S3 File Gateway to provide local performance and preserve end user experience.

Walkthrough

Let’s get started with the setup of an Amazon S3 File Gateway using a Hyper-V VM within a Windows environment.

Disclaimer: Be aware that the AWS Free Tier does not completely cover this deployment. Use the AWS pricing calculator to create an estimate beforehand.

The steps of this process are:

1. Deploy the S3 File Gateway on-premises.
2. Connect the gateway to your AWS account.
3. Create a file share backed by Amazon S3.
4. Use the file share.

Prerequisites

• AWS account and credentials with administrative rights (more information can be found on “How do I create and activate a new AWS account”)
• On-premises: Microsoft Hyper-V (version 2012 R2, 2016, 2019 or 2022), 16 GB memory, 80 GB storage for the S3 File Gateway VM and at least 150 GB for the cache
• On-premises: A user with privileges to join the gateway VM to the AD domain

Step 1: Deploy the Amazon S3 File Gateway

To get started, open a web browser on a Windows client in the same network as of your Hyper-V host.

1. Log in to the AWS Management Console and open the Storage Gateway console.
2. Make sure you have selected the desired Region.

1. Select Create gateway.
2. Choose a name (for example, fgw-smb-fileshare-demo).
3. Check that the preselected time zone is correct for the location of your Storage Gateway.
4. Select Amazon S3 File Gateway.
5. Choose Microsoft Hyper-V.
6. Select Download image (ca. 2.3 GB).
7. Leave this session open.

Import virtual machine

After we have downloaded the Microsoft Hyper-V image, we need to import the Storage Gateway VM on the Hyper-V host. To do so, follow the instructions shown under Set up gateway on Microsoft Hyper-V.

TIP: We have used Import Type as Copy. The time synchronization can be checked under Settings… > Management > Integration Services after the import is finished.

Virtual hard disk for cache

In order to allow a seamless user experience, the Amazon S3 File Gateway uses a cache of at least 150 GB. For the best performance, you can calculate the right size of the cache for your individual situation. We now create this virtual hard disk using the Hyper-V manager.

1. Select the storage gateway VM and open Settings….
2. Add an additional hard drive by creating a new hard disk and select Next.
3. Choose VHDX as disk format and select Next.
4. Choose Fixed size as disk type and select Next.
5. Choose a name (for example, fgw-disk-smb-fileshare-demo.vhdx), select a folder location and select Next.
6. Now, set the size you calculated (150 GB in our case) and select Next.
7. Once the creation has finished, Apply the changes.

Network connectivity

The File Gateway does need local network access for providing the file share to end users or systems and an internet connection to connect to AWS service endpoints. If you are behind a firewall or proxy, make sure you can access the required endpoints and ports. Depending on the individual setup, the Amazon S3 File Gateway VM might have no internet connection. In that case, we need to change that and perform a test of the connectivity. This post assumes you have a DHCP service in your network where VM gets its IP address, as well as DNS servers and a gateway to use.

TIP: If you do not use DHCP, the network settings can be configured within the local VM.

1. Select the storage gateway VM and open Settings….
2. Select the adapter as virtual switch that provides access to your network.
3. Apply the changes.
4. Start the VM and connect.
5. Login using the default user name admin and with password password. We recommend changing the password. You can set the local console password from the Storage Gateway console.

After the previous steps are complete, then you see a screen similar to the one shown in the following screenshot. Make note of the local IP address of the VMs currently connected network adapter for later use.

Follow Test Your Gateway Connection to the Internet to check if networking requirements are met. To learn more, see performing tasks on the VM local console.

Step 2: Connect to Storage Gateway

By now, the local VM should be running and having connectivity to the required endpoints. Let’s activate it.

1. Return to the open AWS Management Console session, where we have started the creation of the Gateway.
2. Select the check box at the bottom under Confirm set up gateway to confirm the setup and select Next.

TIP: If your console session ended, simply re-open the AWS Storage Gateway console – just make sure you are in the right Region.

For the connection options:

1. Select publicly accessible. Note that if you prefer, your gateway can connect to the AWS Cloud over private connectivity, instead of directly over the internet. Either way your traffic is secure as it is encrypted in transit and at rest.
2. There are two options to activate. We’ll choose the IP address option for activation.
3. Paste the local IP address of the Amazon S3 File Gateway VM that you have noted earlier into the field.
4. Before selecting Next, ensure you are in your local network to allow the web browser to connect to the local IP address.

Configure gateway

You can configure the cache storage, logging, or alarms to notify you in case of service degeneration. It takes a few moments to discover local disks.

1. Choose Cache as Allocated to for the 150 GB disk.
2. Leave everything else as default.
3. Finish the configuration with Configure.

TIP: If your Amazon S3 File Gateway has additional disks attached, you can choose which one you want to use for Cache and which ones to ignore by setting them to Unallocated. As your workload demands increase additional cache disks can be added without interruption for best performance.

Step 3: Create a file share backed by Amazon S3

To use Amazon S3, we need to create an S3 bucket. A bucket is a container for objects; an Amazon S3 File Gateway translates files into objects, which is a 1:1 mapping, and includes all of the metadata such as timestamps and permissions.

Create bucket

1. Go to the Amazon S3 console.
2. Select Create bucket.
3. Specify a unique Bucket name (for example, fgw-bucket-smb-share-demo-20220819).

4. Confirm that Block all public access is turned on.

5. Under Default encryption, choose Enable for Server-side encryption and select Amazon S3 key (SSE-S3).
6. Leave everything else as default and select Create bucket.

Create file share

Now we are ready to create the file share.

1. Open the Storage Gateway console.
2. On the left menu, select File shares.
3. Choose Create file share.

File share settings

1. Select the gateway that was created before and fill in the name of the bucket just created.
2. Select Server Message Block (SMB) for the way to access objects. Keep the default settings and select Next.

Amazon S3 storage settings

1. Leave the default settings and select Next.

File access settings

1. Select Active Directory as authentication method.
2. Enter your Domain name, if it is not already set.
3. Enter user name and password of a domain administrator. The password will not be stored, but it is necessary to allow the local VM to perform a domain join.
4. Leave all other settings to their default values and select Next.

5. Once the join was successful, an overview is presented. Select Create to finish the setup.

In the background, the Amazon S3 File Gateway gets updated according to the settings just made. Give the gateway a minute to become Available. In case of any issues, you can use this troubleshooting article.

Mount the file share

Use Windows Map Network Drive wizard to mount the file share.

Step 4: Use the file share

The file share is now ready to use and you can start configuring your needed permissions and place data as needed. You can also assign group permissions to the folder. Now, test the setup and try moving files onto the file share or set permissions and check if they are applied.

When copying a larger file, like the Amazon Linux 2 VM image, you can see that the performance is not affected by the gateway because the local cache buffers within your network.

Comparing the file on the file share with the object in Amazon S3 shows the 1:1 mapping which keeps the original format.

Conclusion

In this post, we covered installing an Amazon S3 File Gateway within a Microsoft Windows setup using Microsoft Hyper-V and creating an SMB file share without needing to manage servers or change the existing environment. We discovered the use of Active Directory users and groups, and saw that files have been uploaded to Amazon S3 while maintaining fast local access to recently accessed data. To further optimize costs, you can leverage Amazon S3 Intelligent-Tiering and Amazon S3 Lifecycle configuration.

By using AWS Storage Gateway, it’s possible to extend your local storage setup and form a hybrid cloud environment for infrequently accessed files. From the different gateway types, we selected the Amazon S3 File Gateway as the simplest option to provide local file shares backed by cloud storage. The Amazon S3 cloud storage allowed files to be stored in a secure, reliable, and cost-efficient way. If you want to know more about when to use which type of File Gateway, check out the blog on Accessing your file workloads from on premises with File Gateway.

Using the Storage Gateway can help you free up local storage capacity by moving infrequently accessed files into a durable, scalable, and highly available cloud storage with cost-effective pricing. Instead of capacity guessing and lengthy procurement processes, it’s possible to start immediately by provisioning hybrid cloud storage and leveraging a pay as you go pricing model. Many customers take advantage of AWS Storage Gateway because it is non-disruptive to existing environments and can be deprovisioned easily.