Creating and activating File Gateway on VMware

Wouldn’t it be great if your on-premises applications could use AWS Cloud Storage, without the need for any modifications? Do you want to use standard file storage protocols, such as SMB or NFS? If so, you can use File Gateway, which is part of AWS Storage Gateway.

File Gateway presents SMB or NFS file shares for on-premises applications to store files as Amazon Simple Storage (Amazon S3) objects and access them with traditional file interfaces. Using File Gateway, you can map NFS or SMB shares as network paths on your backup applications to back up data from on-premises to Amazon S3.

You can deploy File Gateway in your on-premises environment as a Virtual Machine (VM) which uses VMware ESXi or Microsoft Hyper-V hypervisor. In this blog, we go through the steps to deploy a File Gateway OVA image as a VM on VMware ESXi hypervisor. I start creating a File Gateway by downloading a VM image from the AWS Management Console. After deploying and configuring the VM, I activate the File Gateway and assign local disks for cache storage.

After activation, I join File Gateway to an Active Directory domain. Then, I create and map an SMB file share with Active Directory authentication to my Windows client. After mapping the file share, I upload a test file to my SMB share, which is transferred to my Amazon S3 bucket.

File Gateway

File Gateway enables you to store and retrieve objects in Amazon S3 using file protocols, such as NFS or SMB, as shown in the following diagram (Figure 1). Files written through File Gateway can be directly accessed in Amazon S3.

Figure 1: AWS File Gateway

Choose a Host Platform and Download the VM

File Gateway provides the VM in an Open Virtual Appliance (.OVA) package, which consists of the VM image. To download the VM image, log in to the AWS Management Console, go to the AWS Storage Gateway Console and, select Create gateway. Then, select File Gateway and Next (Figure 2).

Figure 2: Select gateway type

Next, select VMware ESXi and Download Image, which downloads the aws-storage-gateway-latest.ova file (Figure 3).

Figure 3: Select host platform

Deploy OVF Template

In this section, I show you the steps to create a File Gateway VM by deploying the OVF template. I also show you the configuration settings for compute, storage, and networks for the File Gateway VM.

After downloading the OVA file, log in to the VMware vSphere web console. Choose the resource pool under ESX host, right-click, and select Deploy OVF Template (Figure 4).

Figure 4: Select resource pool under ESX host

Under Deploy OVF Template, select Local file, navigate to the path where you have the .OVA file downloaded, and select Next (Figure 5).

Figure 5: Deploy OVF Template

Provide a name and select a location for the VM, then select Next (Figure 6).

Figure 6: Provide VM name and selecting location

Choose the compute resource where you want to place this VM, then select Next (Figure 7).

Figure 7: Select a compute resource

In the next section, verify and review the template details. Then, select Next (Figure 8).

Figure 8: Review and verify the template details

Under Select storage, choose the VM Datastore (for the gateway VM root disk and VM files) and select Next (Figure 9). Please choose the thick provisioned format. The disk storage is allocated immediately and ensures that the File Gateway functions without any errors.

Figure 9: Select storage for the VM

Under Select networks, choose the networking vSwitch configuration that has outbound internet connection to AWS Storage Gateway endpoints. Then, select Next (Figure 10).

Figure 10: Select network

Verify your selections for compute, storage, and network configurations for the VM and select Finish (Figure 11).

Figure 11: Verify the selections

Add a new VM disk for cache and verify network connectivity

Here, I show you the steps to create a new VM disk that is used for cache storage on the File Gateway. Then, I perform network connectivity checks to ensure connectivity from the File Gateway VM to Storage Gateway public endpoints.

File Gateway requires a minimum of 4vCPUs, 16 GiB of dedicated RAM, and one local disk allocated to the VM as cache disk. The cache disk stores recently accessed data locally and provides low latency access to that data.

To create a new VM disk, right-click on the VM and select Edit Settings (Figure 12).

Select ADD NEW DEVICE (Figure 13) and choose add hard disk. Provide 150-GB minimum size for the cache disk (Figure 14).

Provisioning a cache disk in proportion to the active working dataset provides lower latency for clients accessing the file shares. You can provision a maximum of 16-TiB size for the cache disk. For optimal performance, it is recommended to select a different datastore for the cache disk than the datastore used for the root disk.

To increase cache on the File Gateway, always create a new VM disk and assign that new disk as a cache disk. Do not extend the size of the disks that are already allocated.

Figure 12: Select Edit Settings for creating a new VM disk

Figure 13: Select ADD NEW DEVICE

Figure 14: Provide the 150-GB disk size

At this point, you must synchronize the VM time with host time to successfully activate the gateway (Figure 15). Ensure that your host clock is set to the correct time and synchronize it with a Network Time Protocol (NTP) server.

Figure 15: Synchronize the VM time with host

After hard disk creation, power on the VM and log in to the VM console using user name “admin” and password “password” (Figure 16). I used DHCP to assign the IP address and DNS servers. If you want to assign a static IP address and configure DNS server IP addresses for the gateway, you can configure them by choosing 2: Network Configuration (Figure 17).

Figure 16: VM console

To ensure that the gateway VM has network connectivity to AWS Storage Gateway public endpoints, select 3: Test Network Connectivity (Figure 17). Here I use public endpoints and activate the gateway into Storage Gateway public endpoints. So, let’s select public and run network checks to us-west-2 Region (Figure 18).

Figure 17: Test Network Connectivity

Figure 18: Network connection checks

Connect and activate gateway

In this section, I show you the steps that are required to activate the File Gateway.

Once the network checks pass, you can activate the gateway from the AWS Storage Gateway web console by providing the IP address of the gateway VM. Choose File Gateway, choose the host platform as VMware ESXi, and select Next (Figure 19):

Figure 19: Select the host platform

Select Public as the endpoint type and Next (Figure 20). Then provide the IP address of the gateway VM IP and choose Connect to gateway (Figure 21). In order to successfully activate the gateway, your browser must be able to connect to the VM IP address on port 80.

Figure 20: Select service endpoint

Figure 21: Provide IP address and connect to gateway

With the gateway connected to successfully, you see the Activate gateway page (Figure 22). If you are unable to connect to the gateway, check that the firewall to the gateway VM IP address is open on port 80.

Provide a time zone and Gateway name and select Activate gateway (Figure 23). Next, you see the message as “Gateway is now active.” At this point, the gateway recognizes the assigned VM disk for local cache storage. In this phase, the gateway is trying to recognize the local cache disk, which takes a few minutes.

Figure 22: Provide gateway name and time zone while activating gateway

Once the gateway recognizes the local disk for cache, the Configure local disks page appears (Figure 23). Then, select Configure logging to go to next page. On the next page, choose Gateway Log Group from the drop-down (Figure 24). You can configure logging with Amazon CloudWatch Logs and get notified about Amazon S3 errors in CloudWatch Logs. If you don’t have an existing CloudWatch log group, you can select Create new Log Group to create a new CloudWatch log group. You can refer to this documentation to create a CloudWatch log group.

Figure 23: Configure local disks

Once the CloudWatch log group is selected, choose Verify VMware HA.

Figure 24: Select Gateway Log Group

As my File Gateway VM is not deployed to a cluster with vSphere HA, I am selecting Exit (Figure 25). If you want to deploy your File Gateway to VMware vSphere cluster with HA, you can refer to this blog for additional information.

Figure 25: Selecting Exit for Verify VMware HA

After selecting Exit, I can see that my File Gateway is successfully created (Figure 26).

Figure 26: Successfully created gateway

Join File Gateway to Active Directory domain, and creating a file share

After the gateway is activated, you can create an SMB or NFS file share, test upload a sample file from the client, and verify that the file transferred successfully to S3. In this example, I joined File Gateway to an Active Directory domain and created an SMB share with Active Directory authentication.

To join File Gateway to an Active Directory domain, go to Actions and select Edit SMB settings. Under Active Directory settings, provide the Domain Name, Domain user, and Domain password. Then, select Save (Figures 27, 28, and 29).

Figure 27: Editing SMB settings to join domain

Figure 28: Join domain request status

Figure 29: Success message after joining gateway to Active Directory domain

To create a file share, you first select Create File share. Then, choose an Amazon S3 bucket in your account, select Active Directory access, and create the file share. You can leave the defaults for IAM role and other values and select create file share (Figure 30). You can refer this documentation for detailed steps to create a file share.

Figure 30: Select create file share

Once the file share is created, you can map the file share using the following command from Windows command prompt (Figure 31). For additional details regarding mapping the file share, you can refer this documentation.

Figure 31: Command to map the SMB file share

You can now see the mapped drive on File Explorer and upload a sample test file. You would see that file uploaded to your Amazon S3 bucket of the file share (Figure 32).

Figure 32: Mapped network drive

You can now see the above file, which is created on my SMB client and is uploaded to my selected Amazon S3 bucket (Figure 33).

Figure 33: S3 bucket console

You can also create an NFS share and export it to your Linux clients. Using File Gateway, you can enable your existing file-based applications, devices, and workflows to use Amazon S3, without modification. After data is transferred to Amazon S3, you can take advantage of Amazon S3 capabilities such as lifecycle management, versioning, and Cross-Region replication. You can also run hybrid cloud workloads using data generated by on-premises applications for processing by AWS services. Among these services include machine learning, big data analytics, and serverless functions. You can also maintain low latency access to your on-premises applications for the cached data.

Summary

In this post, I showed you how to create and activate a File Gateway on VMware ESXi hypervisor. You can also deploy File Gateway on Microsoft’s Hyper-V hypervisor or on an Amazon Elastic Compute Cloud (Amazon EC2) instance. You can refer to this documentation to deploy a File Gateway on Amazon EC2 Host.

Thank you for taking the time to read this blog post. If you may have any questions or feedback, you are invited to share them in the comments section below.