Migration to Amazon FSx for Windows File Server: cost optimization strategies and automation

Businesses today rely heavily on their file server infrastructure to store, manage, and share mission-critical data. However, maintaining and scaling these environments can be a complex and resource-intensive task. As customer needs and data volumes continue to grow, organizations often face the need to modernize their file server infrastructure.

When migrating to Amazon FSx for Windows File Server, the lift-and-shift approach can make the process relatively easy. However, it’s important to ensure you have sufficient resources and bandwidth to handle the heavy I/O load during the migration. Underestimating your bandwidth needs can lead to slow transfer speeds and extended downtime.

Another key consideration is data deduplication. FSx for Windows File Server offers a data deduplication feature that can help you provision the right amount of storage for your workload, without over-provisioning and consequent additional migration effort. Leveraging this feature during migration can help you optimize migration effort and storage costs.

In this post, I will share strategies and best practices to help you navigate migrating an existing Windows Server file share to FSx for Windows File Server successfully. I’ll explore how to leverage automation tools like PowerShell to streamline the migration, reduce risks, and ensure a smooth transition to the highly available and scalable Amazon FSx platform.

Key considerations for migrating to Amazon FSx for Windows File Server

Two of the most common potential pain points when migrating Windows File Servers to Amazon FSx for Windows File Server are:

1. Data Size and deduplication:

1. • If the source file system is using Windows Deduplication, use the Get-DedupStatus PowerShell command to accurately predict the total storage required before creating your Amazon FSx for Windows File Server file system.
   • Size your FSx for Windows File Server file system at least 10% larger than the estimated deduplicated data, providing a buffer for unexpected overhead and future growth.
   • Set up an aggressive deduplication schedule on Amazon FSx Windows File Server and consider migrating the data in batches to optimize storage utilization.

2. Performance tuning:

1. • Temporarily increase the FSx for Windows File Server throughput capacity to accommodate higher data transfer rates and IOPS during the migration, and to provide the required resources for the demanding Data Deduplication jobs.
   • Use the Amazon FSx performance table to determine the appropriate memory and throughput configuration for the Data Deduplication feature. As a general guideline, allocate 1GB of memory for every 1TB.
   • After migration, scale down the throughput to match your long-term usage and optimize costs.

Solution overview

Figure 1: Windows file system migration solution architecture

1. Enable CredSSP authentication required by the script to edit DNS CNAME and Service Principal Name (SPN) records on your behalf.
2. Install-WindowsFeature RSAT-AD-PowerShell, RSAT-DNS-Server modules on source Windows file server in order to query SPN and computer objects and create CNAME records in DNS.
3. Check permissions using PowerShell to query existing file shares and folder permissions.
4. Write file share permissions and configuration to an XML file on the source Windows file server.
5. Check if any local admin groups have permissions set on the folders, if yes display a message asking user to modify permissions to a domain group as local groups will not have access on Amazon FSx for Windows File Server.
6. (Optional) Migrate the data using Robocopy.
7. Recreate the file shares on destination Amazon FSx for Windows File Server using New-FSxSmbShare command.
8. (Optional) Find and delete any existing DNS alias SPNs on the source Windows file server and relink them to the Amazon FSx for Windows File Server Active Directory computer object.
9. (Optional) Create or Recreate the CNAME record of the DNS Alias.
10. Disable CredSSP authentication.

Prerequisites

Before deploying the solution you will need the following resources:

1. If the source Windows file server is located on-premises and you’re using Robocopy to transfer the data, you’ll need a VPN connection to an AWS VPC or an AWS Direct Connect link that provides connectivity between the on-premises Windows file server and the AWS VPC where your Amazon FSx for Windows file system (the destination) is located. For more details on the networking requirements, please refer to [1]Network configurations. Alternatively, the AWS DataSync agent can transfer the data to Amazon FSx for Windows File Server without the need for a VPN, by utilizing service endpoints.
2. If migrating away from an Amazon Elastic Compute Cloud (EC2) Windows file server, ensure the EC2 instance has connectivity to both Amazon FSx for Windows File Server and your domain controller. Refer to [1] for details.
3. An Amazon FSx file system (the destination) configured with solid state drive (SSD) storage. (Note: After the migration is complete, you can move the data to Amazon FSx file systems using hard disk drive (HDD) storage if that is suitable for your application. Refer to best practices for migrating existing file storage to Amazon FSx for Windows File Server).

4. AWS DataSync and Robocopy are both equipped to preserve network access control lists (ACLs, also known as Windows ACLs). Before you begin migration, we recommend that you take a backup of the source Windows server and data before starting the migration.

5. The automation solution has been tested on file servers running Windows Server 2016, 2019 and 2022. (Server 2003 and 2008 are not supported and you would need to make the necessary code changes to the scripts provided).

6. Evaluate if a temporary increase of the volume size for the destination FSx for Windows File Server deployment is required for the migration, to match the data size being migrated. This applies mainly to data deduplication on FSx. (1GB RAM for every 1TB of data) for example, 1024MB file system has 72GB RAM. Read this article for more information on performance.

Deploy the solution in PowerShell

I will take you through the following 8 steps to deploy the solution:

Step 1: Setup & Parameters
Step 2: Check permissions
Step 3: Copy your data from the Windows File Server to Amazon FSx for Windows File Server
Step 4: Recreate Shares
Step 5: (Optional) Remove and Add SPN
Step 6: (Optional) DNS CNAME Record
Step 7: Disable CredSSP
Step 8: Test DNS and CNAME

Step 1: Setup & Parameters

By leveraging the Remote Desktop Protocol (RDP), you can remotely log into the operating system and access a full desktop user interface, this can make copy and paste script files from local machine to the Windows file server easier. If you do not want to use RDP to access the Windows file server, you can remotely connect using PowerShell remoting or if using an EC2 instance, you can use AWS Systems Manager Session Manager.

1.1. Download the PowerShell script files using this link, to the source Windows File Server.

1.2. Extract zip file to a folder, in this example we use C:\Migration.

1.3. Open PowerShell as Administrator, and map the FSx file system to a drive letter: net use Z: \\FSxDNSName\D$.

1.4. Dot Source the MigrationParameters.ps1 so that we have the variables loaded into memory. Run the following commands as shown in Figure 2 cd C:\Migration . .\MigrationParameters.ps1  (note the double dot).

Figure 2: MigrationParameters.ps1 prompting for FSx for Windows File Server administrator credentials

• The MigrationParameters.ps1 file will prompt for inputs that match your FSx for Windows File Server environment as shown in Figure 3 (Optional) If your source Windows file server is an EC2 instance, the default Amazon Machine Image will have the AWS Tools for PowerShell pre-installed.

Figure 3: MigrationParameters.ps1 prompting for user details

As in Figure 4, you can create or attach an EC2 instance profile IAM role that allows fsx:DescribeFileSystems. These permissions allow the script to retrieve information about the existing FSx file systems, such as the file system ID, DNS name, and Remote PowerShell endpoint.

Figure 4: Example of DNS Name and PowerShell endpoint in AWS FSx for Windows  File Server console

The MigrationParameters.ps1 script will prompt the user to provide the necessary inputs. Table 1 offers additional details on the function of each variable. This information may be useful if you need to customize or enhance the script.

Table 1: Variable names and definitions

Now that we have the parameters loaded into memory, we can run 0-Enable-CredSSP.ps1 (Figure 5) on your Windows file server to enable CredSSP authentication. This is required first before the script tries to edit DNS CNAME and Service Principal Name (SPN) records on your behalf. Please refer to this page for more details.

Figure 5: Enable CredSSP authentication output

Step 2: Check permissions

The next step is to validate the folder permissions using 1-CheckPermissions.ps1.

In Active Directory (AD) environments, the default domain policy automatically adds the “Domain Admins” group to the local administrators group of the source Windows file server. This can create the impression that Domain Admins have access to all folders, when in reality, they only gain access by being a member of the local administrators group.

It’s important to note that Amazon FSx for Windows File Server is a managed service, which means that Group Policy cannot be applied directly to the file system. This has a significant impact on the local administrators group permissions, as the Domain Admins group will not be automatically added to the local administrators group on the Amazon FSx file system.

The 1-CheckPermissions.ps1 script (Figure 6) addresses this by thoroughly checking all shared folders and reporting on any that have local Administrators group NTFS and SMB share permissions. This information is vital for ensuring a seamless migration and maintaining the appropriate access controls in your Amazon FSx for Windows File Server environment. Figure 7 illustrates the check permissions workflow.

Figure 6: 1-CheckPermissions.ps1 output showing no problems with file system permissions

Figure 7: Check permissions workflow

Figure 8: PowerShell output of 1-CheckPermissions.ps1 finding incorrect file system permissions

The 1-CheckPermissions.ps1 script can optionally prompt the user to fix any share and NTFS permissions issues (Figure 8:). If the user selects yes, the script will add the domain group specified by $DomainAdminGroup with full control to the share folders. For AWS Managed Active Directory, the group is “AWS Delegated Administrators“.

Step 3: Copy your data from the Windows File Server to Amazon FSx for Windows File Server

1. You could use AWS DataSync service or Robocopy to migrate the data, AWS DataSync service has a simple web interface that makes it easy to setup a number of controls like scheduled transfer and network throttling to help migrate your data without impacting your overall network traffic during production hours.
2. If you have chosen robocopy, execute the 2-RoboCopy.ps1.
   

Step 3.1 (Optional) Data Deduplication:

For migrations involving data deduplication and frequent source changes, block TCP port 445 on the source (except DataSync Agent) to prevent further updates. Then, transfer data in batches to FSx and deduplicate on the fly. This avoids the need to re-copy full source data after deduplication runs, maintaining storage optimization throughout the migration.

When evaluating the raw data size as in Figure 9 pay attention to the “Size” metric, not the “Size On Disk.” The “Size” value will give you a more accurate representation of the space that will be consumed when copied to FSx for Windows File Server.

Figure 9: Properties of folder size on disk showing raw vs dedup data size

Figure 10 represents the migration workflow when dealing with a deduped source file system:

           Figure 10: Deduplication workflow copying raw data & reducing size post process on FSx for Windows File Server

1. Run Optimization Schedule on FSx for Windows File Server, the following snippet is an example of how to do this:

# Edit variables below:
$DestRPSEndpoint = "amznfsxxxxxyyyy.mytestdomain.local"

# This Start time is set for 10 seconds from now (ensure your files server time zone is in UTC) 
$StartTime = (Get-Date).AddSeconds(20)

# DurationHours of 8 hour causing the server to cancel the job after 8 hours if the process has not ended.
$DurationHours = 8 

Invoke-Command -Authentication Kerberos -ComputerName ${DestRPSEndpoint} -ConfigurationName FSxRemoteAdmin -ScriptBlock `
{
New-FSxDedupSchedule -Name CustomOptimization -Type Optimization -Start $Using:StartTime -Days Mon, Tues, Wed, Thurs, Fri, Sat, Sun -Cores 70 -DurationHours $Using:DurationHours -Memory 70 -Priority High 
} 

# Get status
Invoke-Command -Authentication Kerberos -ComputerName ${DestRPSEndpoint} -ConfigurationName FSxRemoteAdmin -ScriptBlock {Get-FSxDedupSchedule -Name CustomOptimization -Type Optimization} 

2. After Optimization completes, run Garbage Collection to reclaim space:

$StartTime = (Get-Date).AddSeconds(20)
 Invoke-Command -Authentication Kerberos -ComputerName ${DestRPSEndpoint} -ConfigurationName FSxRemoteAdmin -ScriptBlock {
 New-FSxDedupSchedule -Name "CustomGarbage" -Type GarbageCollection -Start $Using:StartTime -Days Mon, Tues, Wed, Thurs, Fri, Sat, Sun -DurationHours $Using:DurationHours
} 

# Get status
 Invoke-Command -ComputerName ${DestRPSEndpoint} -ConfigurationName FSxRemoteAdmin -ScriptBlock {Get-FSxDedupSchedule -Name CustomGarbage -Type GarbageCollection} 

Repeat the above deduplication process for each share folder as per Figure 7: workflow. Advanced users can build their own automation of this process using the helper functions found here.

Step 4: Recreate Shares

1. Run 3-RecreateShares.ps1 (Figure 11) which will get the list of shares from the XML export file located at the location defined by the $LogLocation variable.
2. Creates file shares using PowerShell Remoting using the Amazon FSx for Windows File Server PowerShell endpoint defined by $FSxDestRPSEndpoint variable.

Figure 11: 3-RecreateShares.ps1 output showing file shares successfully created on FSx for Windows File Server

Step 5: (Optional) Remove and Add SPN

1. Run 4-Remove-Add-SPN.ps1 if using an alias as illustrated in Figure 12 This will remove SPN from source server computer object.
2. Add SPN to Amazon FSx for Windows File Server computer object.

Figure 12: 4-Remove-Add-SPN.ps1 output showing successful SPN record creation

Step 6: (Optional) DNS CNAME Record

If using alias or CNAME record on source system, update the DNS alias to point to the new FSx file system. Optionally, you can push a Group Policy or script to your clients to delete any outdated drive mappings and redirect them to the new FSx DNS name.

From an architectural standpoint, it’s best practice to use a DNS alias for your file server. This approach allows you to seamlessly migrate to a new region, file system, or technology in the future, minimizing disruption to your end-users.

1. It is required to create any aliases within the FSx For Windows console before running the 5-Alias-CNAME.ps1
2. Run Alias-CNAME.ps1 as shown in Figure 13 which checks if $Alias variable has a value, if yes it will then check if a DNS CNAME record already exists for that alias and remove it and recreate it but point the CNAME to the Amazon FSx for Windows File Server computer object.

Figure 13: 5-Alias-CNAME.ps1 output showing successful CNAME record creation

Step 7: Disable CredSSP

Lastly run 6-Disable-CredSSP.ps1 to undo the changes made by 0-Enable-CredSSP.ps1 as we no longer need CredSSP authentication enabled.

Step 8: Test DNS and CNAME

1. Flush the DNS cache from the current session if you have not done so already as illustrated in Figure 14  This will ensure that a new DNS request is performed against the DNS servers to retrieve the updated CNAME record for the alias. This can be accomplished by running the following command from your existing session: ipconfig /flushdns

Figure 14: Output showing successful ipconfig /flushdns command

2. Connect to Amazon FSx for Windows File Server file system using both the alias and DNS name to test that SPN and CNAME created successfully and that folders appear and can be accessed as in Figure 15.

Figure 15: DNS console showing CNAME record & file Explorer window of migrated share folders

Conclusion

In this post, I shared strategies and best practices to help you navigate migrating an existing Windows Server file share to FSx for Windows successfully. We also explored how to leverage automation tools like PowerShell to streamline the migration, reduce risks, and ensure a smooth transition to the highly available and scalable Amazon FSx for Windows File Server platform.

Whether your Windows file servers are currently running on Amazon EC2 or in your on-premises data center, the strategies and tools presented in this blog post will empower you to execute a well-planned and efficiently automated migration, minimizing downtime and disruption to your end-users.