AWS Storage Blog
Improve application resiliency and performance with Multi-AZ Amazon FSx for OpenZFS
Business-critical applications frequently require high availability to meet Service Level Agreements (SLAs). In addition, users are challenged by compliance requirements and disaster recovery plans to assure workload availability. In many cases, workloads needing high availability also need highly performant, low-latency file storage. Customers who are considering the migration of business-critical application to AWS want to know what safeguards and tools are available to meet these stringent business continuity requirements.
Amazon FSx for OpenZFS (FSx for OpenZFS) is a fully managed file service offering high-performance, low-latency NFS v3 and v4 access, as well as advanced storage management features such as near-instant volume snapshots and thin-provisioned volume clones. Previously, FSx for OpenZFS only offered single Availability Zone (Single-AZ) deployment options: a single file server is used for these deployments. Single-AZ file systems need the user to configure a weekly maintenance window, during which the file system may be unavailable for a few minutes for necessary patching or similar maintenance activities. For many user workloads, this is an acceptable tradeoff for high-performance storage, but other users requested higher availability and durability options for their enterprise applications. An example of this is the use of MariaDB databases on self-managed Amazon EC2 instances. Consequently, in August of 2023, Amazon released the Multi-Availability Zone (AZ) deployment option for FSx for OpenZFS, making it easy to deploy file storage that spans multiple AZs within an AWS Region to support workloads that need increased availability and enhanced durability.
In this post, we demonstrate how users can increase their durability and availability using Amazon FSx for OpenZFS, all without any application modification. We examine the architecture of multi-AZ Amazon FSx for OpenZFS, how to test failover operations, and the results of application-level testing during a failover/failback event. The benefits include providing high availability, and simplified operation recovery for business critical applications including databases.
Architectural overview – FSx for Open ZFS Multi-AZ configuration
Figure 1 shows details on how FSx for OpenZFS is architected to provide improved data durability, fault tolerance, and availability. A Multi-AZ file system is transparently composed of two sets of file servers and two sets of storage disks behind the scenes. Any data written to the filesystem is done synchronously, so that the data is always fully replicated across both AZs.
Figure 1: FSx for OpenZFS architecture
With traditional high availability storage, multiple storage nodes share a floating virtual IP address with independent management IPs on the same subnet. When the active node fails, the fault is detected and the IP address is assumed by the standby node, moving with the storage. Because FSx for OpenZFS can use AWS infrastructure and multiple AZs with their own independent networking subnets, the high-availability mechanism for client access is somewhat different in implementation than with traditional highly available network storage.
Prerequisites for implementing and testing failover/failback
To implement multi-AZ FSx for OpenZFS, you should meet the following prerequisites:
- An AWS Account.
- Two subnets in different Availability Zones (AZ) choosing within a Amazon Virtual Private Cloud (VPC).
- An Amazon FSx for OpenZFS Multi-AZ deployment in the above subnets.
- Running EC2 instances in the subnet you choose to deploy the solution.
- Have a general knowledge of using Linux.
With FSx for OpenZFS, the active and standby file servers are configured in two independent AZs of your choosing within a Virtual Private Cloud network (VPC). When selecting the Standard create option as shown in Figure 2, availability of the file system is maintained to the clients through an Endpoint IP, which you are prompted to configure at the time of filesystem creation. With the Quick create file system option, the default of using an unallocated IP range from within your selected VPC is automatically selected.
Note that when using Quick create (the default), this Endpoint IP is an IP address outside of your AZ subnets, but within the VPC CIDR. When using the Standard create option, alternatively you can use an IP range outside of your VPC, or specify an IP address range yourself. With these options, the Endpoint IP acts in a similar fashion to the use of a floating IP as used in traditional high availability file services. In the case of FSx for OpenZFS, the underlying mechanism is different, allowing for file servers to be in different subnets – a necessary component of a Multi-AZ service. When failover occurs, the route to the active file server is automatically updated to point to the other file server, assuring transparent continuity of service for clients.
These are all implementation details that don’t impact the ease or function of service operation. However, due to the conceptual differences from traditional high availability storage, we thought it was worth mentioning to help clarify configuration choices.
Figure 2: FSx for OpenZFS Multi-AZ Endpoint IP selection from “Standard create”
Testing failover/failback
As a fully managed service, FSx for OpenZFS performs system updates and maintenance on your file system during a user-defined weekly maintenance window. This maintenance window allows the service to be updated on an as-needed basis with minimal disruption. Patching occurs infrequently, typically once every several weeks.
When maintenance is initiated, the filesystem is fully exported from the active file server and becomes momentarily unavailable for new access requests. Then, the filesystem is imported on the opposite file server – that is, the previously standby file server –the network route is updated, and IO can resume. Then, the managed instance used by the service for the active file server is updated. Once maintenance is complete, an identical failover process occurs in the other direction, back to the original active file server. This process is fully transparent to clients accessing the file system.
The failover mechanism used for maintenance is the same behavior occurring with the loss of the primary availability zone: the filesystem is immediately imported to the secondary AZ and made available on loss of the active file server.
To test a failover/failback event with FSx for OpenZFS, change the throughput capacity of the file system as shown in Figure 3. When throughput for the file system is changed, the underlying FSx for OpenZFS server is transparently swapped out behind the scenes to match the requested throughput.
Figure 3: Changing throughput capacity
The process that updates the file system throughput takes only a couple minutes. You should see a Updating to XXX MB/s message in the AWS Management Console as shown in Figure 4.
Figure 4: Changing throughput capacity
As shown in Figure 5, the failover event can be monitored using Amazon CloudTrail by looking for the events ReplaceRoute, as seen in the following figure. In this example, we changed the throughput from 160MB/s to 360MB/s. There is a ReplaceRoute API call for the failover at 12:52PM, followed by the failback at 12:55PM. Note that the User name is FSx, indicating that the FSx service initiated the API call.
Figure 5: CloudTrail Event ReplaceRoute
Application-level failover/failback impact verification
To test the impact of failover/failback with FSx for OpenZFS, we are deploying a MariaDB database server on an Amazon Linux 2 EC2 instance. Then, we use mysqlslap for load-testing.
The mysqlslap tool is a diagnostic program designed to emulate SQL client load for a server and to report the timing of each stage. This lets us measure the impact of multi-AZ failover/failback on a client application workload.
EC2 instance configuration and testing
The following steps are taken on the EC2 instance to prepare it for testing.
- On the Linux client, install the MariaDB server, enable the service, and start the database
$ sudo yum install -y mariadb-server
$ sudo systemctl enable mariadb
$ sudo systemctl start mariadb
- Make sure to set the root password for the database, as security is our top priority at AWS. We will need this password in Step 4.
$ sudo mysql_secure_installation- Move the database to the FSx for OpenZFS volume.
$ sudo systemctl stop mariadb
$ sudo mv /var/lib/mysql /mnt/fsx/data/mysql
Update the config files:
<p>/etc/my.cnf & /etc/my.cnf.d/client.cnf</p><p>$ sudo systemctl start mariadb</p>Verify that the new location is on the FSx NFS mount.
MariaDB [(none)]> select @@datadir;
+----------------------+
| @@datadir |
+----------------------+
| /mnt/fsx/data/mysql/ |
+----------------------+ 1 row in set (0.00 sec)- Start the mysqlslap load using the auto-generate-sql feature. Replace the word password with the root password you set earlier.
sudo mysqlslap –user=root -p password –host=localhost –concurrency=200 –iterations=2000 –number-int-cols=5 –number-char-cols=20 –auto-generate-sql –verbose ;
- Monitor performance using Amazon CloudWatch, which is integrated with FSx for OpenZFS in the Console.
- Change the Throughput capacity on the filesystem to trigger a failover event.
- Monitor CloudWatch and MariaDB to see the impact of the failover and failback.
Result analysis
In the introduction to this blog, we promised that we would cover how the use of FSx for OpenZFS could improve existing application durability and availability with the use of Multi-AZ deployments. Now we will examine the results of the above testing and the impact had upon the application of a failover event.
During the mysqlslap test, the workload stayed steady with an average throughput of 42MB/s, 639 ops/sec, and with an average RTT of 5.06 ms. This demonstrates that the workload was able to perform normally throughout the failover event without interruption, maintaining a consistently low latency to storage.
The throughput change resulted in ReplaceRoute API calls at 16:06:16 and 16:09:52 shown in Figure 6:
Figure 6: ReplaceRoute API calls
Graphing the data (shown in Figure 7) from CloudWatch, we see that the throughput and ops/sec were steady state except for the failover/failback events.
Figure 7: Throughput (MB/s) and Operations/sec (CloudWatch)
Running nfsiostat during the test we captured RTT latency, which was steady at 5ms except for brief spikes when the failover/failback occurs, as shown in Figure 8. While there was an increase in latency during the failover/failback operations, it should be noted that this demonstrates that the EC2 instance operating system was able to continue operationally through the failover with minimal impact.
Figure 8: NFS round trip time (RTT) in milliseconds from seconds since start of nfsiostat collection
Cleaning up
To avoid incurring future charges, delete the resources created by this solution.
- Terminate any EC2 Instances created for this work.
- Delete the Amazon FSx for OpenZFS Multi-AZ deployment.
- Delete the Amazon VPC and associated components.
Conclusion
In this post, we demonstrated how users can increase their durability and availability using Amazon FSx for OpenZFS, all without any application modification. We examined the architecture of multi-AZ Amazon FSx for OpenZFS, how to test failover operations, and the results of application-level testing during a failover/failback event.
We recommend Multi-AZ file systems for most production workloads given the high availability and durability model it provides. You should use a Multi-AZ file system for workloads that need high availability to make sure of availability across events such as hardware failures, file system maintenance, and AZ disruption. This additional capability adds resiliency for users to maintain tight SLAs with their internal application consumers. Amazon FSx for OpenZFS is available in most AWS Regions and now offers high availability between multiple AZs, making sure that during a failover/failback event, applications continue to meet business objectives without disruption. Learn more in the Amazon FSx for OpenZFS user guide, and get started building now.