AWS Storage Blog

Performance analysis for different Amazon EFS throughput modes via Amazon CloudWatch

When I talk with customers about their file storage, I frequently get asked “How can I determine the right throughput capacity for my file storage?” The simple answer is through monitoring the performance of your workload to determine the right performance configuration for your file storage.

Throughput modes for Amazon Elastic File System (EFS) determine the amount of data and metadata transferred in a second (throughput) via requests to your elastic storage. Determining the right throughput usage for your workload requires the ability to monitor throughput usage of your workload.

In this blog post, I demonstrate how to set up an Amazon CloudWatch dashboard to monitor Amazon EFS throughput. I also demonstrate how to use a CloudWatch agent on Amazon EC2 instances to monitor client-side throughput of your workload. I run benchmarking tests to highlight performance differences between throughput modes of Amazon EFS using a Cloudwatch dashboard. This will help you determine the right performance configuration for your workload without having to specify or provision throughput capacity.

Solution overview

I have created Amazon EFS in a VPC in us-east-1 region. Using mount helper, I am mounting the file system on five EC2 instances running within the same Amazon Virtual private Cloud (VPC).

Amazon EFS offers three throughput modes: Bursting, Elastic, and Provisioned. In Bursting Throughput mode, throughput scales with the amount of storage in Amazon EFS Standard or Amazon EFS One Zone storage class. If your application is throughput constrained on Bursting mode, then you should consider using Provisioned or Elastic Throughput modes. You should use Provisioned Throughput mode if you know your workload’s performance requirements. If you are unsure of your application’s peak throughput needs or if your application is very spiky, with low baseline activity (such that it uses less than 5% of capacity on average when you provision for peak needs) then you should use Elastic Throughput mode. Monitoring throughput usage of your workload is important to decide what is the right throughput mode.

I am running performance benchmarking tests against different throughput modes of EFS. I am also creating a CloudWatch dashboard to monitor the test results.

Architecture diagram for this blog with an EFS mounted on five EC2 Instances

Solution walkthrough

  1. Create Amazon EFS file system and mount it on five t3a.small EC2 instances.
  2. Install fio benchmarking tool on all EC2 instances.
  3. Create a script to send EC2 instances throughput usage data to CloudWatch.
  4. Set up a CloudWatch Dashboard to monitor Amazon EFS throughput.
  5. Monitor Aggregated Clients throughput for Amazon EFS.
  6. Run benchmarking test against different throughput modes for Amazon EFS.

Prerequisites

For this walkthrough, you should have the following:

  • An AWS account
  • Five t3a.small EC2 instances running Amazon Linux 2 in the same VPC and availability zone as your Amazon EFS file system
  • IAM Role attached to EC2 instances to grant permissions that CloudWatch agent needs to write metrics to CloudWatch

Create Amazon EFS file system and mount it on EC2 Instances

To create your file system:

  1. Open the Amazon EFS Console.
  2. Choose Create file system to open the Create file system dialog box.
  3. (Optional) Enter a Name for your file system.
  4. For Storage Class choose One zone as it is cheaper than Standard storage class. We are going to mount this Amazon EFS file system on EC2 instances within the same Availability Zone (AZ).
  5. Choose Create to create a file system.

To mount your file system using the mount helper on EC2 Linux instances:

  1. Add a rule to the mount target security group to allow inbound access from the EC2 security group on TCP port 2049. The security group in the Source column is the security group associated with the EC2 instance. Check Security Groups for Amazon EFS for more details.
  2. Open a terminal window on your EC2 instance through Secure Shell (SSH), and log in with user ec2-user. For more information, see Connect to your Linux instance using SSH for Linux instances.
  3. Create a directory /efs that you will use as the file system mount point using the following command:
sudo mkdir /efs
  1. Install amazon-efs-utils package on the EC2 instance.
sudo yum -y install amazon-efs-utils
  1. Run the following commands to mount your file system.
sudo mount -t efs <file-system-id> /efs

Repeat the preceding steps for all five EC2 instances.

Install fio on EC2 instances

To install fio on Amazon Linux, run the following command on all five EC2 instances:

sudo yum install -y fio

Create a script to send EC2 throughput usage data to CloudWatch

We will create a bash script to gather read and write throughput metrics from nfsiostat tool. The script is executed every minute via cronjob to send throughput metrics to CloudWatch.

  1. Create a file to store bash script and make it executable.
sudo touch /usr/local/bin/efs_cloudwatch_metrcis.sh
sudo chmod 750 /usr/local/bin/efs_cloudwatch_metrcis.sh
  1. Paste the following script in file sh file using your own preferred text editor in Linux.
#!/bin/bash
nfs_value=$(/sbin/nfsiostat 1 2|tail -4)

#Calculating Read and Write Throughput in MiB
read_throughput_kB=$(echo $nfs_value|cut -d ' ' -f13)
write_throughput_kB=$(echo $nfs_value|cut -d ' ' -f31)
read_throughput_mB=$(echo "scale=2; $read_throughput_kB / 1024"|bc -l)
write_throughput_mB=$(echo "scale=2; $write_throughput_kB / 1024"|bc -l)
EC2_INSTANCE_ID="`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`"

#Sending Throughput metrics to Cloudwatch
/bin/aws cloudwatch put-metric-data --metric-name ec2-efs-read-throughput --dimensions InstanceID=$EC2_INSTANCE_ID --namespace "CustomEC2" --value $read_throughput_mB --region us-east-1

/bin/aws cloudwatch put-metric-data --metric-name ec2-efs-write-throughput –dimensions InstanceID=$EC2_INSTANCE_ID --namespace "CustomEC2" --value $write_throughput_mB --region us-east-1
  1. To run this script every minute via crontab, run the following commands:
sudo crontab -l > cron_bkp
sudo echo "*/1 * * * * /usr/local/bin/efs_cloudwatch_metrics.sh >/dev/null 2>&1" >> cron_bkp
sudo crontab cron_bkp
sudo rm cron_bkp

Set up a CloudWatch Dashboard to monitor Amazon EFS throughput

MeteredIOBytes is a CloudWatch metric for Amazon EFS that provides actual number of metered bytes for each file system operation, including data read, data write, and metadata operations. I am using metric math feature of CloudWatch to calculate throughput value for my file system. To create a dashboard from the console:

  1. Open the CloudWatch console.
  2. In the navigation pane, choose Dashboards, and then choose Create dashboard.
  3. In the Create new dashboard dialog box, enter a name for the dashboard EFS-Dashboard, and then choose Create dashboard.
  4. Open the CloudWatch console.
  5. In the navigation pane, choose Metrics, then choose Amazon EFS and then choose File System Metrics.
  6. Select MeteredIOBytes of your Amazon EFS file system and add the following math expression:

Throughput(MiB/s) = [TotalIOBytes[Sum]/(period*1024*1024)

Throughput usage monitoring graph of in CloudWatch using Metric math feature

Select Action, choose Add to Dashboard and then Select EFS-Dashboard.

Monitor Aggregated Clients throughput for Amazon EFS

  1. Open the CloudWatch console.
  2. In the navigation pane, choose Metrics → All Metrics → CustomEC2 → InstanceID.
  3. Select read throughput metrics for all five EC2 instances. Choose Add Math → Empty Expression and use the following expression to plot Aggregated Read Throughput of all EC2 Instances:

SUM([m1+m2+m3+m4+m5])

where m1, m2, m3, m4 and m5 are the Read throughput metrics of five EC2 instances used for this walkthrough.

  1. Select write throughput metrics for all five EC2 instances. Choose Add Math → Empty Expression and use the following expression to plot Aggregated Write Throughput of all EC2 Instances:

SUM([m6,m7,m8,m9,m10])

where m6, m7, m8, m9 and m10 are the Write throughput metrics of five EC2 instances used for this walkthrough.

  1. Select both of these new metrics, Action → Add to Dashboard → Select EFS-Dashboard.

Run benchmarking test against different throughput modes for Amazon EFS

Open a terminal window on your EC2 instance through Secure Shell (SSH), and log in with user ec2-user. Run the following commands on all five EC2 instances.

EC2_INSTANCE_ID="`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`"
sudo mkdir /efs/$EC2_INSTANCE_ID/$(date +"%d")

sudo fio --directory=/efs/$EC2_INSTANCE_ID/$(date +"%d") --ioengine=libaio --name fio_test_file --direct=1 --rw=read --randrepeat=0 --bs=32k --size=4G --numjobs=16 --time_based --runtime=1800 --group_reporting --norandommap &

The file system used in this walkthrough was empty and had a full burst bucket balance, so the throughput usage can go up to 100 MBps. This can be observed in both Amazon EFS Average Throughput and Aggregated Clients throughput metrics in the dashboard.

Cloudwatch Dashboard for performance benchmarking results of Burst throughput mode

Now go to the Amazon EFS console and select the file system you created in the first step. Choose Edit and change the throughput mode from Burst to Provisioned and specify Provisioned Throughput (MiB/s) as 300. Look at the EFS-Dashboard after few minutes and you can notice a spike in throughput usage from 100 MiB/s to 300 MiB/s because of this change.

Cloudwatch Dashboard for performance benchmarking results of provisioned throughput mode

You need to wait at least 24 hours before you can change the throughput mode again. To avoid this 24 hour wait time, create a new One Storage class Amazon EFS file system with 300 MiB/s provisioned throughput. Use the same security group for your new Amazon EFS file system. Change EFS Average Throughput metric in the EFS-Dashboard to include the new filesystem ID. You should also unmount old EFS and mount new EFS over /efs folder on all five EC2 instances as shown here:

  1. Unmount old EFS on EC2 instance:
sudo umount /efs
  1. Run the following commands to mount your new file system.
sudo mount -t efs <new-file-system-id> /efs

Now run the benchmarking test again on all the five EC2 instances. After few minutes of running the benchmarking, change the throughput mode from provisioned to elastic. You can notice in the CloudWatch dashboard that the available throughput has increased by changing the throughput mode to elastic.

Cloudwatch Dashboard for performance benchmarking results of Elastic Throughput

We can see that elastic throughput is designed to automatically deliver the throughput performance your application needs while you pay only for the amount of data read or written.

This concludes this walkthrough as you were able to monitor the change in available throughput for different throughput modes for Amazon EFS in CloudWatch.

Cleaning up

To avoid incurring unwanted future charges, terminate the EC2 instances and delete the Amazon EFS file system that you created for this walkthrough.

Conclusion

In this blog post, I showed how to monitor your Amazon EFS file system throughput using Amazon CloudWatch. I also demonstrated how to use CloudWatch agent on EC2 Instances to monitor client-side throughput for EFS. With the correct set up, you can determine the right throughput usage of your workload. I used this dashboard to analyze performance benchmarking results for different throughput modes of EFS. By comparing performance of different throughput modes, I showed that elastic throughput can further simplify running workloads and applications on AWS by providing file storage that doesn’t require any performance provisioning. Using Elastic throughput can also help you launch spiky and unpredictable workloads with performance requirements that are difficult to forecast.

Thanks for reading this blog post. For more tutorials on using Amazon EFS performance and modes, you can visit the Amazon EFS User Guide. If you have any comments, questions, or feedback, leave a comment in the comments section below.