AWS Storage Gateway provides simplified monitoring for File Gateway

Proactively monitoring your AWS Storage Gateway can keep you notified about performance issues and resource constraints if your workloads change over time. Monitoring can be used to indicate if you have network constraints, if the allocated cache storage is not sufficient, or if your root disk is not optimally handling increased workloads.

In this post, we discuss how to leverage the consolidated monitoring available in AWS Storage Gateway by providing examples for File Gateway. First, we cover the new metrics section of the Storage Gateway console. Then we set up Amazon CloudWatch Alarms to trigger an alert based on a condition we define. Finally, we set up alerts using the native integration of Storage Gateway and Amazon CloudWatch Logs.

AWS Storage Gateway overview

AWS Storage Gateway is a hybrid cloud storage service that provides on-premises applications access to virtually unlimited cloud storage. Storage Gateway uses Network File System (NFS), Server Message Block (SMB), iSCSI, and iSCSI-VTL interfaces through file, tape, and volume gateways. You can use the service for backing up and archiving data to AWS, using on-premises file shares backed by cloud storage, and providing low-latency access to in-cloud data for on-premises applications.

You can deploy Storage Gateway on a virtual appliance (VMware ESXi, Microsoft Hyper-V, Linux KVM) on premises, as a hardware appliance on premises, or on an Amazon EC2 instance.

File Gateway provides a file interface into Amazon S3. Customers can store and retrieve objects in Amazon S3 using industry-standard file protocols such as NFS and SMB. Customers use File Gateway for backing up and archiving data to AWS, providing cloud-backed storage to on-premises file shares, and providing on-premises applications low-latency access to data in cloud.

Monitoring File Gateway

While you typically deploy File Gateway at your premises, the gateway is managed, monitored, and updated from the AWS Management Console. Storage Gateway is integrated with Amazon CloudWatch for logging, metrics, and alarms, enabling you to view gateway-level data in the Storage Gateway management console.

File Gateway publishes several health and performance logs and metrics to Amazon CloudWatch, providing you with continuous visibility into the operations of your gateway. Using these metrics, you can configure CloudWatch alarms to notify you of changes in your workloads so you can quickly take action. For example, you could receive notifications when CPU usage exceeds a threshold you define as optimal for your workloads.

With this visibility into the performance of your gateway, you can confirm you are running optimally and modify your environment if needed.

Consolidated view for monitoring

AWS Storage Gateway provides a consolidated view of CloudWatch Metrics and any CloudWatch Alarms you have configured on your gateway without leaving the Storage Gateway management console. This lets you manage and monitor your gateway in a single place. It also enables you to quickly identify and act on any triggered alarms, all without having to configure a custom dashboard or leave the Storage Gateway management console.

When you view the gateway’s monitoring tab in the Storage Gateway management console, you see the CloudWatch Metrics for the gateway. Any CloudWatch Alarms that have been configured for the gateway are also visible.

CloudWatch Alarms

With a single consolidated view in the Storage Gateway management console, you can monitor CloudWatch Alarms. This makes gateway metrics such as cache status, disk performance, and health notifications for VMware vSphere high availability easily viewable alongside your metrics dashboard. Specifically, alarms for high cache percent dirty, high I/O wait, and health notification failures can be valuable for identifying and responding to changes in gateway performance.

Clicking on the name of an alarm directs you to the Amazon CloudWatch console. The console provides alarm details, including when the alarm was triggered along with a graph displaying the relevant metrics. You can also choose to receive alarm notifications by subscribing an alarm to Amazon SNS from the CloudWatch console.

In the following example, we set a metric to alarm when there are more than five simultaneous NFS connections called “NfsSessions-more-than-5.” The alarm state is OK when the threshold stays under five connections. The alarm becomes active when the threshold is exceeded. The graph is useful to visualize and identify patterns such as periods of time where thresholds are exceeded.

Viewing CloudWatch Alarms on mobile devices

Using the AWS Management Console mobile application, you can track alerts on-the-go while you are away from your keyboard. The following screenshot provides an example of viewing CloudWatch Alarms on the mobile application:

CloudWatch Logs

You can configure CloudWatch Logs to log errors, which is useful for identifying errors such as insufficient bucket access permissions. Logs can also indicate when applications attempt to use gateways to access data that has transitioned to Amazon S3 Glacier or S3 Glacier Deep Archive and is not immediately accessible.

In addition to using File Gateway for active workloads, customers can optionally use S3 Lifecycle policies to automatically archive data to lower-cost storage classes for data archiving and long-term backup. Please refer to our blog post for backing up on-premises applications using Storage Gateway for more information.

Logging for your File Gateway can be enabled at creation, or you can enable it after creation in the console under Edit gateway information.

In our case, we select Create a new log group for logging:

The following is an example of the log entry that is created when you try to access an object stored in the Amazon S3 Glacier storage class:

{
"severity": "ERROR",
"bucket": "mybucket",
"roleArn": "arn:aws:iam::12345678:role/service-role/StorageGatewayBucketAccessRole5b0be8fc-1196-4c5e-8835-1492d43822",
"source": "share-0C803173",
"type": "InaccessibleStorageClass",
"operation": "s3:GetObject",
"key": "archive/photo1.jpg",
"gateway": "sgw-02CE296B",
"timestamp": "1592284178329"
}

These error logs can be valuable in diagnosing file access issues. They can also be used to invoke AWS Lambda functions to automate workflows such as automating the restoration of Amazon S3 Glacier objects when accessed.

Cleaning up

To avoid incurring any unwanted future costs, remember to delete any example resources you set up if you just wanted to take Amazon CloudWatch for a run. You are charged standard rates for Amazon CloudWatch Logs, Amazon CloudWatch Events, and Amazon CloudWatch Metrics if you configure CloudWatch for health notifications or audit logs. Please refer to Amazon CloudWatch pricing.

Conclusion

In this post we showed how to use the native monitoring and alerting capabilities of AWS Storage Gateway. Configuring alerts helps customers confirm their gateways are operating optimally.

AWS Storage Gateway is easy to manage and monitor without having to leave the Storage Gateway management console. The service is integrated with Amazon CloudWatch for logging, metrics, and alarms, which provides customers the ability to view gateway health and performance metrics from a single console. Monitoring your Storage Gateway helps to ensure that your data is being transferred to the AWS Cloud with minimal delay. As workloads change, it is important to make sure that the gateway has the necessary resources to manage the workload.

AWS is a customer obsessed organization. We deliver features and functionality based on customer feedback. Please continue to submit feedback so that we can continue to deliver the products and enhancements that are valuable to you. Thanks for reading this blog post, if you have any comments or questions, don’t hesitate to leave them in the comments section.