AWS Storage Gateway adds File Gateway audit logs

UPDATE 9/8/2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

As customers expand their use of cloud services, they must often align their security and compliance processes with existing enterprise requirements. In a hybrid cloud storage environment that includes both on-premises storage and cloud storage, it can be challenging for customers to monitor user activity related to their data. This is particularly the case as the data is moving from their on-premises data center to the cloud. For many customers, there is a hybrid storage requirement to support applications that must remain on-premises, and File Gateway provides them with on-premises access to unlimited cloud storage.

File Gateway provides on-premises applications with file-based, cached access to virtually unlimited cloud storage using standard SMB and NFS storage protocols. File Gateway can be used with on-premises and Amazon EC2-based applications that need file protocol access to Amazon Simple Storage Service (Amazon S3) object storage. Today, we are announcing that AWS Storage Gateway now enables logging end-user operations on files and folders for SMB file shares when using File Gateway. In this blog post, I discuss how File Gateway audit logs is useful for monitoring and compliance, and how audit logs can also be helpful in gaining insights into user access. Furthermore, I discuss how File Gateway audit logs can be helpful for troubleshooting with the additional information the logs provide, and how to configure the audit logs.

Overview

Logging user access to files and folders within file shares is a key part of your organization’s internal security policy. Moreover, it is required to meet compliance standards that exist across Financial Services, Healthcare and Life Sciences, Government, and many other industries. As you begin migrating data to the cloud, you may be required to monitor user activity related to moving data and accessing files and folders from cloud storage. You must not only safeguard your data, but to also prove ownership, track who has accessed specific data, and provide reports to auditors.

File Gateway enables logging to provide you with details about user access to files and folders within a SMB file share. This further enables you to monitor user activities and act if inappropriate activities are identified. File Gateway audit logs provide benefits to you in three areas:

• Compliance and monitoring
• Insights into user access patterns
• Troubleshooting issues between SMB clients and the File Gateway

Compliance and monitoring

Many organizations are subject to government regulations and industry requirements they must follow in order to improve security, ensure consistency of controls, and mitigate risk. Consequently, organizations are required to log data and analyze it on a regular basis. In doing so, they defend against threats and enable compliance with PCI, ISO27001, Sarbanes-Oxley, General Data Protection Regulation (GDPR), and the Health Insurance Portability and Accountability Act (HIPAA).

With File Gateway audit logs, SMB user operations for files and folders are logged and then published to Amazon CloudWatch Logs. The operations logged for files and folders are create, delete, read, write, rename, and change of permissions. User information for each operation, including timestamp, Active Directory domain, user name, and client IP address, is also logged. Additionally, the audit logs can be exported to third-party security information and event management (SIEM) applications from Amazon CloudWatch or Amazon S3.

The following is an example of a “create file” log entry:

{
    "sourceAddress": "XXX.XXX.XXX.XXX",
    "accountDomain": "EC2",
    "accountName": "Admin",
    "groupId": "XXXXX",
    "source": "share-XXXXXXXX",
    "type": "FileSystemAudit",
    "ownerId": "XXXXX",
    "accessMode": "0777",
    "mtime": "1584903101747582202",
    "version": "1.0",
    "objectType": "File",
    "bucket": "s3-bucket-1",
    "objectName": "/topfolder/tests/iteration-1/integtest.01",
    "ctime": "1584903101747582202",
    "fileSizeInBytes": "0",
    "shareName": "my-fileshare-name",
    "operation": "Create",
    "gateway": "sgw-XXXXXXXX",
    "timestamp": "1584903101751",
    "status": "Success"
}

Gain insights on user access

To gain insight into File Gateway audit logs, you can use CloudWatch Logs Insights, Amazon Athena, Amazon Elasticsearch Service, or any SIEM application to review and analyze the log information. Using CloudWatch Logs Insights, you can interactively search and analyze your log data. In addition, it allows you to perform queries and visualize query results to identify patterns and usage trends. These queries can be optionally added to a CloudWatch Dashboard, which includes a rich set of sample queries to get you started.

For example, audit logs may be used to identify user and user group usage patterns. If you’re trying to optimize scheduling of batch jobs by identifying usage for a given user or group of users, you can plan and schedule batch jobs in accordance to the non-peak periods. This ensures that batch jobs are only scheduled to run in the time windows when the users would be least impacted.

In addition to batch job planning and optimization, IT administrators can use CloudWatch queries and visualizations to gain insights into the following:

• Identify the most active users for a SMB file share
• Identify workload usage patterns for an application user group
• Identify when a SMB file share is most active or when it’s being accessed
• Find out the types of data files being stored based on the object name
• Understand error trends based on file operation log entries
• Identify IP addresses and/or subnets accessing the SMB file shares to optimize network planning and layout

Troubleshooting

Chasing file access issues can be tedious and time-consuming. File Gateway audit logs provide another tool for you to use when troubleshooting issues in your environment. As an example, imagine a scenario where you cannot identify the IP address and the application that is accessing a file share. To solve this, you can now look at the IP address and user name in specific log entries. Failed file system requests are logged under the following conditions:

• Data cannot be written to the gateway if the cache disk is full
• The file or folder being accessed doesn’t exist anymore on the gateway

This information can be used for troubleshooting in those scenarios.

IT administrators can proactively set up CloudWatch alarms to generate notifications for specific filtered logged events. Administrators can define these events to provide visibility into excessive types of file access operations such as a threshold for too many file or folder deletions in a specified period. You can search and filter the audit log data by creating one or more metric filters. Metric filters define the terms and patterns to look for in the audit log data as it is sent to CloudWatch Logs. CloudWatch Logs use these metric filters to turn log data into numerical CloudWatch metrics that you can graph or set an alarm on. You can use any type of CloudWatch statistic, including percentile statistics, when viewing these metrics or setting alarms.

Configuring File Gateway audit logs

To get started, go to the AWS Management Console and select your Storage Gateway. You can enable File Gateway audit logs on an existing file share or when you create a new file share.

The following is a screenshot from the step where you enable File Gateway audit logs for an existing file share:

Once you have enabled audit logging, you see the following in the file share Details tab:

Step-by-step instructions can be found in the AWS Storage Gateway User Guide. File Gateway audit logs can be enabled in every Region where AWS Storage Gateway is available. You can view the File Gateway audit logs through the CloudWatch Management Console:

It should be noted that the audit log data must be secured from unauthorized access and tampering. Proper segregation of duties between those who administer system and network accounts and those who can access the audit logs must be established. Doing so prevents logs from being altered, accounts from being created or deleted, or any other malicious activities from happening.

If you configure File Gateway audit logs, you are also charged standard rates for Amazon CloudWatch Logs, Amazon CloudWatch Events, and Amazon CloudWatch Metrics. For more information, visit the Amazon CloudWatch Pricing Page.

Summary

In this blog post, I discussed how you can use audit logs to satisfy internal security policies and meet regulatory compliance requirements. I also discussed how audit logs can be useful when troubleshooting issues in your environment, given all the additional information that can be made available to you. Additionally, I covered how you can quickly configure File Gateway audit logs to provide audit trails and insights for file and folder activity in SMB file shares.

File Gateway audit logs is another example of how AWS Storage Gateway is supporting our customers’ transition to AWS storage services by adding enterprise security and compliance features. Audit logs can help customers strengthen their compliance and security infrastructure, giving them greater peace of mind and reducing the challenges associated with monitoring data (especially data in transit). This can save customers time and stress, as simplified monitoring and compliance enables customers to focus on other core areas of their business. To learn more and get started with File Gateway audit logs, check out the following links:

• Get started with AWS Storage Gateway
• Monitoring File Gateway
• Analyzing Log Data with CloudWatch Logs Insights

Thanks for reading about File Gateway audit logs. Please leave a comment in the comments section if you have any questions.