Monitoring your AWS SFTP environment

Organizations across the board use the Secure File Transfer Protocol (SFTP), also known as the Secure Shell (SSH) File Transfer Protocol, to share files for their business needs. SFTP is a well-established protocol that allows for easy and secure movement of data between existing systems that may otherwise have trouble finding a common interface mechanism. Certain organizations have operationalized their SFTP environment but lack easy visibility into their end users’ activities. As organizations onboard more users and applications, they want to ensure they have proper visibility and quick access to the operational metrics that are important to them.

Providing this capability with AWS Transfer for SFTP (AWS SFTP) is easy. AWS SFTP is a fully managed service that enables file transfers directly to and from your Amazon Simple Storage Service (Amazon S3) buckets and can be launched in minutes. Customers from many industries, including financial services, retail, healthcare, and more, have adopted the service as a highly scalable and available solution to their SFTP requirements.

In this post, I show you how to create a simple Amazon CloudWatch Dashboard to display information about your SFTP service. You can display information such as the number of users connecting, files uploaded and downloaded, amount of data transferred, and errors. I parse the logs sent to CloudWatch Logs and use CloudWatch Log Insights to build dynamic queries that I visualize in widgets. By using this dashboard, you can get deeper insights into how your users are interacting with the service and quickly identify trends or anomalies.

Step 1: Create AWS SFTP server

Before we begin, you need an AWS SFTP endpoint so that you can generate some traffic and create some logs. If you have an existing AWS SFTP server running that has been configured correctly with a CloudWatch logging role, you can skip this part.

For more detailed instruction on how to create an AWS SFTP server, check out Getting Started with AWS Transfer for SFTP.

When you get to the “Create an SFTP Server” step, configure the logging role to make sure that the service sends the logs to a CloudWatch Logs stream.

You can create your own policy from scratch or use the AWS managed policy “AWSTransferLoggingAccess,” which has the required permissions to write to CloudWatch Logs group.

Once your endpoint is running, upload and download some files so that you have some log data to visualize.

On to the dashboard!

Step 2: Create a dashboard

1. Head over to the CloudWatch Console.
2. Select Dashboards.
3. Select Create Dashboard.
4. Provide a name for your dashboard (for example, MySFTPDashboard) and select Create Dashboard.

Widget: Bytes in and out

Our first widget is going to display total bytes transferred into and out of the endpoint.

1. Choose Add Widget.

2. Choose Number and then Configure.
3. From the available metrics, select Transfer then By Server Metrics. Then select the BytesIn and BytesOut metrics for the endpoint that you would like to monitor.
4. Select the Graphed metrics
   
5. You can choose how you would like this widget to calculate the value. We are going to change the Statistic setting for both metrics to Sum and the Period to 30 days. This displays the total amount of data that has been uploaded and downloaded over the trailing 30 days.

Widget: Connected users

Our next widget will graph unique user sessions over time as a stacked line graph.

1. Go to Logs then Insights.
2. In the log groups box, select your SFTP servers log group.
3. In the code window, copy and paste the following search function:

filter @message like "OPEN"
    | parse @message "*.* OPEN Path=* Mode=*" as user, sessionID, path, mode
    | fields @timestamp
    | sort @timestamp desc
    | stats count_distinct(sessionID) as uniquesessions by bin(10min)

4. Select Run Query.
5. On the visualization tab, select stacked area.
6. Select Actions then Add to dashboard.
7. Select your dashboard, check that the type is stacked line and provide a friendly name (for example, Number of users).

Widget: Uploads and downloads

These widgets display the number of files that are being uploaded and downloaded to Amazon S3.

1. Return to Logs then Insights.
2. In the log groups box, select your SFTP servers log group.
3. In the code window, copy and paste the below search function.

    filter @message like "READ"
    | fields @timestamp
    | sort @timestamp desc
    | stats count(*) as FileRead by bin(10min)

4. Select Run Query.
5. Click Actions then Add to dashboard.
6. Select your dashboard, check that the visualization type is set to “line,” and finally provide a friendly name (for example, Get Operations).
7. On the new widget, click the vertical ellipses (⋮) and choose Duplicate.
8. On the Duplicate tab, edit the name (by selecting the pencil) and change it to “Put Operations”).
9. Click the vertical ellipses (⋮) and choose Edit.
10. In the query box change “READ” to “WRITE” and click save changes.

Widget: Error counts

This widget displays the number of errors that are being logged by the endpoint. It is useful to see if there are any sudden changes in the baseline over time that would reflect an issue in your environment.

1. Go to Logs then Insights.
2. In the log groups box, select your SFTP servers log group.
3. In the code window, copy and paste the below search function.

filter @message like "ERROR"
| fields @timestamp
| sort @timestamp desc
| stats count(*) as Errors by bin(24hr)

4. Select Run Query.
5. On the visualization tab, select stacked area.
6. Select Actions then Add to dashboard.
7. Select your dashboard, check that the type is stacked line and provide a friendly name (for example, Error Count).

Widget: Top downloads

Our last widget shows what the top 10 downloaded files are.

1. Return to Logs then Insights.
2. In the log groups box, select your SFTP servers log group.
3. In the code window, copy and paste the below search function.

filter @message like "OPEN"
| parse @message "*.* OPEN Path=* Mode=*" as user, sessionID, path, mode
| fields @timestamp
| filter mode like "READ"
| sort path desc
| stats count(path) as NumberOfRequests by path
| limit 10

4. Choose Run Query.
5. Select Actions then Add to dashboard.
6. Select your dashboard, check that the type is Query Result and provide a friendly name (for example, Top File Downloads).

Conclusion

In this post, I showed you how to build an Amazon CloudWatch operations dashboard that provides you with insights into the key operational metrics of your AWS SFTP environment. This enables you to view varied and useful information, such as:

• The amount of data transferred in and out of the service.
• The number of unique users and user sessions.
• The number of files uploaded and downloaded.
• The number of error counts.
• The top 10 downloaded file names.

By duplicating the above steps and editing the query strings, you can build your own custom widgets that parse the AWS SFTP logs and provide additional metrics.

The dashboard you’ve just built should help provide additional insights into your AWS SFTP environment. The dashboard enables you to quickly spot trends or unexpected issues without having to step through or search the logs by hand. Such capabilities save you time and money while sparing operational overhead, letting you focus on things more important to your business.

Cleaning up

If you are done with the resources you deployed today, including the AWS SFTP endpoint, don’t forget to clean up and check for any permissions that are no longer required.

Additional Resources

To learn more, visit the AWS SFTP and Amazon CloudWatch Dashboard documentation. Also, dive deeper into AWS SFTP by checking out some of my older posts on this service:

• Enable password authentication for AWS Transfer for SFTP using AWS Secrets Manager
• Simplify your AWS SFTP structure with chroot and logical directories

As always, thanks for reading. If you have any questions, please leave them in the comments section below.