AWS Cloud Operations Blog
How to search through your AWS Systems Manager Session Manager console logs – Part 2
AWS System Manager in combination with Amazon Key Management Services (KMS), Amazon CloudWatch, and Amazon Open Search can provide administrators with the ability to encrypt and securely store user session logs and search the log data for information. These tools are easy to integrate and provide powerful analytical capabilities without undifferentiated heavy lifting.
In the first part of this series, we created and configured permissions for Customer managed keys in Amazon Key Management Service, and we configured AWS Systems Manager Session Manager and Amazon CloudWatch to use Amazon KMS Customer managed keys to securely store session logs to Amazon CloudWatch. In this part, we will query the logs in Amazon CloudWatch, create custom metrics and metric namespaces, and finally show you how to configure the delivery of the logs to Amazon OpenSearch for advanced analytics.
Query the results in CloudWatch
Now that we have log streams being written to our secure CloudWatch log group, let’s try to query the log streams for commands that the user may have executed during the Session Manager session. The names of the log stream contain the user ID (AWS Console ID of the user initiating the Session Manger Session), followed by a set of alpha-numeric characters and this is helpful for the first level of filtration. If you are looking for logs for a specific user, you can filter the log stream name by the user ID, and then click on the log stream to see the details. While the first filtration may have helped you identify the user that you are interested in, you are still required to sift through the log entries to find what was executed by the user. CloudWatch provides you with word search capability to make it easier. As an example, let’s try to find the use of the “sudo” command by all users in the past 12 hours.
From CloudWatch Log groups, select the secure log group that we had created for our session manager logs. You see the list of all of the log streams that represents various users using Session Manger to access the Amazon Elastic Compute Cloud (EC2) instances. Select the Search all button.
This presents a screen with all of the log events listed as rows. In the search bar, enter the word “sudo”, and select enter. Click the 12h to limit the time frame to the past 12 hours. The results are filtered to only log events that contain the word “sudo”, and that have occurred in the past 12 hours. You can select each of the returned rows to examine the exact command entered and the system’s response. This method uses plain text search to narrow down the list of events for you.
There are 10 log events that were returned in our case where the text “sudo” was part of the event log in the past 12 hours. If we need to be more specific and find out how many of these events were used to install software, then we can modify the search by using ‘sudo yum install’. Replace the sudo in the search bar with “sudo yum install” in quotes, and select enter. This will list only the events where the specified text in quotes was found. In our case, only two matches were found, and the user has used the sudo command to install Apache Web server (httpd) in one case and the nano editor in the second case.
Creating metrics using metric filters
CloudWatch provides capabilities that let you generate CloudWatch metrics based on findings in the event logs. Metrics Filters are mechanisms that let you search for and match terms, phrases, or values in your log event. When the metric filter finds a match for the term, you can tie the match to increment the value of a CloudWatch metric. In our case, let’s take the example of trying to identify the number of commands that the user executed during the session that required elevated privileges, such as using “sudo”.
Let’s also set up the sudo-count metric in CloudWatch. From Log groups, select the check box next to the secure log group name, and from action select Create metric filter.
In the next screen for Filter pattern, enter “sudo”. In the Test pattern section, select a log data on which you can try out the pattern match. Once selected, the log events message section is loaded with a sample list of events. Select the Test pattern button, and now the results section will show the events that had “sudo”. Now, select Next.
Enter a name for the metric filter. I used “sudo-filter”. For the namespace, leave the Create new option selected and enter “ssmlogs-ec2” as the value. For the metric name, enter “sudo-count”. The metric name should be unique within the namespace, as we are creating a new namespace, this would be unique. In the Metric value field, enter “1”. This is the value assigned to the filter match. Every time a log event with the word “sudo” is found that corresponds to a value of 1, set the Default value to 0, and in the Unit field select “Count”. Now, select the Next button for the final step.
Review the values, and select the Create metric filter button.
You get a confirmation that the Metric filter has been created. As a next step let’s create a widget that shows the number of times the sudo command was used in the past hour. Use the Systems manager Session Manager to start some sessions on the EC2 instances, and make sure that you execute some commands with sudo. Note that the metric filter and custom metric only collect metric from data that are generated after the metric was created. Note that any previous sudo invocations may not get counted. In the left navigation pane, select All metrics under the Metric menu. Now you will see the namespace ssm-logs-e2 that we just created listed under the Custom Namespaces section.
Select ssmlogs-ec2, and then Metrics with no dimensions. In the corresponding screen, select the checkbox next to sudo-count, select the Graphed metric tab, and select “Sum” from the statistics drop-down. In the top panel, select 3h for the sudo in the past 3 hours, and as we are showing the number of times sudo was executed, select “Number” in the graph type. Now the graph area shows the number of times that sudo was found in the log entries in the past hour.
Select the View graphed metrics button to customize the graph further, such as adding a dynamic label or mathematical expressions. Here, I have added the namespace (ns) name in front of the metric and changed the label of the metric to “sudos.” It will read “[ns: ssmlogs-ec2] sudos” in our case. Select the edit icon next to the label name and enter “[ns: ${PROP(‘Namespace’)}] sudos” to configure this. The label will be updated, and selecting the number in the graph will display the details of the metric as a pop-up.
Based on your use case, the custom metric you have configured can now be added to a dashboard or be used to trigger an alarm if the value of the metric exceeds a certain threshold.
Advanced example: Delivery to Amazon OpenSearch Service
You can configure CloudWatch log group to stream the Session Manger event logs in near real-time to an Amazon OpenSearch service for advanced analytics using the CloudWatch log subscription feature. The logs can be streamed to an existing OpenSearch domain, or you can create a domain quickly using the command line. A detailed guide to create a new domain using the console and command line can be found here. We will use a domain named “my-ssm-logs-domain” for this post. Note, if you are creating a new OpenSearch domain, it takes about 15 minutes for the domain to be initialized. Please proceed with the next steps only after the domain initialization is completed and the Domain status is “Active”.
CloudWatch log subscriptions use Lambda functions to stream log events to an OpenSearch domain. An IAM role with appropriate trust relationships and permissions must be configured in AWS IAM to enable Lambda function to communicate with the OpenSearch domain. As the first step, let’s configure a new policy.
From the console, select IAM service and from the left navigation menu select Policies. Select the Create Policy button on the top right, and in the subsequent screen select the JSON editor. Copy the policy below to the editor and update the <REGION>, <ACCOUNT-ID>, and the <TARGET-DOMAIN-NAME> values to match your situation.
Select the next and Review button. We will name this policy mySSMDomainESPolicy, and select Create policy to complete the policy creation.
Next, we will create a role that will be used by Lambda function to stream the log events. From the left navigation menu on the AWS IAM console, select Role, and then select the Create role button on the top right. As this role is for Lambda, we will select Lambda as the trusted entity from the AWS service list. Select the next button, and in the corresponding screen search for mySSMDomainESPolicy. This is the policy we created in the previous step. Select the policy by selecting the checkbox next to the name, and select next and finally to the review role page. We will name the role LambdaSSMLogsStream and select Create role. This will create the IAM role with necessary permissions that a Lambda function can use to send event logs to our OpenSearch domain. Note, if you would like to see Lambda function execution logs in CloudWatch when the Lambda function gets executed, then attach the AWSLambdaBasicExecutionRole policy to the role. The permissions in this policy are required for Lambda function to be able to write execution logs to CloudWatch.
Once the OpenSearch domain status is active, navigate to CloudWatch in console, and under Log groups select the log group we created in the previous section. From the actions menu, select Subscription filters, and in the submenu select “Create Amazon OpenSearch Service subscription filter”.
In the subsequent screen, for destination select the Amazon Open Search cluster domain name from the drop-down menu. In the Lambda Function section, select the role LambdaSSMLogsStream from the drop-down menu.
In the log format drop-down menu, select JSON as the format. A subscription filter lets you send only filtered log events to OpenSearch. In our case, we will only send commands that had “sudo” in the text. In the Subscription filter pattern, enter “sudo”, and in the Subscription filter name, enter “sudo-filter”.
We can use the Test pattern section to test and confirm if our pattern will pick the relevant events. From the drop-down menu, select a log file from the log group, and log events in the file are shown in the message box. Select the Test pattern button, and the Results section is now populated with events from the log that has sudo. Select the Start streaming button to start the streaming.
Note that previous events (prior to when you configured the subscription filter) will not be sent to OpenSearch, and only new events will be forwarded. Let’s trigger some new events that contain some sudo commands by using Session Manager to validate the streaming configuration. Also, note that we have specified a sudo filter, and only events containing the word “sudo” will be streamed to OpenSearch.
After generating some activity in the Systems Manager and confirming that the logs appear in CloudWatch, we will log in to the Amazon OpenSearch dashboard and see if we can find the Systems Manager logs that contained the sudo command sent over to Amazon OpenSearch. The sudo command is used as an indicative example in this case. This solution can be extended based on the need of the organization to create dashboards that will help administrators find the information that they are looking for from the Systems Manager logs.
Cleanup
You may incur charges for resources created following the steps in this post. To disable the integration of AWS Session Manger System Manager logs, navigate in the AWS console to AWS System Manager, and under Session Manager preferences, edit the preferences and disable KMS and CloudWatch integration. To disable and remove AWS KMS keys, navigate to the Customer Managed Key section and disable the key that was created as part of this post. To disable integration with Amazon Opensearch service, navigate to Amazon CloudWatch in the AWS console, and under the secure log group that was created, select and delete the subscription filter. The secure log group can also be deleted from the console.
Conclusion
In this post, we searched through the logs using CloudWatch console, configured a metric filter based on a filter pattern, configured a new custom namespace, and added a new custom metric to the name space. We also created a dashboard widget based on the custom metric. Then, we configured an IAM policy to enable the Lambda function to take logs from Amazon CloudWatch and send them to the Amazon Opensearch service. We created a subscription filter to send the logs to a previously created Amazon Opensearch domain, and we were able to query the logs in the Amazon Opensearch dashboard. To learn more about AWS System Manager, you can go here, and for more information about Amazon CloudWatch, navigate here.