How to search through your AWS Systems Manager Session Manager console logs – Part 1

AWS Systems Manager (SSM) in combination with AWS Key Management Services (KMS), Amazon CloudWatch, and Amazon OpenSearch allow administrators to encrypt and securely store user session logs, as well as search the log data for information. These tools are easy to integrate and provide powerful analytical capabilities without the undifferentiated heavy lifting.

In this series, we share the procedure for configuring AWS SSM Session Manager to securely store the session logs in Amazon CloudWatch, generate relevant metrics from the logs, and search through the logs for specific events. In Part 1, we will show you how to create a Customer Managed Key (CMK) in Amazon KMS, configure necessary permissions for usage of the keys, create a secure log group in Amazon CloudWatch, and securely send session logs to Amazon CloudWatch. In Part 2, we will query the logs in Amazon CloudWatch, create custom metrics and metric namespaces, and finally show you how to configure the delivery of the logs to Amazon OpenSearch for advanced analytics.

Background

Session Manager is an AWS SSM capability that lets you manage your Amazon Elastic Compute Cloud (Amazon EC2) instances, on-premises instances, and virtual machines (VMs) through an interactive one-click browser-based shell, or through the AWS Command Line Interface (CLI). AWS SSM provides secure instance management without the need to open inbound ports, maintain bastion hosts, or manage SSH keys. Traditionally system administrators use underlying operating system capabilities to log SSH session activities locally to a log file, and then have another mechanism to export this file if needed for investigation. AWS SSM Session Manager can automate the process of logging sessions, as well as transporting and storing the session logs in an encrypted central log store in Amazon CloudWatch. Amazon CloudWatch provides powerful capabilities to extract metrics from logs collected, along with sharing the information only to personnel with the right permissions levels.

Solution overview

The solution uses AWS Identity and Access Manager (IAM) roles to provide the right permissions levels to access the console logs. AWS KMS is used to create CMKs that encrypt the console logs. The solution also configures a secure log group in Amazon CloudWatch to store the Session Manager console logs.

Some important notes about this solution:

• This solution uses AWS KMS single-Region key, which is stored and used only in the Region in which it was created. The Multi-Region key option can be selected if your implementation requires the key to be accessible in multiple-regions. However, this may entail additional configuration and steps that are not discussed in this post.

Prerequisites

Before starting, you will need the following prerequisites:

1. You must already be using AWS SSM. If not, then you can use our Quick Setup process to rapidly create the required IAM roles and configure the service.
2. Each EC2 instance or on-premises server that you wish to access through Session Manager must have the SSM Agent installed and properly configured. Many of the base Amazon Machine Images (AMIs) come with this agent pre-installed. See this link for details about this configuration.
3. The IAM role used by your instance profile must include either the AmazonSSMManagedInstanceCore managed policy, or equivalent permissions to what this would provide.
4. Select a name for your new CloudWatch Logs log group in advance, ensuring that your log group name must be unique.

With these prerequisites met, we can proceed.

Step one: Create Amazon KMS key and policies

First, we must create a new CMK within Amazon KMS. Using a CMK to protect your session logs is required to enable certain highly sensitive data obfuscation tasks, such as masking keyboard strokes during password reset operations. Note that the creation of a KMS key will incur a small charge on your account. You can find more about pricing for Amazon KMS here

The use of KMS encryption for session log data is a best practice.

From the KMS console, first select CMKs, and then Create key.

Next, specify that this is a symmetric key, and use KMS for the key material origin. Note that some customers may use Cloud Hardware Security Module (HSM) or import their own key material. In that case, they can proceed accordingly. The regionality is selected to be Single-Region key.

Enter a name for your new CMK. We will use “session-manager-logs” for this solution. Alternatively, you can use a name that is meaningful to you and your organization. Click Next.

Now, the CMK creation process will ask you to select which users or roles you wish to permit to administer this key. This is a very sensitive action, and must be constrained to as small of a group as possible. I suggest adding only your most privileged users from your organization.

The users of this key must also be defined. This must also be as limited of a group as possible, but it will need to include those whom you will allow to read your session log data as well as the IAM role that you use with your instance profile. For example, if you are using a custom role “EC2RoleSSMCWSecureLogs” as the IAM role for your EC2 instances, then this role will need key usage permissions and will need to be selected in the list.

Choose Next, and choose Finish to complete the CMK creation. You will see a success message that the customer master key was created with the alias “session-manager-logs” (or the name that you had specified in Step 2).

Next, we must manually edit the key policy to include a section that grants the CloudWatch Logs service access to this key. Select the alias or Key ID hyperlink to view the details of the key that we just created. Note the Amazon Resource Name (ARN) for the newly created key, as we will need this value when creating the secure log group. Under the Key Policy section, select the “Switch to policy view” button.

In the resulting screen, select the “Edit” button to manually edit the key policy.

Paste the policy statement section given as follows into the policy as a new grant, being sure to replace <REGION>, <ACCOUNTNUMBER>, and <LOGGROUPNAME> with the appropriate values.

{
            "Sid": "Allow access for CloudWatch secure log group",
"Effect": "Allow",
            "Principal": {
                "Service": "logs.<REGION>.amazonaws.com"
            },
            "Action": [
                "kms:Encrypt*",
                "kms:Decrypt*",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:Describe*"
            ],
            "Resource": "*",
            "Condition": {
                "ArnEquals": {
                    "kms:EncryptionContext:aws:logs:arn": "arn:aws:logs:<REGION>>:<ACCCOUNTNUMBER>:log-group:<LOGGROUPNAME>"
                }
            }
        }

Select Save Changes to save the policy changes made. Note the ARN for the key. With this step completed, we now have created a Customer Managed symmetric key and given the log group named “ssm-secure-consolelog-group” permission to use the key for the actions specified in the action block of the policy.

Step 2: Create CloudWatch Logs log group

For the next step, we will create an encrypted log group with the same name that we had specified in the policy section for the CMK.

From the CloudWatch Logs console, select Create log group.

In the Log group details page, make sure that you enter the same name value that was specified in the policy for the CMK (“ssm-secure-consolelog-group” in our case). Select a retention period that matches your need, and enter the ARN for the new KMS key that we noted earlier. Select the create button to complete the creation of the Log group.

With the encrypted log group created, next we will configure our Amazon EC2 instances to send session manager console access logs to the secure CloudWatch log group that we created.

Step 3: Update your EC2 instance profile

For your EC2 instances to be able to send the Session Manger logs to CloudWatch, you must update the IAM Role used by your EC2 instance by permitting it to communicate with the CloudWatch Logs log group that you just created. The AmazonSSMManagedInstanceCore managed policy doesn’t include these permissions, so they must be added manually prior to using Session Manager.

As part of the prerequisites, you should have already set up a role attached with an AWS Managed AmazonSSMManagedInstanceCore policy and used it as the IAM Role for your EC2 Instances. In the following example, you can see an IAM role “EC2RoleSSMCWSecureLogs” that is used by the EC2 instance.

Selecting the IAM Role hyperlink will open Amazon IAM with the role selected. You must add the following policy to your existing role (“EC2RoleSSMCWSecureLogs” in this example), which will allow the local SSM agent to use your new log destination, again being sure to replace the <REGION>, <ACCOUNTNUMBER> and <LOGGROUPNAME>.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "logs:DescribeLogGroups",
            "Resource": "arn:aws:logs:<REGION>:<ACCOUNTNUMBER>:log-group:*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:CreateLogGroup",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:<REGION>:<ACCOUNTNUMBER>:log-group:<LOGGROUPNAME>:log-stream:*",
                "arn:aws:logs:<REGION>:<ACCOUNTNUMBER>:log-group:<LOGGROUPNAME>"
            ]
        }
    ]
}

You will add this inline policy-start by selecting the Add inline policy link. This will open the policy editor.

In the policy editor, select the JSON tab, and then copy and paste the policy statement. Again, remember to replace the ,<REGION>, <ACCOUNTNUMBER> and <LOGGROUPNAME> with values that are accurate for you. Make sure that there are no errors, and select the Review Policy button.

In the Review policy page, specify a name for the inline policy statement, and select the Create policy button.

The policy is now created inside the Role as an inline policy.

Step 4: Enable logging to Amazon CloudWatch Logs

Now that we have properly configured the KMS keys, key access policy, CloudWatch log group, and access for EC2 to the CloudWatch log group, we can enable the delivery of session logs to CloudWatch. To do so, navigate to the AWS SSM console, select Session Manager, then Preferences, and you can see that the KMS Encryption and CloudWatch logging are disabled by default. You must enable it for the solution to work. Select the Edit button to proceed.

First, enable KMS encryption by selecting the check-box and selecting the KMS key with the name that we created earlier from the drop down of KMS Keys.

Scroll down the preference page to the section on CloudWatch. Here, enable CloudWatch logging, and then select the log group that you created earlier.

Save your changes to complete the process. This concludes the set-up part of the solution. Now, to check if we have the solution working, start a session using the session manager, and type a few commands in the session prompt. Open CloudWatch and under Logs menu, select Log groups and select the secure log group that we created, and we can see log files with information about the session, the commands that were entered, and the response that was displayed to the user during the session.

Troubleshooting

Most often, when issues prevent logs from being uploaded to CloudWatch Logs, you can diagnose these from the Amazon SSM Agent logs. If you don’t see logs being delivered to CloudWatch as expected, then check the logs from your EC2 instance (or on-premises server), and see if errors are related to key permissions. Here is an example of a misconfigured key permission mapping:

grep ERROR /var/log/amazon/ssm/amazon-ssm-agent.log |grep ssm-session-worker | tail 
2021-03-29 23:09:29 ERROR [ssm-session-worker] [me@me.local] [DataBackend] [pluginName=Standard_Stream] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/managed-ec2-ssm-role/i-024259ed24a2cbaeb is not authorized to perform: logs:DescribeLogGroups on resource: arn:aws:logs:us-east-1:<ACCOUNT_NUMBER>:log-group::log-stream:
2021-03-29 23:09:29 ERROR [ssm-session-worker] [me@me.local-] [DataBackend] [pluginName=Standard_Stream] Validation failed, err: We couldn't start the session because encryption is not set up on the selected CloudWatch Logs log group. Either encrypt the log group or choose an option to enable logging without encryption.

Conclusion

Part 1 of this series outlines the process to save and search AWS SSM Session Manager logs in Amazon CloudWatch. We created an encryption key using Amazon KMS and used the key to encrypt Session Manager logs that are sent to Amazon CloudWatch. In Part 2, we will show you how to search through the logs using the CloudWatch console, and configure a metric filter based on a filter pattern in a new metric namespace. We will also configure the log data to be sent to Amazon OpenSearch for detailed analytics.

About the authors