Streaming AWS Systems Manager Run Command output to Amazon CloudWatch Logs

April 12, 2021:  We updated the blog post  to reflect new Run Command service quotas.

AWS Systems Manager Run Command is designed to let you remotely and securely manage instances. Run Command provides a simple way of automating common administrative tasks like running shell scripts, installing software or patches, and more. All actions taken with Run Command are recorded by AWS CloudTrail to audit changes and allows Amazon CloudWatch Events notification set up to notify when the command finishes. The Run Command API has a limit of 24,000 characters for the output, so specifying an Amazon S3 location will ensure that the full output is captured.

Today we announce the ability to stream Run Command output to Amazon CloudWatch Logs in addition to an Amazon S3 location. With this option, Run Command output and error logs will be uploaded to CloudWatch streams on a periodic basis. You can monitor your output logs, in near real-time, search for specific phrases, values, or patterns, and even create alarms based on the search.

Steps to create CloudWatch logs for a Run Command execution

To use the CloudWatch upload functionality, your instances must be running Systems Manager (SSM) Agent version 2.2.800.0 or later.

Step 1. Configuring IAM role – If your instance is using a role containing the AWS managed policy (AmazonEC2RoleforSSM), no change is needed. If you are using your own policy, please ensure that the instance role has access to create groups and streams in CloudWatch and to put log events.

Sample IAM policy to provide instance access to CloudWatch for Run Command logs:

    {

            “Effect”: “Allow”,

            “Action”: [

                “logs:CreateLogGroup”,

                “logs:CreateLogStream”,

                “logs:DescribeLogGroups”,

                “logs:DescribeLogStreams”,

                “logs:PutLogEvents”

            ],

            “Resource”: “*”

        },

Step 2. Executing Command – In the AWS Management Console under Run Command, you can specify CloudWatch as an output option when sending a command. The CloudWatch log group name is optional, and if you don’t provide it, a log group with the name format “aws/ssm/<DocumentName>” will be automatically created if it doesn’t exist.

You can also execute the command from the AWS CLI using SendCommand and specifying the “cloud-watch-output-config” section, which contains the “CloudWatchOutputEnabled” parameter and optionally the “CloudWatchLogGroupName” parameter to specify the log group name.

Example:

aws ssm send-command –document-name “AWS-RunPowerShellScript” –parameters commands=[“echo helloWorld”] –targets “Key=instanceids,Values=i-0cb2b964d3e14fd9f” –cloud-watch-output-config ‘{“CloudWatchLogGroupName”:”TestCWL”,”CloudWatchOutputEnabled”:true}’

Step 3. Viewing Output – After the command starts executing, you can view the output in CloudWatch in near real time. The log stream name will be of the format:

“<CommandID>/<InstanceID>/<PluginID>/stdout”

“<CommandID>/<InstanceID>/<PluginID>/stderr”

Log Streams are only created when output data is available. For example, if there is no error data for an execution, the stderr stream will not be created.

Output from the execution is uploaded periodically every couple of seconds or when the buffer exceeds 200 KB, whichever happens first.

Sample Output:

Example Log Group and Streams:
Group – /aws/ssm/AWS-RunShellScript

Streams –

68c4cfb2-5be3-4265-9d24-86071299ead9/i-04cc0e814b2414f77/AWS-RunPowerShellScript/stdout
24/68c4cfb2-5be3-4265-9d24-86071299ead9/i-04cc0e814b2414f77/AWS-RunPowerShellScript/stderr

Conclusion
Run Command simplifies managing remote instances at high scale. By using CloudWatch logs, you can now monitor command execution progress in near real time and set up alarms based on the logs. This provides a new and powerful way to keep track of your Run Command executions.

About the Author
Munindra Das is a Software Development Manager for Amazon Systems Manager. Systems Manager is a management service that helps you automatically collect remotely administer, software inventory, apply OS patches, create system images, and configure Windows and Linux operating systems. By providing a management approach that is designed for the scale and agility of the cloud but extends into your on-premises data center, Systems Manager makes it easier for you to seamlessly bridge your existing infrastructure with AWS.