How do I troubleshoot Systems Manager Run Command failures?

Last updated: 2021-03-24

I'm trying to use the AWS Systems Manager Run Command to run a command document on my managed instances, but the process fails. How can I troubleshoot this issue?

Resolution

Prerequisites

AWS Systems Manager Run Command allows you to automate common administrative tasks and perform one-time configuration changes at scale. Before you can manage instances using Run Command, you must configure an AWS Identity and Access Management (IAM) user policy. The user policy is required for any user who will run commands.

  1. Verify that an IAM instance profile role for Systems Manager is attached to your Amazon Elastic Compute Cloud (Amazon EC2) instances. For more information, see Create an IAM instance profile for Systems Manager.
  2. Review the IAM policy created for the role or user. The policy must include permissions for ec2messages API calls, because the endpoint is required to send and receive commands.

Troubleshoot Run Command failures

In the Systems Manager console, the instance must be listed under Managed instances, and the SSM Agent ping status must be Online.

Review Run Command status details

  1. Review the Run Command status details.
  2. Open the Systems Manager console, and then choose Run Command from the navigation pane.
  3. Choose the hyperlinked Command ID to open the Command status page.
  4. From the Targets and outputs section, choose the hyperlinked Instance ID, and then review the output.

If the output is truncated, connect to the EC2 instance using SSH, and then navigate to the following directories to view the full error details. Note the exit status codes, and then see Troubleshooting Systems Manager Run Command for further troubleshooting steps.

For Linux and macOS:

  • /var/lib/amazon/ssm/<instance-id>/document/orchestration/<command-id>/<Plugin-name>/<Step-name>/stdout
  • /var/lib/amazon/ssm/<instance-id>/document/orchestration/<command-id>/<Plugin-name>/<Step-name>/stderr

For Windows:

  • %ProgramData%\Amazon\SSM\InstanceData\<ManagedInstance-ID>\document\orchestration\<Command-ID>\<plug-in>\<step_number.plug-in>\stdout
  • %ProgramData%\Amazon\SSM\InstanceData\<ManagedInstance-ID>\document\orchestration\<Command-ID>\<plug-in>\<step_number.plug-in>\stderr

Review SSM Agent logs

Review the SSM Agent logs for more details about the failure:

For Linux and macOS, locate the logs in the following directories:

  • /var/log/amazon/ssm/amazon-ssm-agent.log
  • /var/log/amazon/ssm/errors.log
  • /var/log/amazon/ssm/audits/amazon-ssm-agent-audit-YYYY-MM-DD

For Windows, locate the logs in the following directories:

  • %PROGRAMDATA%\Amazon\SSM\Logs\amazon-ssm-agent.log
  • %PROGRAMDATA%\Amazon\SSM\Logs\errors.log
  • %PROGRAMDATA%\Amazon\SSM\Logs\audits\amazon-ssm-agent-audit-YYYY-MM-DD


Did this article help?


Do you need billing or technical support?