How can I troubleshoot a State Manager association that failed or that is stuck in pending status?

Last updated: 2021-03-24

I created a State Manager association scheduled to run on my managed Amazon Elastic Compute Cloud (Amazon EC2) instance. However, the association status is failed or stuck in pending. How can I troubleshoot this issue?

Resolution

AWS Systems Manager State Manager association is a configuration assigned to your managed instances. The configuration defines the state that you want to maintain on your instances.

When you create a State Manager association, Systems Manager binds the schedule, targets, document, and parameter information that you specify to the managed instances. The association status is initially pending while the system tries to reach all targets and immediately apply the state specified in the association.

Troubleshoot an association stuck in pending/failed status

If the State Manager association remains stuck in pending or failed state, first confirm that the latest version of SSM Agent is installed.

Then, verify the status of the resource where the association is applied, and view the history to confirm if there were any invocations.

  1. From the Systems Manager console State Manager Associations page, choose the hyperlinked Association id for the association that is stuck in pending or failed state.
  2. Choose the Execution history tab to view the invocation history.
  3. If the history lists invocations, choose the hyperlinked Execution id to see the resource type, status, and other details. Then, proceed to the Identify the cause of the failure section of this article.

If there aren't any invocations listed in the history, verify that the instance is a managed instance. From the Systems Manager console, the instance must be listed under Managed instances, and the SSM Agent ping status must be Online.

If your instance doesn’t appear under Managed instances, see Why is my EC2 instance not appearing under Managed Instances in the Systems Manager console?

If the SSM Agent ping status is Connection Lost, see How can I troubleshoot a Systems Manager managed instance in Connection Lost status?

Identify the cause of the failure

If the history lists invocations, from Execution ID Association execution targets page, select the target instance Resource ID, and then choose Output. The output displays details and an error message about why the association failed.

Note: The output differs depending on the Systems Manager document that you use. For more information, see AWS Systems Manager documents.

Review SSM Agent logs

Review the SSM Agent logs for more details about the Run Command document failure:

For Linux and macOS, locate the logs in the following directories:

  • /var/log/amazon/ssm/amazon-ssm-agent.log
  • /var/log/amazon/ssm/errors.log
  • /var/log/amazon/ssm/audits/amazon-ssm-agent-audit-YYYY-MM-DD

Note: SSM Agent stderr and stdout files write to the /var/lib/amazon/ssm directory.

For Windows, locate the logs in the following directories:

  • %PROGRAMDATA%\Amazon\SSM\Logs\amazon-ssm-agent.log
  • %PROGRAMDATA%\Amazon\SSM\Logs\errors.log
  • %PROGRAMDATA%\Amazon\SSM\Logs\audits\amazon-ssm-agent-audit-YYYY-MM-DD

Did this article help?


Do you need billing or technical support?