Managing aged access keys through AWS Config remediations

One of the security best practices that is time-consuming to manage is enforcing IAM access key rotation for IAM users. Access keys give IAM users the ability to connect to Amazon EC2 instances. Therefore rotating these regularly (for example, every 90 days) is one of the key steps in protecting your resources from unauthorized access. However, it is often the case that unless staff is dedicated to managing them, enforcing regular rotation can quickly become overlooked. This is more common with companies, which manage hundreds of IAM users across dozens of AWS accounts.

An option to enforce compliance is to use one of AWS Config’s features, automatic remediation. These are declared in AWS Systems Manager automation documents, which are invoked by AWS Config when a resource is found to be non-compliant. As its name implies, these steps are meant to remediate non-compliant resources, but in our case,  we use them as part of a notification system.

In this post, I show you how to set up Systems Manager and AWS Config automatic remediations with a centralized notification system. I also go over some of the options available for further processing.

Prerequisites

Multi-account management

This post assumes that you are using AWS Organizations or have set up a master account, which monitors and aggregates the results from other accounts within your company. Therefore it uses AWS CloudFormation StackSets to automatically propagate and execute stacks from a master account to the target accounts. If you don’t have this setup or want to avoid using StackSets, you can alternatively deploy Step 2 as individual stacks in each target account.

If you use AWS Organizations you will need your Organization ID. When you sign in to the organization’s management account in the AWS Organizations console, you can view details of the organization. The Organization ID will be shown on the left side of the console.

If you don’t use AWS Organizations but will use a master account with target accounts you will need a list of the AWS account IDs for all target accounts.

To deploy using StackSets, make sure that your master and target accounts are set up with the right permissions for these to execute. If you don’t use AWS Organizations or you prefer to have control over the IAM roles used, set up self-managed permissions. If you use AWS Organizations you have the option of using service-managed permissions instead.

AWS Config configuration

You must have AWS Config enabled in your master and targets accounts. Additionally, make sure that your setup is monitoring all global resources or at minimum IAM user resources. For more information, check getting started with AWS Config.

To check the aggregated results from all target accounts in the AWS Config console of your master account, you must have authorized your master account to aggregate the results.

Overview of solution

The described solution relies on a master account from where you monitor and manage the IAM users across multiple target accounts. In each target account, you add an automatic remediation to the AWS Config access-keys-rotated rule. The remediation executes a Systems Manager automation document, which resolves the user name and then publishes the information to Amazon SNS for further processing.

Here is how the process works:

1. In every target account, AWS Config executes the rule and invokes the SSM automation document for every non-compliant resource
2. The automation document does the following steps.
   • Resolve the IAM user name from the IAM user resourceId through an API call
   • Publish a customized message to an SNS topic in the master account

Further processing and alternative customizations can be done via Amazon Simple Queue Service (Amazon SQS) queues, Jira, Slack, AWS Lambda, AWS tagging, etc.

AWS Config invokes the remediation automation document. AWS Config passes non-complying ResourceId and an IAM role AutomationAssumeRole. This role grants all the necessary permissions for AWS Config to execute the automation document.

The first step in the automation document is to resolve the corresponding IAM user name (for example, AliceAdmin) from the IAM user ResourceId (for example, AIDAAXW). This is done by invoking the aws:executeAwsApi command. This command is powerful since it lets you execute API calls from within an automation document. This command is executed without the need for provisioning and paying for any infrastructure. In our case, we invoke the equivalent of the following AWS CLI call: aws configservice list-discovered-resources. 

- name: configStep
  action: "aws:executeAwsApi"
  inputs:
    Service: config
    Api: ListDiscoveredResources
    resourceType: "AWS::IAM::User"
    resourceIds:
      - "{{ResourceId}}"
  outputs:
    - Name: configUserName
      Selector: "$.resourceIdentifiers[0].resourceName"
      Type: String

The next step in the automation document is to craft a message to publish to an SNS topic. It appends the Account ID into the message by using an SSM system variable global:ACCOUNT_ID. It also adds the previous step’s user name using the format step-name.Output-name (for example, configStep.configUserName) to the message.

- name: publishMessage
  action: "aws:executeAutomation"
  maxAttempts: 1
  timeoutSeconds: 30
  onFailure: Abort
  inputs:
    DocumentName: AWS-PublishSNSNotification
    RuntimeParameters:
      TopicArn: !Ref SNSTopic
      Message: Account "{{global:ACCOUNT_ID}}" User "{{configStep.configUserName}}" needs to rotate their access key.

It then uses the aws:executeAutomation command to invoke other automation documents. In this case, we’re invoking the AWS managed automation AWS-PublishSNSNotification, which publishes to Amazon SNS.

Walkthrough

Following are the steps to be performed.

1. Deploy AWS CloudFormation stack to set up the master account with an SNS topic.
2. Deploy AWS CloudFormation stack to set up the target accounts with following steps.
   • Set up IAM roles.
   • Create the SSM automation document.
   • Enable the AWS Config rule access-keys-rotated.
   • Link the automation document as the automatic remediation step.

Let’s get started!

Download the AWS CloudFormation template from AccessKeyRotationParentAccount.yaml and save it to a file on your computer.

Navigate to the AWS CloudFormation console and select stacks on the sidebar menu. Then select create stack, and from the pull-down menu select with new resources (standard).

On the Create stack page, select Upload a template file, and select file. Select the YAML file you saved and then click Next.

In the specify stack details page:

1. Enter AccessKeyRotationMaster as the stack name.
2. Choose the MultiAccountMethod to indicate if you will be using AWS Organizations or a list of target accounts.
3. Enter either the AWS Organizations ID or the comma-separated list of target accounts you want to monitor for compliance.
4. Click Next.

On the Configure stack options page, add any necessary tags, and select Next. Review all the information and select create stack to submit your stack configuration. Once the page refreshes, you find stack in CREATE_IN_PROGRESS status and the status changes to CREATE_COMPLETE once it’s done. When it is finished, navigate to the Outputs tab and copy the newly created SNS topic Amazon Resource Name (ARN) in the value column.

Download the AWS CloudFormation template from AccessKeyRotationChildAccounts.yaml and save it to a file on your computer.

Go to the AWS CloudFormation console and select StackSets on the sidebar menu, then select Create StackSet.

On the Choose a template page:

1. in the Permissions section:
   • if you are using AWS Organizations, choose whether to use service-managed or self-managed permissions (see Prerequisites)
   • if you are using self-managed permissions or do not use AWS Organizations, enter the IAM execution role name (see Prerequisites)
2. select Upload a template file and select file. Select the YAML file you saved and then select Next.

On the Specify StackSet details page, use the following details.

1. Enter AccessKeyRotationTarget as the StackSet name
2. Edit the maxAccessKeyAge and MaximumExecutionFrequency parameters to your requirements.
   • maxAccessKeyAge denotes up to how many days old a key can be and still be compliant.
   • MaximumExecutionFrequency denotes how frequently AWS Config evaluates the access-keys-rotated rule.
3. Enter the KMS key ARN from the AccessKeyRotationMaster step. This key is used to encrypt messages in SNS.
4. Enter the SNS topic ARN from the AccessKeyRotationMaster step. This is the SNS topic that receives the messages of non-compliant IAM users.
5. Click Next.

On the Configure StackSet options page, add any required tags and click Next.

In the Set deployment options page:

• if using AWS Organizations, choose whether to deploy to the organization or specific organizational units
  
• if not using AWS Organizations, enter the target AWS account numbers and AWS Regions where you’d like to execute this automated process as a comma separated list (or you can upload a .csv file with the numbers).

Select the region to deploy to and click Next. On the next page, review the options chosen, select the “I acknowledge” Check box at the bottom of the page, and select Submit.

At this point you can refresh your browser to check the status of the operation you submitted. Once it’s done, the status turns to SUCCEEDED and AWS Config runs the rule to find non-compliant IAM users in each target account and publishes a message for each non-compliant IAM user found to your SNS topic in the master account.

Check Rules on target account

Once your deployments are finished, you can find your rules in action. Log in to one of the target accounts and go to the AWS Config console, select Rules on the sidebar menu. Scroll down the results and select the access-keys-rotated rule.

On this screen, you find the maxAccessKeyAge and time-interval configuration that was set up via AWS CloudFormation and the list of IAM users with non-compliant access keys for the target account. The Action status reveals whether the automatic remediation ran successfully or with errors.

Click the Edit button to look at the automatic remediation you created. On this screen you find the details of the remediation. The remediation action chosen is the AWS CloudFormation generated automation document and is set up to run automatically. The Resource Id parameter is mapped to the resourceId parameter defined in the automation document. The AutomationAssumeRole parameter is hardcoded to an IAM role that was created via AWS CloudFormation.

Further processing options

There are many options for further processing of the messages generated by the automation document. You can apply different levels of customization to automate this process further or to facilitate administrators with limited access or expertise to manage the notification process. For example, the options are to add a subscription to the generated SNS topic as follows.

1. Send the messages to an Amazon SQS queue to be monitored by someone with lesser access or fed into an application.
2. Send an email to a security administrator who may not have access to the AWS account.
3. Invoke a Lambda or an endpoint

Alternatively, you could modify the AWS CloudFormation template from Step 2 and instead of invoking the AWS-PublishSNSNotification in the SSM automation document you could select the following options.

1. Create and assign a Jira ticket using the pre-built AWS-CreateJiraIssue automation
2. Feed the messages into a slack channel by invoking a slack blueprint Lambda through the pre-built AWS-invokeLambdaFunction automation document. More info at Slack integration blueprints.
3. Invoke a REST API endpoint by using the aws:executeScript command, which lets you run Python and PowerShell scripts. More info at creating automation documents that run scripts.

A fully automated option could also be achieved with requirement of all IAM users to have a tag with an email address. Then edit the automation document to invoke the aws iam get-user AWS CLI call to read the email address tag, and email the non-complying IAM users directly. This option would reduce the manual process of figuring out who each IAM user is.

Cleanup process

It is important to note that once the initial run has been successful and captured all non-compliant IAM users, you may want to change the remediation to run only on configuration changes, instead of on a daily basis. This should cut down on costs dramatically.

If you want to delete the configuration you can:

• delete the resources in the target accounts by deleting the stack instances in the AccessKeyRotationTarget StackSet.
• delete the SNS topic by deleting the AWS CloudFormation AccessKeyRotationMaster Stack in the master account.

Conclusion

In this post, I showed you how to set up an SSM automation document as an automatic remediation for the access-keys-rotated AWS Config rule. The document expands the information passed by AWS Config and crafts an actionable message, publishing it to an SNS topic. I also provided some options available for processing the message further via different API operations and solution for administrators with different levels of access to monitor the compliance data.

AWS Config and Systems Manager are available in all major AWS Regions. For more information, check Systems Manager automation and remediation with AWS Config rules.

About the Author

Melina Schweizer is a Senior Solutions Architect at AWS. She works developing infrastructure for AWS GameDay to help provide the backbone for gamified learning experiences. In her spare time, Melina enjoys playing the piano, gardening, and vacationing in Europe with her family.