Auto-remediate best practice deviations detected by AWS Trusted Advisor

AWS Trusted Advisor inspects your AWS infrastructure and provides best practice recommendations when opportunities exist to reduce cost, optimize your AWS infrastructure, improve system availability and performance, help close security gaps and monitor service quotas. Trusted Advisor recommendations are based on best practices identified by AWS services experts and learnings from serving thousands of customers over time.

In this blog, we present an architecture pattern that you can implement to automatically remediate best practice deviations detected by AWS Trusted Advisor. Auto remediation of the AWS Trusted Advisor recommendations help:

• Streamline your operations,
• Respond to security vulnerabilities quickly
• Maintain an optimized AWS environment.

Exposed IAM Access Keys pose a security risk to your AWS accounts and users. Customers are ultimately responsible for the safety and security of access keys. In our example, we will show you a solution that automatically remediates the Exposed Access Keys security check.

Overview of solution

Here’s a general approach:

1. Sign in to the Trusted Advisor console
2. Identify security recommendation: Use Trusted Advisor to identify security recommendations in your AWS infrastructure. Trusted Advisor checks for common security vulnerabilities, such as Exposed Access Keys, Amazon S3 Bucket Permissions, Security Groups with Unrestricted Access and many more. A detailed list of Trusted Advisor Recommendations is available here. If AWS Security Hub is enabled in the Account, Trusted Advisor helps to identify potential security gaps reported in the AWS Security Hub Foundational Security Best Practices.
3. Configure event-driven automation: Subscribe to Trusted Advisor events in Amazon EventBridge to receive notifications about events you intend to automatically remediate. Using an Amazon EventBridge rule, forward the event to an AWS Lambda target .
4. Implement auto-remediation actions: Create a Lambda function as the target to respond to the event from Amazon EventBridge and execute remediation steps.
5. Notify security team: Ensure that your Lambda function logs the details of the remediation actions taken. Use Amazon Simple Notification Service (Amazon SNS) for sending notifications about the auto-remediation status.

Solution Architecture

The following  diagram illustrates the solution architecture to automatically remediate the AWS Trusted Advisor security check Exposed Access Keys recommendation.

Figure 1: Solution architecture for automatic remediation

Solution Workflow

The solution workflow is composed of the following steps:

1. Trusted Advisor detects the publicly exposed key and sends an Exposed Access Keys security event to the default event bus in EventBridge. The structure of the Exposed Access Key event looks like the following:

{
  "version": "0",
  "id": "1234abcd-ab12-123a-123a-1234567890ab",
  "detail-type": "Trusted Advisor Check Item Refresh Notification",
  "source": "aws.trustedadvisor",
  "account": "123456789012",
  "time": "2018-01-12T19:38:24Z",
  "region": "us-east-1",
  "resources": [],
  "detail": {
    "check-name": "Exposed Access Keys",
    "check-item-detail": {
      "Case ID": "12345678-1234-1234-abcd-1234567890ab",
      "Usage (USD per Day)": "0",
      "User Name (IAM or Root)": "my-username",
      "Deadline": "1440453299248",
      "Access Key ID": "AKIAIOSFODNN7EXAMPLE",
      "Time Updated": "1440021299248",
      "Fraud Type": "Exposed",
      "Location": "www.example.com"
    },
    "status": "ERROR",
    "resource_id": "",
    "uuid": "aa12345f-55c7-498e-b7ac-123456781234"
  }
}

To monitor AWS Trusted Advisor check results with Amazon EventBridge, please check the product documentation here.

2. A rule created in Amazon EventBridge filters out AWS Trusted Advisor events matching the Exposed Access Keys event as shown in step 1 above. It then triggers AWS Step Functions to initiate the remediation workflow.

3. The step function workflow remediates the security exposure by invoking three different Lambda functions, each accomplishing a specific task.

4. The Lambda functions perform following tasks

4.1 Lambda Function: delete_access_key_pair: This function disables or deletes the exposed access key pair in IAM.
4.2. Lambda Function: lookup_cloud_trail_events: This function queries AWS CloudTrail logs to fetch the most recent API invocations linked to the user associated with the exposed access keys. The default lookup interval is set to 24 hours, but it can be adjusted as per user preference.
4.3. Lambda Function: notify_security: This function sends a notification to the recipient about the remediation action taken. It includes the data retrieved from CloudTrail logs by the Lambda function “lookup_cloud_trail_events”. Alerts can be sent to a pre-configured Amazon SNS topic with an email subscription. The alert destination and recipients can be customized by the user according to their needs.

Technical Architecture Implementation

This section explains how to implement the architecture in a single AWS account, or across multiple accounts within AWS Organization.

Deploying the solution in a single AWS account

The prerequisites to deploy this solution are:

1. Trusted Advisor should be enabled in the account.

2. The account must have a Business, Enterprise On-Ramp, or Enterprise Support plan to create EventBridge rules for Trusted Advisor Recommendations. You can verify your Account Support plan from the AWS Support Center. For more information on AWS Support plans, see Compare AWS Support plans.

3. For more information on Monitoring AWS Trusted Advisor check results with Amazon EventBridge, review the AWS Trusted Advisor and EventBridge page.

A more detailed step by step guide to deploy and test the solution using AWS CloudFormation is available in AWS Trusted Advisor GitHub.

Deploying the solution in a multi-account AWS Organization

AWS Organizations helps you centrally govern your environment as you grow and scale your workloads on AWS.

To implement the solution across multiple accounts within AWS Organizations, it is necessary to consolidate AWS Trusted Advisor events from each individual account into a designated central or governance account.

The following diagram (Figure 2) illustrates the solution architecture to extend a solution to AWS organization with multiple accounts. AWS Trusted Advisor events are sent to the “default” event bus of each member account. The events are forwarded to a target event bus in EventBridge of the central governance account. An EventBridge rule in the central governance account triggers AWS Step Functions to initiate the remediation workflow in response to events received from participating member accounts.

Figure 2: Architecture for deploying across multiple Organization accounts

A step by step approach to centralize EventBridge events from member AWS Organizations accounts into a central account using AWS CloudFormation StackSets and CDK is discussed here.

Customizing the solution

We have discussed an approach to automatically remediate the Exposed Access keys recommendation from Trusted Advisor’s security pillar. This architecture pattern can be extended to automatically remediate other security recommendations, and recommendations from Trusted Advisor’s fault tolerance, cost optimization, and performance pillars. To do so, identify the desired Trusted Advisor recommendation to auto-remediate. The recommendation name can be found in the “check-name” field of the event delivered by EventBridge. See the sample event under the “Solution Workflow” above for reference. Implement custom remediation actions in the associated target Lambda function by following the approach as discussed in the sections above.

For other examples to auto-remediate deviations detected by AWS Trusted Advisor recommendations visit AWS Trusted Advisor Tools GitHub.

Conclusion

In this blog post, we detailed a solution to automate remediation of AWS Trusted Advisor recommendations. The solution is applied to a single AWS account and AWS Organization. The solution is useful for AWS Trusted Advisor recommendations that need rapid response times and continuous protection of your AWS resources. It eliminates manual processes as they can be prone to human error. Automatic remediation also allows for scalability by enabling simultaneous resolution across multiple resources or accounts, and reducing the burden on administrators. This proactive approach reduces the window of vulnerability, minimizes deviations from AWS best practice recommendations, and maintains a secure and optimized environment.

About the authors: