AWS Cloud Operations & Migrations Blog

Standardize compliance in AWS using DevOps and a Cloud Center of Excellence (CCOE) approach

In large scale cloud migrations and deployments, a Cloud Center of Excellence (CCOE) organizational setup enables project and account owners to move at their own speed, reuse components, and integrate them into their application environments. The CCOE team can develop better tools and automation for the application teams to leverage, and they can be sure that best practices are applied throughout the organization.

This post provides a solution utilizing DevOps automation with AWS Service Catalog to enable CCOE administrators to enforce a standardized compliance posture across every managed AWS account in the organization.

The CCOE administrator can add/remove/update additional compliance rules. These updates flow through a devops-based AWS CodePipeline deployment to the managed accounts, resulting in updated Service Catalog Portfolios in the managed accounts. The Service Catalog Portfolio consists of Service Catalog Products containing AWS Config rules and integrated AWS Systems Manager (SSM) remediation runbooks. The solution is set up in 1 step with Infrastructure as Code (IaC) automation utilizing AWS CloudFormation.

Solution overview

This solution is deployed in a multi-account organization in AWS Organizations. The organization consists of a shared services account. In this account, CCOE administrators deploy shared services like the AWS CodePipeline, as well as other services such as logging, networking, and monitoring. This post refers to all other accounts in the organization where the Config rules with integrated Systems Manager remediation runbooks are deployed as managed accounts. From a hub-and-spoke perspective, the shared services account is the hub account, and the managed accounts are spoke accounts where the Config rules with integrated Systems Manager remediation runbooks get deployed.

Personas

We recommend delineating the administration tasks for this solution into two distinct personas. The CCOE AWS administrator is responsible for the centralized deployment of AWS services and automation within the entire organization. The end user/AWS administrator is responsible for the AWS services administration and deployment for their specific application within the organization. Here’s a description of the administration functions conducted by these two personas for this solution:

CCOE AWS administrator

The CCOE administrator conducts initial setup from the shared services account. After setup, the DevOps infrastructure (AWS CodePipeline) is provisioned in the shared services account, and then an AWS Service Catalog Portfolio with Config rules and integrated remediation runbooks is provisioned in the managed accounts.

The CCOE administrator also conducts code updates from the local Git repository. The administrator checks in updated Config rules, remediation runbook templates, and (optionally) an updated buildspec.yaml file. The updated code flows via AWS CodePipeline in the AWS shared services account and updates the AWS Service Catalog Portfolio in the managed accounts.

End user / AWS administrator in the managed accounts

The end user launches the Config rules and integrated remediation runbooks from the AWS Service Catalog console in the managed account.

Solution architecture

Figure 1 shows the overall architecture for the automated DevOps-based, multi-account compliance deployment. The solution architecture is comprised of components deployed in the shared services account and those deployed in the managed accounts.

The AWS CodePipeline service in the shared services account consists of two stages: AWS CodeCommit and AWS CodeBuild. The AWS CodeCommit stage is triggered whenever a code update or check-in to the AWS CodeCommit Git Repository occurs. In that case, it downloads the updated code into the Amazon S3 Artifact repository of the AWS CodePipeline. The AWS CodeBuild stage of the AWS CodePipeline copies the updated code to a staging S3 bucket and invokes AWS CloudFormation. Then, AWS CloudFormation creates a StackSet for the AWS Service Catalog portfolio in the shared services account, as well as stack instances for the AWS Service Catalog portfolio in the managed accounts.

The AWS Service Catalog Portfolio in the managed accounts consists of Config rules with integrated SSM remediation runbooks. Reference the following architecture diagram.

Architecture diagram displaying infrastructure as code components of the shared services account provisioning Service Catalog portfolios in the managed account.

Figure 1: Architecture diagram

Solution components

This solution is available for download with a detailed README. It consists of the following AWS CloudFormation templates:

Walkthrough

This section walks through the prerequisites and steps to set up and deploy this solution. I also explain how to conduct updates.

Prerequisites

As a CCOE AWS administrator signed in to the AWS shared services account, set up the following resources:

  1. Enable AWS Config in the shared service account as well as all of your managed accounts in the organization. Conduct step 1 from the Automate configuration compliance at scale blog post in order to utilize Systems Manager Quick Setup so you can accomplish it with just a few clicks from your console.
  2. Integrate AWS Cloud9 as a local Git repository with AWS CodeCommit as the remote Git repository
    1. Complete Step 1 from this AWS CodeCommit tutorial to create a CodeCommit Git repository. Provide a name for your CodeCommit Git repository (for example “CCOECompliance”)
    2. Using Cloud9 is one of the easiest ways in AWS to set up a local Git repository with CodeCommit as the remote Git repository. Follow these steps to clone and integrate your CodeCommit repository into Cloud9.
    3. On Cloud 9, change directories to your cloned repository and create a subfolder called compliance . Download these files from this solution’s GitHub repo and upload them to your Cloud9 local Git repository. Your local Cloud9 Git repository should now contain the following files in this structure.
  3. In the following files available for download from the solution, substitute the <accountid> parameter with the AWS Account ID of the shared services account. Substitute the <region> parameter with the AWS Region of your shared services account. Substitute the <managedaccount> and <managedregion> parameters each with comma-separated AWS Account IDs and comma-separated AWS Regions of the managed accounts where the solution will be deployed
  4. You are now ready to commit your source code/templates to the remote CodeCommit Git repository. Use standard git commands from your Cloud9 cloned repository’s root folder (where the buildspec.yml file resides) in the Cloud9 console:
    • git add .
    • git commit -m “initial commit”
    • git push origin ‘your branch name‘
  5. Create an S3 staging bucket with this naming convention: s3-configremediations-<accountid>-<region>. Create a folder called compliance in your S3 bucket. These folder names must match the folder names in your local Git repository. Create these folders with any names as long as they are the same used while creating your local Git repository there.

Solution setup

Now you can complete the 1-step set up as a CCOE AWS administrator.

From the shared services AWS account, create a stack in the AWS CloudFormation console to launch the aws-servicecatalog-codepipeline.yml template. The template takes the following parameters:

  • RepositoryName: CodeCommit repository for the Config remediation CloudFormation templates
  • BranchName: Branch in the CodeCommit repository for the Config Remediation CloudFormation templates
  • S3StagingBucketPrefix: S3 Staging Bucket prefix that stages the code copied from CodeCommit. In our case, this is s3-configremediations-<accountid>-<region>

Running the aws-servicecatalog-codepipeline.yml template during the initial setup causes the following to occur.

  1. The template provisions AWS CodePipeline in the AWS Shared Services account.
  2. The AWS CodeCommit stage of AWS CodePipeline downloads the code from the AWS CodeCommit Git repository into the Amazon S3 artifact repository of AWS CodePipeline.
  3. The AWS CodeBuild stage of AWS CodePipeline utilizes the AWS Service Catalog Portfolio template, runs the buildspec.yaml file commands to stage the code in an S3 bucket, and then uses AWS CloudFormation StackSets to launch the aws-configremediations-servicecatalog-portfolio in the managed accounts. This last step creates the AWS Service Catalog Portfolio in the managed accounts with the Config rules and integrated remediation runbooks Service Catalog Products. This also creates an AWS IAM group(EndUserGroup), an AWS IAM role (EndUserRole) for the end user and a launch constraint in the managed accounts. This allows an IAM end user in the managed accounts to log in and directly utilize the AWS Service Catalog console to launch the Config rules with remediation runbooks products.

Validate compliance

Now we will validate the standardized compliance posture that CCOE teams can enforce across managed accounts via AWS Service Catalog.

Step 1: Launch the Service Catalog Product

  1. Log in to the IAM console of the AWS managed account as an AWS administrator, create an IAM user that is a member of the EnduserGroup, and then log out of the managed account.
  2. Next, navigate to the AWS Service Catalog console of the managed account as the IAM end user that was created, navigate to the left sidebar, and choose Products. Select the ‘AWS ConfigRemediations Compliance Product’ product, accept the defaults, and select Launch Product. The Service Catalog product screen will auto refresh until the product has been launched. Select Provisioned Products from the left sidebar in order to validate that the product has been launched and the status shows available.

Step 2: Review provisioned Config rules with attached remediation runbooks

  1. Navigate to the AWS Config Console of the managed account. Several AWS Config rules — specifically ‘cloud-trail-log-file-validation-enabled’, ‘ReleaseElasticIP’, ‘cloud_trail_cloud_watch_logs_enabled’, ‘cmk-backing-key-rotation-enabled’, and ‘cis-iam-password-policy’ — with associated remediation runbooks have been provisioned for you based on the Service Catalog product launch. These Config rules with associated remediations provide continuous compliance for your AWS environment based on the evaluation of these provisioned rules.

Step 3: Verify automated ‘attack’ and validate compliance

  1. This post has provided built-in automation that launches a ‘compliance attack’ by simulating AWS resource misconfigurations.
  2. Log in as an AWS administrator in the managed account.
    1. Check that an AWS CloudTrail called ‘ReinforceTrail’ has been provisioned without log file validation and CloudWatch Logs monitoring enabled. Check that an AWS KMS Customer Master Key with description ‘Test Key Rotation’ has key rotation disabled, and that there’s an Elastic IP address with an unassociated EC2 instance.
    2. Navigate to the AWS Systems Manager console, select Automation in the left panel, and then select Since we have launched the Service Catalog Product in the managed account, our standardized compliance posture triggers AWS Config Remediations that leverage our integrated and custom AWS Systems Manager remediation runbooks. Monitor the Execution status in the Systems Manager Automation console for every automation and verify their successful automation status.
    3. Finally, validate the managed account compliance posture by validating that each misconfigured resource is now in the desired state. Check that the AWS CloudTrail called ‘ReinforceTrail’ has both log file validated and CloudWatch Logs monitoring enabled. Check that an AWS KMS Customer Master Key with description ‘Test Key Rotation’ has key rotation enabled and that the unassociated Elastic IP address has been removed.

Conduct updates

The CCOE administrator can add/remove/update additional compliance rules. These updates flow through the pipeline to the managed accounts, which result in updated Service Catalog Portfolios in the managed accounts. Then, the managed account administrators simply launch updated Service Catalog Products. This is how CCOE teams continuously enforce standardized compliance posture across managed accounts via AWS Service Catalog.

In my example, the CCOE administrator plans to update the compliance posture of managed accounts by adding an additional compliance rule that detects unrestricted SSH access and then remediates it automatically by removing such inbound rules from EC2 Security Groups in the managed accounts.

Step 1: Conduct the update

  1. In your local Code9 GitHub repository:
    1. Navigate to the ‘compliance’ folder where you placed the aws-servicecatalog-configremediations-v1.yml file and replace it with the aws-servicecatalog-configremediations-v2.yml
    2. Update the aws-servicecatalog-prescriptivecompliance.yml CloudFormation template in the root folder. Find the Resources section in this template, specifically the ‘ConfigRemediationsCompliance’ Resource. Update the ProvisioningArtifactParameters section with the following:
      • Description: This is version 2.0 of the AWS ConfigRemediations Compliance Product
      • Name: Version – 2.0
      • Info: LoadTemplateFromURL: !Sub “${S3StagingBucketURL}compliance/aws-servicecatalog-configremediations-v2.yml”
    3. Make a copy of the existing ‘buildspec.yml’ file. Rename ‘buildspec-updates.yml’ to buildspec.yml.
    4. Commit the changes from your local GitHub repository by utilizing git bash commands and update the remote CodeCommit Repository:
      • git add .
      • git commit -m “updated commit”
      • git push origin ‘your branch name‘

Step 2: Validate the update

Conduct the following to validate the update:

  1. Log in to the AWS CodePipeline console of the AWS Shared Services account.
  2. Validate that the AWS CodePipeline gets triggered by showing a status of In Progress.
  3. Wait for the AWS CodePipeline to complete successfully by verifying that each AWS CodePipeline stage shows a Success
  4. Navigate to the AWS CloudFormation console. From the left panel, choose StackSets and then Operations.
  5. Verify that the ‘aws-servicecatalog-prescriptivecompliance’ StackSet is updated successfully by verifying that the deployment status shows

So far, this process does the following: the updated buildspec.yaml file invokes update-stackset on AWS CloudFormation instead of create-stackset. After a code change has been made in the local repository, the changes are committed and pushed to a remote AWS CodeCommit repository in a bash terminal via standard git commands. AWS CodePipeline automatically recognizes the commit and updates the Config rules with integrated remediation runbooks products in AWS Service Catalog for the managed accounts. The automated pipeline for managing AWS Service Catalog is now set up, and it responds to template changes via a git commit.

Step 3: Launch an ‘attack’ and validate compliance

  1. Log in as an AWS administrator in the managed account:
    1. Conduct a misconfiguration by providing unrestricted SSH access to an EC2 Security Group. Choose Security Groups in the Amazon EC2 console. Choose the Security group ID of the default VPC. Choose Edit inbound rules. Select Add rule. SelectSSH as the Type and 0.0.0.0/0 as the
    2. Navigate to the AWS Systems Manager console, select Automation in the left panel, and then select Since we have launched an updated Service Catalog Product in the managed account, our standardized compliance posture triggers a new AWS Config Managed rule with an integrated AWS Systems Manager remediation runbook — ‘RestrictDefaultSecurityGroup’. This provides automated detection and automated remediation for the PCI.EC2.2 compliance violation — ‘VPC default security group should prohibit inbound and outbound traffic’. Monitor the Execution status in the Systems Manager Automation console for the automation execution, and verify its successful automation status.
    3. Finally, navigate back to the EC2 console, choose Security Groups, and choose the Security group ID of the default VPC. Validate the managed account compliance posture by verifying that the inbound rule that provided unrestricted SSH access to this EC2 Security Group has been removed.

Cleanup

To clean up your account after trying the solution outlined in this blog post, conduct the following:

  1. Log in as an end user to the AWS Service Catalog console of the AWS managed account.
  2. Navigate to the Provisioned product list page.
  3. Select the provisioned product that was deployed, AWS ConfigRemediations Compliance Product product, and then choose Terminate provisioned product.
  4. Verify the provisioned product that you want to delete, and then choose Terminate.

Conclusion

This post provides a solution that utilizes AWS Service Catalog to enable CCOE administrators to enforce a standardized compliance posture across every managed account in the organization. The solution combines DevOps automation and IaC with an easy 1-step setup via several AWS services, specifically CodePipeline and Config integrated with custom Systems Manager remediation runbooks. This solution enables CCOE AWS administrators and infrastructure engineers to standardize the management and deployment of compliance in their organization.

About the author

A picture of Kanishk Mahajan

Kanishk Mahajan

Kanishk Mahajan has been leading AWS Cloud transformation, solution architecture, and delivery teams for customers for several years. His areas of specialty include management and governance, migrations and modernizations, and security and compliance. He is a Technical Field Community (TFC) member at AWS in each of those domains.