Enabling Self Service for Cloud Custodian policies on AWS using AWS Service Catalog

Customers are increasingly seeking tools and solutions that can help them achieve their desired outcomes more efficiently and effectively. In the context of cloud management, the need for self-service capabilities has become more pronounced as organizations strive to optimize their cloud resources, improve security, and enhance their overall cloud operations. AWS Service Catalog offers the self-service capability to centrally manage the cloud resources to achieve governance at scale.

Cloud custodian is a cloud agnostic open-source tool that is used by organizations across industry verticals to manage their cloud environments by helping ensure compliance to security policies, tag policies, garbage collection of unused resources, and cost management. Cloud custodian works by defining policies in a YAML file and running the defined policies against AWS account.

Generally, Enterprise AWS customers manage these custodian policies centrally and will leverage CI/CD pipeline to roll out the custodian policies to AWS accounts. This allows customer to ensure AWS accounts provisioned as part of a multi-account environment meet their organization’s compliance requirements. Sometimes customers are looking for ways to make the custodian policies as a self-service tool which will allow application team to choose the desired policy which will make all of their applications compliant and standard.

In this post, we show you how to leverage AWS Service Catalog to manage and enable the self-service capability for the custodian policies in a multi-account environment. This will help you to start thinking on defining compliance requirements for your applications and also accelerate provisioning by leveraging the self-service tool.

Overview of Solution

In this solution an AWS CodeCommit repository is used to host the Cloud custodian policy file and the file containing IAM role permissions necessary for the custodian policy. From the below Figure 1, you can see that, once the above two files are added to the repository by the Portfolio administrators, (1) an AWS CodePipeline gets invoked which will (2) validate the custodian policy, (3) convert the custodian policy to SAM template and (4) uploads the template file to S3 bucket. (5) Then an AWS EventBridge rule triggers the lambda function which creates the AWS Service Catalog product for Cloud custodian policies. The Cloud custodian Service Catalog tool provisions the custodian policies and necessary IAM roles required for those policies.

Figure 1: The Solution architecture for enabling Self Service for custodian policies on AWS

Walkthrough

The infrastructure and automation code for this solution can be found in the following GitHub repository custodian service catalog pipeline repository and below are steps we will be implementing as part of this solution:

1. Deploy the Infrastructure for the Service Catalog automation in Service Catalog Hub account.
2. Adding the custodian policy to the AWS CodeCommit repository.
3. Validate the Service Catalog product creation.
4. Updating the custodian policy.
5. Validate the provisioning and testing of the custodian policy Service Catalog product in the developer account.

Prerequisites

For this walk-through, you should have the following prerequisites:

1. Two AWS accounts as part of AWS Organizations with administrator privileges for the services used in this solution.
   1.  Service Catalog Hub Account – Used to provision the custodian policy as a Service Catalog product in a Portfolio.
   2. Developer account – End-user to launch the Service Catalog product.
2. Ensure Trusted Access is Enabled for Service Catalog. Review the steps to enable trusted access for Service Catalog.
3. Ensure the Service Catalog Hub Account is enabled as Delegated administrator account for AWS Service Catalog in the Organizations Management Account. Run the below CLI command to register an account as delegated admin from the Organizations Management Account.

   aws organizations register-delegated-administrator \--account-id service-catalog-account-id \--service-principal servicecatalog.amazonaws.com

4. Run the below CLI command to add a custom policy that allows delegated admin account to query accounts within the Organization (replace {service catalog account} with the 12-digit Account ID). The command should be run on the Organization’s Management Account.

   This command requires CLI 2.10+

   aws organizations put-resource-policy \ --content '{"Version": "2012-10-17","Statement": [{"Sid": "Statement","Effect": "Allow","Principal": {"AWS": "arn:aws:iam::'"${service_catalog_account_ID}"':root"},"Action": ["organizations:ListDelegatedAdministrators","organizations:ListParents","organizations:ListChildren","organizations:DescribeAccount"],"Resource": "*"}]}'
   

5. The solution requires Service catalog portfolio provisioned in the Service Catalog delegated admin account with below features:
   1. Portfolio sharing enabled for entire AWS Organization or Specific AWS Organization Unit to which the Service Catalog product to be made available.
   2. Portfolio access to be provided to necessary IAM identity to launch the Service Catalog product.
6. An IAM Role with launch constraint for CloudFormation product permissions, and permissions to provision custodian resources should exist in each account to which the Service Catalog product will be made available. This role which will be associated to the Service Catalog product as a Launch Constraint role.

For this solution walkthrough, we are provisioning the portfolio and launch constrain role from the templates present in the repository. The Portfolio provisioned as CloudFormation stack in the Service Catalog Hub Account is shared across the AWS Organization and is provided access for the AWS IAM Identity Center Identity “AWSAdministratorAccess”. The Launch role is provisioned as a Service Managed CloudFormation Stack Set to the Service Catalog Hub Account and Developer Account. This role has the permissions to provision Lambda and EventBridge rule – which are the resources that will be provisioned part of the example custodian policy in this blog.

Deployment Steps

Step 1: Deploy the Infrastructure for the Service Catalog Automation

In this step, you will provision the infrastructure for the service catalog automation as a CloudFormation stack in the Service Catalog Hub Account which has the Service Catalog portfolio provisioned.

1. Log in to the Service Catalog Hub account and select the AWS Region in which custodian Hub Portfolio is provisioned.
2. Navigate to CloudFormation console and create the stack using custodian_automation_pipeline.yaml template.
3. Enter the Stack name. In our example, we use custodian-Service-Catalog-Infra. Under the Parameters section, enter values for the following parameters:
   1. pPortfolioProviderName – The Team that is offering the custodian product to the end users in the Organization.
   2. pSCTemplateS3Bucket – The s3 Bucket that will hold the CloudFormation template for the custodian Service Catalog product.
   3. pSCTemplateS3LoggingBucket – The S3 Bucket that will be the logging bucket for the custodian policy bucket.
   4. pSCLaunchConstraintRoleName – The Launch Constraint role for the provisioning of Service Catalog products. Note: This role should be present in Service Catalog Hub Account and the Developer Account in which the product will be provisioned.
   5. pSCPortfolioName – The Portfolio that will hold the custodian Service Catalog products.
   6. pPipelineName – Name of the pipeline that manages the product deployment of custodian policies.
   7. pCodeCommitRepoName – Name of the AWS CodeCommit repository that will contain the custodian policy and permission files for the role.
   8. pCodepipelineArtifactBucket – The S3 bucket that will hold the CodePipeline artifacts.
   9. pOrganizationID – The Organization ID to allow GetObject on template S3 Bucket for custodian policy.

      Figure 2: The Pipeline Infrastructure CloudFormation Parameters

4. On the Review page, validate the parameters and check the box I acknowledge that AWS CloudFormation might create IAM. Then select Submit.
5. When the CloudFormation stack status changes to CREATE_COMPLETE, navigate to the Resources tab and validate that all the resources have been created successfully.

NOTE: If you’re leveraging the existing portfolio, the Lambda Role “ExecutionRoleForCustodianProductCreation” should be provided access to the portfolio.

Step 2: Adding the Custodian policy to the AWS CodeCommit Repository

In this step, you will create a new Cloud custodian policy that will be available to end users via AWS Service Catalog. For this solution walkthrough, we will provision a custodian policy that terminates the EC2 instance that are provisioned with un-encrypted EBS volumes.

NOTE: The solution doesn’t support custodian policy in pull mode.

1. 1. Clone the GitHub repository by running the following AWS Command Line Interface (AWS CLI) commands from a terminal window.

      git clone https://github.com/aws-samples/self-service-cloud-custodian-policies-using-aws-service-catalog.git custodian-repo
      cd custodian-repo

   2. Clone and copy all the codecommit_files from the custodian service catalog pipeline repository to the custodian-repo created in the previous step.

Figure 3: The Folder Structure after copying the files.

1. 3. Create a sub folder in the cc_policies folder for each custodian policy you will be adding to the repository. Each folder is a separate Cloud custodian AWS Service Catalog product and folder name will be used for the Product name. The folder should contain custodian policy YAML file and permissions.yml that will contain permissions for the Lambda Role. For this walkthrough, we will create a custodian policy that will prevent the provisioning of EC2 instance with unencrypted EBS volumes. The files for provisioning the custodian policy “encrypted-volumes” is provided in the repository.

Custodian policy:

policies:
- name: ec2-require-encrypted-volumes
  resource: aws.ec2
  description: Provision a lambda and cloud watch event target that looks at all new instances and terminates those with unencrypted volumes
  mode:
    type: cloudtrail
    role: custodian-lambda-role-encrypted-volumes
    events:
      - RunInstances
  filters:
    - type: ebs
      key: Encrypted
      value: false
  actions:
    - terminate

Permissions:

PolicyName: "encrypted-volumes"
Permissions:
 Action:
 "ec2:Describe*"
- "ec2:CreateTags"
- "ec2:DeleteTags"
- "ec2:TerminateInstances"
Resource:
- "*"

NOTE: The Role used in the custodian policy should be in the format custodian-lambda-role-< PolicyName>

Figure 4: The Folder Structure after adding the custodian policy.

1. 4. Add the code to the repository by running the following commands.

git add .
git commit -m "Adding the encrypted volumes custodian policy with the initial buildspec files"
git push

1. Once the changes are pushed, deployment takes about 3-5 minutes because of the CodePipelines and Lambda automation for Service Catalog product creation. Check the progress on the AWS CodePipeline Console on your Service Catalog Hub Account.

Step 3: Validate the Service Catalog product Creation

Once the product creation CodePipeline is completed successfully, follow the below steps to validate the Service Catalog product creation.

1. 1. 1. Navigate to Service Catalog service in the Service Catalog Hub Account.
      2. Select the Portfolio which was passed as the parameter to the service catalog automation CloudFormation stack in step 1.

   Figure 5: The Service Catalog product created successfully

   3. Navigate to S3 and review the metadata of the policy file <policy-name>.yml – there are metadata for Version and policy name. In this walkthrough, the S3 object encrypted-volumes.yml will have Version with Value “1” and policy with value “encrypted-volumes”

Figure 6: The S3 Object metadata for the Service Catalog product Template File

Step 4: Updating the Custodian policy

In this step, we will update the existing custodian policy and validate the version update of the Service Catalog product.

1. 1. 1. We will add “auto-tag aws userName on resources” custodian policy to the existing policy Custodian policy.

   - name: ec2-auto-tag-user
     resource: ec2
     mode:
       type: cloudtrail
       role: custodian-lambda-role-encrypted-volumes
       events:
         - RunInstances
     filters:
       - tag:CreatorName: absent
     actions:
       - type: auto-tag-user
         tag: CreatorName
         principal_id_tag: CreatorId

   1. 2. Commit and push the changes to code commit repository by running the following commands.

   git commit -a -m "Updated the encrypted volumes custodian policy with the auto tag custodian policy"
   git push

   1. 3. Once the Service Catalog product pipeline is completed successfully, navigate to Service Catalog console and you will notice the previous version is Deprecated and the new version is made available. Note: Users cannot launch new provisioned products using a deprecated product version. If a provisioned product launched previously uses a now deprecated version, users can only update that provisioned product using the existing version or a new version.

   

   Figure 7: The Service Catalog product version update

   1. 4. Navigate to S3 and review the metadata of the S3 object encrypted-volumes.yml. The Version is updated to “2” as well.

   Figure 8: The S3 Object metadata version update for the Service Catalog product template file

Step 5: Validate the provisioning and testing of the Custodian Policy Service Catalog product in the Developer Account.

In this step, will validate the custodian policy by provisioning the Service Catalog product from the developer account.

1. 1. 1. Login the Developer account which has the Service Catalog Portfolio shared with the IAM Identity that’s provided access.
      2. Navigate to Service Catalog console, select the product “custodian-encrypted-volumes” and Launch Product.

   Figure 9: The Service Catalog product provisioning in the Developer Account

   1. 3. Once the Product provisioned successfully, to test the custodian product, we will launch an EC2 instance which is unencrypted EBS volume.

   Figure 10: Provisioning an EC2 Instance with Unencrypted EBS Volume

   4.  The instance will be terminated at the time of provisioning due the custodian policy Lambda. Note: You could also configure the Cloud custodian policy to alternatively send SNS notifications to your subscribed topic instead of taking a remediating action such as shutting down the instance.

Figure 11: Custodian automation terminating the provisioned unencrypted instance

Cleaning up

If you are following along the blog and would want to terminate the resources to avoid incurring future charges, follow the below steps:

1. 1. Login to the Developer Account to delete the provisioned product.
   2. Login to the Service Catalog Hub Account and terminate the product “custodian-encrypted-volumes” following the deleting products steps.
   3. Empty the object from the custodian S3 bucket, Logging bucket and CodePipeline bucket prior to deleting the stack which provisioned the custodian automation pipeline.
   4. If you have provisioned the Launch Constraint role as part of this blog, delete the stack instances, then delete the StackSets from the AWS CloudFormation console.
   5. If you have provisioned the portfolio as part of the blog, first remove the portfolio share of the portfolio and then delete the CloudFormation stack.

Conclusion

Cloud custodian enables users to keep their AWS environment secure and well managed by defining policies. One of the key benefits of self-service for defining compliance requirements is the ability to empower users and teams to take ownership of their cloud resources. In this blog, we showed you how to enable Self-Service capability for custodian policy leveraging Service Catalog in the multi-account environment. We also walked through deploying a sample custodian policy as Service Catalog product in the Service Catalog Hub Account and provisioning the custodian policy as Service Catalog product from the developer account.

In this post, we deployed couple of example custodian policies as single Service Catalog product but a Service Catalog Portfolio can hold more than Service Catalog product. Review the custodian policies and enable more than one policy as part of the portfolio as separate Service Catalog product based on your organization requirements. If you need help in the design and implementation of such solution, feel free to reach out to us or your local AWS Professional Services team. We hope that you’ve found this post informative and we look forward to hearing how you use this approach.

About Authors