Using AWS CloudTrail to propagate tags across related AWS resources – Part 2

AWS allows customers to assign metadata to their AWS resources in the form of tags. Each tag consists of a customer-defined key and an optional value. Tags can make it easier to manage, search for, and filter resources by purpose, owner, environment, or other criteria. AWS tags can be used for many purposes like organizing your AWS resources or for your cost and usage report, automation, IT service management, access control, and security risk management.

Many customers require some or all AWS resources created by an AWS service to add a tag for consistency. This can be a default tag or custom tag for example ‘deployment ID’. For example, you might need to propagate custom tags from an Amazon Aurora Serverless to the VPC endpoints and network interfaces it creates or, when you create an Amazon SageMaker Studio Control Panel, to the Amazon EFS volume and security group it creates.

You can use AWS Tag Editor and AWS Resource Groups to find and add tags after the resources are created. Another option is to automatically propagate tags when the AWS resources are created. In the previous blog post, we shared an example of how you can propagate custom tags automatically to the resources when you create Aurora serverless. In this blog post, we will share an example of how you can re-use the same mechanism to automatically propagate custom tags to resources when you create Sagemaker Studio Control Panel. You can use the same mechanism for other use-cases (for example, to propagate tags upon resource modification or after restoring an AWS resource from a backup).

Overview of solution

AWS CloudTrail helps you enable governance, compliance, and operational and risk auditing of your AWS account. Every action by an AWS service is an API call that is recorded in CloudTrail as an event. In the solution we describe in this post, we create a rule in Amazon EventBridge that triggers an AWS Lambda function. The AWS Lambda function gets a list of tags from the primary resource that we’re launching and applies tags to resources that are associated with it.

Figure 1: Services used in the solution

Solution steps:

1. Find the AWS resources created by an AWS service that you need to tag.
2. Create a rule in EventBridge that is triggered by an API call recorded in CloudTrail.
3. Trigger a Lambda function to get the primary resource and tag, find associated resources, and apply the tag.

Solution walkthrough

We’ll share two examples of how you can tag AWS resources that are created by AWS services in your account.

Prerequisites

To complete the steps in this walkthrough, you need the following:

• An AWS account.
• Basic understanding of Python using Boto3 APIs and AWS CLI

Solution to automatically tag resources when you create SageMaker Studio Control Panel

Complete these steps or use the CloudFormation template to tag EFS volumes or security groups when you create a SageMaker Studio domain.

1. Sign in to the IAM console, choose Policies, and then choose Create Policy.
2. Choose the JSON tab and paste the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowCreateTagsForSecurityGroups",
            "Effect": "Allow",
            "Action": "ec2:CreateTags",
            "Resource": "arn:aws:ec2:*:ACCOUNTNUMBER:security-group/*"
        },
        {
            "Sid": "AllowToCreateTagsAndLogs",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "sagemaker:ListTags",
                "logs:CreateLogGroup",
                "logs:PutLogEvents",
                "elasticfilesystem:CreateTags"
            ],
            "Resource": "*"
        }
    ] 
}

Figure 2: JSON

3. Choose Next: Tags and then choose Next: Review
4. On the Review policy page, enter sagemakerTaggingPolicy for the policy name and then choose Create policy.
5. To create a role for the Lambda function, in the left navigation pane, choose Roles and then choose Create role.
6. On Create role, under Choose a use case, choose Lambda, and then choose Next: Permissions.
7. Search for the policy you just created, choose Next: Tags, and then choose Next: Review.
8. For the put role name, enter sagemakerTagPropagatorRole.
9. Go to the AWS Lambda console and create a function with the following details:
   • Function name: propagateSagemakerTags
   • Runtime: Python 3.8
10. For permissions, choose Use an existing role, and then choose sagemakerTagPropagatorRole from the dropdown.
11. Paste the following code for the Lambda function:

import boto3

#Initializing Sagemaker and EFS Client
sagemaker = boto3.client('sagemaker')
efs = boto3.client('efs')
ec2 = boto3.client('ec2')

def lambda_handler(event, context):
    print(event)
    if (event["detail"]["eventName"] == "CreateFileSystem"):
        #Pulling out domain arn from the cloudwatch event trail
        efsId = event["detail"]["responseElements"]["fileSystemId"]
        for tags in event["detail"]["responseElements"]["tags"]:
            if (tags["key"] == "ManagedByAmazonSageMakerResource"):
                domainArn = tags["value"]
                break
        
        #Retrieving tags from the domain 
        resp = sagemaker.list_tags(ResourceArn=domainArn)
        tags = resp["Tags"]
    
        #Attaching custom tags pulled from domain to particular file system              
        res = efs.create_tags(FileSystemId=efsId, Tags=tags)
    
    elif (event["detail"]["eventName"] == "CreateTags"):
        #Pulling out domain arn from the cloudwatch event trail
        try:
            resourceId = event["detail"]["requestParameters"]["resourcesSet"]["items"][0]["resourceId"]
        except Exception as e:
            print("Error in fetching resourceID: " + str(e))
            return 

        try:
            for tags in event["detail"]["requestParameters"]["tagSet"]["items"]:
                if (tags["key"] == "ManagedByAmazonSageMakerResource"):
                    domainArn = tags["value"]
                    break        
        except Exception as e:
            print("Error fetching Tags: " + str(e))
            return


        #Retrieving tags from the domain 
        try:
            resp = sagemaker.list_tags(ResourceArn=domainArn)
            tags = resp["Tags"]
        except Exception as e:
            print ("Domain Arn not provided")
            return
        
        #Attaching custom tags pulled from domain to security groups  
        try:
            res = ec2.create_tags(Resources=[resourceId], Tags=tags)
        except Exception as e:
            print("Trying to tag resources that does not come under EC2 service")
            return

12. Some resources may take longer to create than the default lambda function timeout of 30 seconds, we will update the timeout period to 1 min. To edit the timeout period, choose the Configuration tab, choose General configuration, and then edit the timeout period to 1 min 00 sec.
13. Go to the Amazon EventBridge console, choose Events, choose Rules, and then choose Create rule.
14. For the rule name, enter tagPropagatorForSageMakerEFS.
15. In Define pattern, choose Event pattern. For Event matching pattern, choose Custom pattern. In the Event pattern field, paste the following, and then choose Save.

{
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "source": [
    "aws.elasticfilesystem"
  ],
  "detail": {
    "eventSource": [
      "elasticfilesystem.amazonaws.com"
    ],
    "eventName": [
      "CreateFileSystem"
    ]
  }
}

16. On Select targets, under Target, choose Lambda function. Under Function, choose propagateSagemakerTags from the dropdown, and then choose Create.
17. In the left navigation pane, choose Events, choose Rules, and then choose Create rule.
18. For the rule name, enter tagPropagatorForSageMakerSG.
19. In Define pattern, choose Event pattern. For Event matching pattern, choose Pre-defined pattern by service. For Service provider, choose AWS. For Service name, choose EC2. For Event type, choose AWS API Call via CloudTrail. Choose Specific operation(s) and then enter CreateTags.
20. On Select targets, under Target, choose Lambda function. Under Function, choose propagateSagemakerTags from the dropdown, and then choose Create.

You can verify that the tags are attached to the EFS volume and security groups. The next time you create a SageMaker Studio domain, the custom tags that you add to SageMaker Studio will be applied automatically to the EFS volume and security group.

Cleaning up

If you created the environment using CloudFormation template, here are the instructions to delete the CloudFormation stack.

Conclusion

In this post, we showed how you can create an automation to propagate tags whenever a resource is created. You can extend this logic to any of your use-case that requires you to tag all the subsequent resources when a new AWS service is created.

About the authors