AWS Cloud Operations Blog

Monitor tag changes on AWS resources with serverless workflows and Amazon CloudWatch Events

Introduction

Amazon CloudWatch Events now supports tag changes on AWS resources. Using this new CloudWatch Event type, you can build CloudWatch event rules to match tag changes and route them to one or more targets like an AWS Lambda function to trigger automated workflows. In this blog post, I’ll provide an example for using AWS Lambda to build cost-effective serverless solutions to securely process tag changes to AWS resources.

Tag changes generate Amazon CloudWatch Events

Amazon CloudWatch Events delivers a near real-time stream of system events that describe changes in AWS resources. Many AWS resources support tags, which are custom, user-generated attributes to easily organize and categorize AWS resources. Common use cases for tags are cost allocation categorization, access-control security, and automation. With Amazon CloudWatch Events, you can monitor for changes to tags and track the tag state on AWS resources. Previously to achieve similar functionality, you might have continuously polled APIs and orchestrated multiple calls. Now, any change to a tag including individual service APIs, the Resource Groups Tag Editor, and the Resource Groups Tagging API will trigger the tag change on resource event. The following code is an example of a new CloudWatch event.

{
  "version": "0",
  "id": "bddcf1d6-0251-35a1-aab0-adc1fb47c11c",
  "detail-type": "Tag Change on Resource",
  "source": "aws.tag",
  "account": "123456789012",
  "time": "2018-09-18T20:41:38Z",
  "region": "us-east-1",
  "resources": [
    "arn:aws:ec2:us-east-1:123456789012:instance/i-0000000aaaaaaaaaa"
  ],
  "detail": {
    "changed-tag-keys": [
      "a-new-key",
      "an-updated-key",
      "a-deleted-key"
    ],
    "service": "ec2",
    "resource-type": "instance",
    "version": 3,
    "tags": {
      "a-new-key": "tag-value-on-new-key-just-added",
      "an-updated-key": "tag-value-was-just-changed",
      "an-unchanged-key": "tag-value-still-the-same"
    }
  }
}

All CloudWatch Events have the same top-level fields:

  • Version By default, this is set to 0 (zero) in all events.
  • Id – A unique value is generated for every event. This can be helpful in tracing events as they move through rules to targets and are processed.
  • detail-type – Identifies, in combination with the source field, the fields and values that appear in the detail field.
  • Source – Identifies the service that sourced the event. The source for tag changes is ‘aws.tag’.
  • Time – The event timestamp.
  • Region – Identifies the AWS Region where the event originated.
  • Resources – This JSON array contains Amazon Resource Names (ARNs) that identify resources that are involved in the event. This is the resource where tags have changed.
  • Detail – A JSON object, whose content is different depending on event type. For tag change on resource, the following detailed fields are included:
    • Changed-tag-keys are the tag keys that have been changed in this event.
    • Service is the service that the resource belongs to, in this example the service is EC2.
    • Resource-type is the type of resource of the service, in this example it is an EC2 instance.
    • Version is the version of the tag set. The version starts at 1 and increments when tags are changed. You can use the version to verify the order of tag change events.
    • Tags is the current tags on the resource after the change.

You can find more information at Event Patterns in CloudWatch Events Documentation.

CloudWatch Events natively lets you create rules that match specific event patterns based on the different fields. We’ll demonstrate this through a basic example. We’ll show how an EC2 instance is automatically stopped if a specified tag isn’t associated with the EC2 instance. We’ll use the Amazon CloudWatch Event fields to create a pattern to match the tag events for EC2 instance and trigger an AWS Lambda function.

AWS Lambda and serverless

AWS Lambda is a serverless paradigm that lets you execute code in the cloud, without the overhead of provisioning or managing servers. You can execute code only when it’s needed, without thinking about servers, and pay only for the exact compute time you use. Even though it’s called serverless, it doesn’t mean there are no servers. It means you don’t have to provision, configure, or manage the servers that are used to run your code, which can save you a lot of effort. To learn more about AWS Lambda, visit AWS Lambda Product Overview.

Example: Monitoring changes to tags and automatically stopping EC2 instances missing required tags

As customers manage a growing pool of AWS resources and AWS accounts, tags make it easier to properly categorize resources. Tags are commonly used for critical use cases such as cost allocation and security. When you manage AWS resources, you’ll want all your resources consistently tagged. Often, when a resource is provisioned, it gets all the appropriate tags. However, a later process can result in a tag change that drifts from the corporate tag policy. By monitoring changes to tags, your team can keep an eye on the tag drift and immediately respond if necessary. You will have more confidence that all resources are categorized appropriately.

Taking it one step further, we will demonstrate how to monitor for tag changes on EC2 instances in order to verify that the EC2 instance continues to have the required tags. If the EC2 instance has its tags change and no longer has the required tags, we will use serverless technology to automatically shut down the EC2 instance. Why would I do this? I would do this to ensure that all resources are tagged according to my corporate tag policy, for example, to make it easier to allocate costs.

In this example, we will be using JavaScript and Node.js 6.10. We will work in the AWS Region us-east-1. We will use the anonymous account ID 123456789012.

Step 1. Set up AWS Lambda

In the AWS Lambda console, go to Create function and select Author from scratch. We will create a JavaScript function called AutoEC2Termination using the auto-stop-ec2-role as shown in the following screenshot. You don’t need to select a policy template.

Remove the default code from ‘index.js’ and paste the following code. Save.

// Load the SDK for JavaScript
var AWS = require('aws-sdk');
// Set the region
AWS.config.update({region: 'us-east-1'});
// Create EC2 service object
ec2 = new AWS.EC2({apiVersion: '2016-11-15'});

exports.handler = (event, context, callback) => {
  var detail = event.detail

  // no action if this event is not about ec2 instance
  if (!(detail["service"] === "ec2" && detail["resource-type"] === "instance")) return

  var tags = detail["tags"]

  console.log(tags)

  // if associated tags not contain the expected tag pair
  if (!tags.hasOwnProperty("valid-key") || tags["valid-key"] !== "valid-value"){
    console.log("This is an invalid instance.")

    var resourceSplit = event.resources[0].split("/")
    var params = {
      InstanceIds: [resourceSplit[resourceSplit.length - 1]], // extract the last part of resource name as instance id
      DryRun: true
    };

    // call EC2 to stop the selected instances
    ec2.stopInstances(params, function(err, data) {
      if (err && err.code === 'DryRunOperation') {
        params.DryRun = false;
        ec2.stopInstances(params, function(err, data) {
          if (err) {
            callback(err, "fail");
          } else if (data) {
            console.log("Success", data.StoppingInstances);
            callback(null, "Success");
          }
        });
      } else {
        callback(err)
      }
    });
  } else {
    console.log("This is a good instance.")
    callback(null, "no action");
  }
};

This Lambda function checks the tags of an EC2 instance. If the EC2 instance is missing the required tag key “valid-key” or is not using the “valid-value,” then the Lambda function attempts to stop the EC2 instance. You can change this logical check or the tag requirements for your own specific use cases.

Step 2. Set up IAM permissions

We need to grant the permission to stop an EC2 instance to the role “auto-stop-ec2-role.” The minimum permission access required is the action ec2:StopInstances . To create a new IAM policy, see how to Use the New Visual Editor to Create and Modify Your AWS IAM Policies. For help on creating EC2 specific IAM policies, see Amazon EC2: Allows starting or stopping an EC2 Instance and Modifying a Security Group, Programmatically and in the Console.

Step 3: Test Lambda function

Return to the AWS Lambda console. For the Lambda function we created, configure a test event named “SampleTagChangeEventOnEC2” using the JSON data from the previous sample event.

When you choose the Test button, you should see the following error log in the CloudWatch Log console. This demonstrates that your Lambda function has successfully attempted to stop a dummy EC2 instance.

Step 4: Create CloudWatch Events rule

Now we are going to create the CloudWatch Events rule that matches the Tag Change on Resource event and points to our Lambda function. In the CloudWatch Events console, choose Create Rule and paste the following code.

{
  "source": [
    "aws.tag"
  ],
  "detail-type": [
    "Tag Change on Resource"
  ],
  "detail": {
    "service": [
      "ec2"
    ],
    "resource-type": [
      "instance"
    ]
  }
}

This is a fine-grained rule that matches against ‘Tag Change on Resource’ events for EC2 instances. Now we need to add our Lambda function as the target. On the right side of the screen select AutoEC2Termination, which is the Lambda function we previously created. Then choose Configure details.

Give this new rule the name “ec2-instance-rule” and create rule.

Step 5: Grant CloudWatch Events permissions to invoke the Lambda function

Use the CLI to grant CloudWatch Events permission to invoke our Lambda function by typing the following command.

aws lambda add-permission --statement-id "TrustCWEToInvokeAutoEC2Termination" \
--action "lambda:InvokeFunction" \
--principal "events.amazonaws.com" \
--function-name "arn:aws:lambda:us-east-1:123456789012:function:AutoEC2Termination" \
--source-arn "arn:aws:events:us-east-1:123456789012:rule/ec2-instance-rule" \
--region us-east-1

Step 6: You are done!

Test for your final result by creating an EC2 instance with tag foo:bar. You will see the instance stopped automatically and the log message in the CloudWatch Logs console will look like the following:

Success [ { CurrentState: { Code: 64, Name: 'stopping' },
InstanceId: 'i-2222222222222',
PreviousState: { Code: 16, Name: 'running' } } ]

Conclusion

In this example, we demonstrated how to create a CloudWatch Events rule to match against the new tag change on a resource event for EC2 instances. The rule pointed to a Lambda function that automatically shuts down the EC2 instance if it does not have the correct tags.

The Amazon CloudWatch Events support for tag changes on AWS resources opens possibilities to build event-driven automation across many AWS services. Combining this new capability with AWS Lambda provides builders with a set of tools to build serverless solutions that securely access AWS resources, scale on demand, and are cost effective.

Other possible use cases for the new Tag Change on Resource CloudWatch event includes:

  • Trigger a warning if someone accesses your resource from an unusual IP address – Use a tag to store the source IP address of each visitor that accesses your resource. Changes to the tag generates a CloudWatch event that you can use to compare the source IP address to a list of valid IP addresses, and trigger a warning email if source IP address is invalid.
  • Monitor if there have been any changes to your tag-based access control for a resource – If you have set up access to a resource based on users specified in a tag, you can use CloudWatch events generated by any changes to the tag to prompt an audit by your security team.

We welcome your ideas and questions in the AWS Tags developer forum.

About the Author

Qiutong Song is a Software Engineer at AWS. As part of the platform team, he enjoys building highly scalable systems to reduce customer’s pain when using AWS.