How to Monitor AWS Account Configuration Changes and API Calls to Amazon EC2 Security Groups

You can use AWS security controls to detect and mitigate risks to your AWS resources. The purpose of each security control is defined by its control objective. For example, the control objective of an Amazon VPC security group is to permit only designated traffic to enter or leave a network interface. Let’s say you have an Internet-facing ecommerce website, and your security administrator has determined that only HTTP (TCP port 80) and HTTPS (TCP 443) traffic should be allowed access to the public subnet. As a result, your administrator configures a security group to meet this control objective.

What if, though, someone were to inadvertently change this security group’s rules and enable FTP or other protocols to access the public subnet from any location on the Internet? That expanded access could weaken the security posture of your assets. Consequently, your administrator might need to monitor the integrity of your company’s security controls so that the controls maintain their desired effectiveness.

In this blog post, I explore two methods for detecting unintended changes to VPC security groups. The two methods address not only control objectives but also control failures.

The two methods and when to use them

In this post, Method 1 uses AWS Config to monitor changes to a security group’s configuration as part of an organization’s overall compliance auditing program. Method 1 views a change to a VPC security group as a compliance risk. Use this method when you want to bolster your company’s compliance management.

Method 2 uses AWS CloudTrail and Amazon CloudWatch Events to identify AWS API calls that could change the configurations of VPC security groups. Method 2 views a change to a VPC security group as a potential security incident that should be identified in near real time. Use this method when you want to support your company’s monitoring of security operations.

Both of these methods can be effective parts of a defense-in-depth approach to control monitoring.

Method 1: Use AWS Config to check the configuration of a security group

The first method uses AWS Config, a fully managed service that provides you with an AWS resource inventory, configuration history, and configuration change notifications to help enable security and governance. You can create AWS Config rules that automatically check the configuration of AWS resources that are recorded by AWS Config. For this example, I use a Config rule that is invoked whenever a change is made to a security group. Attach the Config rule to an AWS Lambda function that examines the ingress rules of a security group to see if the group remains in compliance with the rules.

The following Lambda function code defines a list named REQUIRED_PERMISSIONS with elements that represent a protocol, port range, and IP range that together define a security permission. This JSON notation is identical to what you would use when creating a security group with the AWS EC2 authorize-security-group-ingress command.

REQUIRED_PERMISSIONS = [
{
    "IpProtocol" : "tcp",
    "FromPort" : 80,
    "ToPort" : 80,
    "UserIdGroupPairs" : [],
    "IpRanges" : [{"CidrIp" : "0.0.0.0/0"}],
    "PrefixListIds" : []
},
{
    "IpProtocol" : "tcp",
    "FromPort" : 443,
    "ToPort" : 443,
    "UserIdGroupPairs" : [],
    "IpRanges" : [{"CidrIp" : "0.0.0.0/0"}],
    "PrefixListIds" : []
}]

In this function, the first list entry allows TCP port 80 (HTTP) from anywhere on the internet (0.0.0.0/0). The second list entry does the same for TCP port 443 (HTTPS).

The Lambda function then examines the configuration event that was passed to it as a result of the configuration change. The configuration event contains a list variable named ip_permissions that comes from the IpPermissions field of the Config event payload. The list represents the permissions of the security group after the configuration change is made. The function then examines the new permissions to see if they match the permissions listed in the preceding REQUIRED_PERMISSIONS list, as shown in the following code.

authorize_permissions =
    [item for item in REQUIRED_PERMISSIONS
        if item not in ip_permissions]
revoke_permissions =
    [item for item in ip_permissions
        if item not in REQUIRED_PERMISSIONS]

The authorize_permissions list contains the permissions that are in the REQUIRED_PERMISSIONS list that are not in the configuration event’s permission list. Thus, authorize_permissions represents the permissions that must be authorized (added) to the security group to bring it into alignment with the REQUIRED_PERMISSIONS list.

The revoke_permissions list contains the permissions that are in the configuration event’s permission list but not in the REQUIRED_PERMISSIONS list. Therefore, the revoke_permissions list represents the permissions that need to be revoked (removed) from the security group to bring it into alignment with the REQUIRED_PERMISSIONS list.

The function then performs the necessary authorizations and revocations using the authorize_security_group_ingress() and revoke_security_group_ingress() API calls. All of the results are logged by Amazon CloudWatch Logs.

To implement Method 1, follow these steps:

1. Create a Lambda execution role and an AWS Config role.
2. Enable AWS Config to record the configuration of security groups so that AWS Config can monitor security groups for changes to their configurations. The following screenshot shows an example of the settings.
   
3. Create a security group that enables ingress TCP ports 80, 443, 465, and 993. You will use this security group only for the purposes of this blog post. The group should appear as the following screenshot shows.
   
4. Create the Python Lambda function using the code from the AWS Config rules library on GitHub.
5. Create the AWS Config rule using the Lambda function you created in Step 4. For Trigger type, choose Configuration changes. For Scope of changes, choose EC2: SecurityGroup, and then type the ID of the security group you created in Step 3. The following screenshot shows these configuration settings.
   
6. Run the Config rule. This will queue the rule for execution, and the rule should run to completion in about 10 minutes.
7. Check the security group you created in Step 3. You should see that only ingress TCP ports 80 and 443 remain, as shown in the following screenshot.
   

The use of AWS Config in Method 1 allows for the configuration of a security group to be tracked along with other AWS resources. Changes to the security group’s configuration are reported during the next Config compliance evaluation, typically within 10 minutes. The notifications of changes to your security groups can be used to support your organization’s compliance management program.

Method 2: Use CloudTrail and CloudWatch Events to monitor API calls

In the first method, I approached the solution from the perspective of the object being changed—in this case, a security group. In the second method, I will focus on the “actors,” namely the API calls that may attempt to change the security group. By using CloudTrail with CloudWatch Events, you can invoke a Lambda function when specific API calls such as authorize_security_group_ingress() and revoke_security_group_ingress() are made. The API call event payload contains an IpPermission list that you can scan, as you did in Method 1.

When you create the Lambda function for CloudWatch Events, you can create an event selector, which functions as a filter for the Lambda function so that the function is only invoked for specific events. This approach has two advantages: it avoids unnecessary calls to Lambda, and you can simplify your Python code to handle only the desired events. Consider the following event selector.

{
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventSource": [
      "ec2.amazonaws.com"
    ],
    "eventName": [
      "AuthorizeSecurityGroupIngress",
      "RevokeSecurityGroupIngress"
    ],
    "requestParameters": {
      "groupId": [
        "sg-abc12345"
      ]
    }
  }
}

This selector looks for CloudTrail API events (AWS API Call via CloudTrail) that involve the two API calls we previously discussed with the security group you wish to examine (in this case, sg-abc12345).

The function code for Method 2 differs from that of Method 1 in one important way: the Lambda function for Method 2 does not make any changes to the security group. The reason for this difference is that the code would need to use the same APIs to adjust the security group that triggered the Lambda function in the first place, potentially resulting in recursion.

To implement Method 2, follow these steps:

1. Create a Lambda execution role and policy.
2. Create the Python Lambda function using the code from AWS Labs.
3. Create a security group with no ingress permissions (meaning no TCP or UDP ports). Use this security group only for the purposes of this blog post, and do not attach this group to a resource.
4. Enable CloudTrail.
5. Create a CloudWatch Events rule that is triggered by API calls. Use an event selector as described previously in this post as well as the Lambda function you created in Step 2.
6. Add a rule to the security group you created in Step 3 of Method 2 to allow TCP port 445 inbound. TCP port 445 has no special significance; any TCP ports other than 80 and 443 would work.
   
7. After a few minutes, you should see a message in CloudWatch Logs telling you that TCP ports 80 and 443 must be authorized and that TCP port 445 must be revoked, as shown in the following screenshot.
   

The use of CloudTrail and CloudWatch Events in Method 2 allows for the near real-time detection of API calls that could change the configuration of a VPC security group. The notifications of changes are posted to CloudWatch Logs, providing useful information to your organization’s security operations management program.

Summary

You can layer Config, CloudTrail, and CloudWatch Events on top of Amazon VPC security groups to provide a defense-in-depth approach to security. Though VPC security groups provide critical filtering capabilities, Config rules, CloudTrail, and CloudWatch Events take the protection to a deeper level by monitoring security groups and notifying you of potentially unintended changes.

Whether you use Method 1 or Method 2 depends on your goals. If you are focused on compliance management, Method 1 enables you to incorporate the configuration of security groups into your organization’s compliance management program. If your concern is more about incident detection, Method 2 offers a faster way to detect changes to a security group’s configuration. Both methods can help you add security to your AWS infrastructure.

If you have comments about this blog post, submit them in the “Comments” section below. If you have questions about implementing this post’s two methods, start a new thread on the VPC forum.

– Jeff

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.