AWS Security Blog

How to Use AWS Config to Monitor for and Respond to Amazon S3 Buckets Allowing Public Access

AWS Config enables continuous monitoring of your AWS resources, making it simple to assess, audit, and record resource configurations and changes. AWS Config does this through the use of rules that define the desired configuration state of your AWS resources. AWS Config provides a number of AWS managed rules that address a wide range of security concerns such as checking if you encrypted your Amazon Elastic Block Store (Amazon EBS) volumes, tagged your resources appropriately, and enabled multi-factor authentication (MFA) for root accounts. You can also create custom rules to codify your compliance requirements through the use of AWS Lambda functions.

In this post we’ll show you how to use AWS Config to monitor our Amazon Simple Storage Service (S3) bucket ACLs and policies for violations which allow public read or public write access. If AWS Config finds a policy violation, we’ll have it trigger an Amazon CloudWatch Event rule to trigger an AWS Lambda function which either corrects the S3 bucket ACL, or notifies you via Amazon Simple Notification Service (Amazon SNS) that the policy is in violation and allows public read or public write access. We’ll show you how to do this in five main steps.

  1. Enable AWS Config to monitor Amazon S3 bucket ACLs and policies for compliance violations.
  2. Create an IAM Role and Policy that grants a Lambda function permissions to read S3 bucket policies and send alerts through SNS.
  3. Create and configure a CloudWatch Events rule that triggers the Lambda function when AWS Config detects an S3 bucket ACL or policy violation.
  4. Create a Lambda function that uses the IAM role to review S3 bucket ACLs and policies, correct the ACLs, and notify your team of out-of-compliance policies.
  5. Verify the monitoring solution.

Note: This post assumes your compliance policies require the buckets you monitor not allow public read or write access. If you have intentionally open buckets serving static content, for example, you can use this post as a jumping-off point for a solution tailored to your needs.

At the end of this post, we provide an AWS CloudFormation template that implements the solution outlined. The template enables you to deploy the solution in multiple regions quickly.

Important: The use of some of the resources deployed, including those deployed using the provided CloudFormation template, will incur costs as long as they are in use. AWS Config Rules incur costs in each region they are active.

Architecture

Here’s an architecture diagram of what we’ll implement:
 

Architecture diagram

Figure 1: Architecture diagram


 

Step 1: Enable AWS Config and Amazon S3 Bucket monitoring

The following steps demonstrate how to set up AWS Config to monitor Amazon S3 buckets.

  1. Sign into the AWS Management Console and open the AWS Config console.
  2. If this is your first time using AWS Config, select Get started. If you’ve already used AWS Config, select Settings.
  3. In the Settings page, under Resource types to record, clear the All resources checkbox. In the Specific types list, select Bucket under S3.
     
    The Settings dialog box showing the "Specific types" list

    Figure 2: The Settings dialog box showing the “Specific types” list


     
  4. Choose the Amazon S3 bucket for storing configuration history and snapshots. We’ll create a new Amazon S3 bucket.
     
    Creating an S3 bucket

    Figure 3: Creating an S3 bucket


     

    1. If you prefer to use an existing Amazon S3 bucket in your account, select the Choose a bucket from your account radio button and, using the dropdown, select an existing bucket.
       
      Selecting an existing S3 bucket

      Figure 4: Selecting an existing S3 bucket


       
  5. Under Amazon SNS topic, check the box next to Stream configuration changes and notifications to an Amazon SNS topic, and then select the radio button to Create a topic.
    1. Alternatively, you can choose a topic that you have previously created and subscribed to.
       
      Selecting a topic that you've previously created and subscribed to

      Figure 5: Selecting a topic that you’ve previously created and subscribed to


       
    2. If you created a new SNS topic you need to subscribe to it to receive notifications. We’ll cover this in a later step.
  6. Under AWS Config role, choose Create a role (unless you already have a role you want to use). We’re using the auto-suggested role name.
     
    Creating a role

    Figure 6: Creating a role


     
  7. Select Next.
  8. Configure Amazon S3 bucket monitoring rules:
    1. On the AWS Config rules page, search for S3 and choose the s3-bucket-publice-read-prohibited and s3-bucket-public-write-prohibited rules, then click Next.
       
      AWS Config rules dialog

      Figure 7: AWS Config rules dialog


       
    2. On the Review page, select Confirm. AWS Config is now analyzing your Amazon S3 buckets, capturing their current configurations, and evaluating the configurations against the rules we selected.
  9. If you created a new Amazon SNS topic, open the Amazon SNS Management Console and locate the topic you created:
     
    Amazon SNS topic list

    Figure 8: Amazon SNS topic list


     
  10. Copy the ARN of the topic (the string that begins with arn:) because you’ll need it in a later step.
  11. Select the checkbox next to the topic, and then, under the Actions menu, select Subscribe to topic.
  12. Select Email as the protocol, enter your email address, and then select Create subscription.
  13. After several minutes, you’ll receive an email asking you to confirm your subscription for notifications for this topic. Select the link to confirm the subscription.

Step 2: Create a Role for Lambda

Our Lambda will need permissions that enable it to inspect and modify Amazon S3 bucket ACLs and policies, log to CloudWatch Logs, and publishing to an Amazon SNS topic. We’ll now set up a custom AWS Identity and Access Management (IAM) policy and role to support these actions and assign them to the Lambda function we’ll create in the next section.

  1. In the AWS Management Console, under Services, select IAM to access the IAM Console.
  2. Create a policy with the following permissions, or copy the following policy: 
    
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SNSPublish",
            "Effect": "Allow",
            "Action": [
                "sns:Publish"
            ],
            "Resource": "*"
        },
        {
            "Sid": "S3GetBucketACLandPolicy",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketAcl",
                "s3:GetBucketPolicy"
            ],
            "Resource": "*"
        },
        {
            "Sid": "S3PutBucketACLAccess",
            "Effect": "Allow",
            "Action": "s3:PutBucketAcl",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Sid": "LambdaBasicExecutionAccess",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}
  3. Create a role for your Lambda function:
    1. Select Lambda from the list of services that will use this role.
    2. Select the check box next to the policy you created previously, and then select Next: Review
    3. Name your role, give it a description, and then select Create Role. In this example, we’re naming the role LambdaS3PolicySecuringRole.

Step 3: Create and Configure a CloudWatch Rule

In this section, we’ll create a CloudWatch Rule to trigger the Lambda function when AWS Config determines that your Amazon S3 buckets are non-compliant.

  1. In the AWS Management Console, under Services, select CloudWatch.
  2. On the left-hand side, under Events, select Rules.
  3. Click Create rule.
  4. In Step 1: Create rule, under Event Source, select the dropdown list and select Build custom event pattern.
  5. Copy the following pattern and paste it into the text box: 
    
{
  "source": [
    "aws.config"
  ],
  "detail": {
    "requestParameters": {
      "evaluations": {
        "complianceType": [
          "NON_COMPLIANT"
        ]
      }
    },
    "additionalEventData": {
      "managedRuleIdentifier": [
        "S3_BUCKET_PUBLIC_READ_PROHIBITED",
        "S3_BUCKET_PUBLIC_WRITE_PROHIBITED"
      ]
    }
  }
}

     
    The pattern matches events generated by AWS Config when it checks the Amazon S3 bucket for public accessibility.

  6. We’ll add a Lambda target later. For now, select your Amazon SNS topic created earlier, and then select Configure details.
     
    The "Create rule" dialog

    Figure 9: The “Create rule” dialog


     
  7. Give your rule a name and description. For this example, we’ll name ours AWSConfigFoundOpenBucket
  8. Click Create rule.

Step 4: Create a Lambda Function

In this section, we’ll create a new Lambda function to examine an Amazon S3 bucket’s ACL and bucket policy. If the bucket ACL is found to allow public access, the Lambda function overwrites it to be private. If a bucket policy is found, the Lambda function creates an SNS message, puts the policy in the message body, and publishes it to the Amazon SNS topic we created. Bucket policies can be complex, and overwriting your policy may cause unexpected loss of access, so this Lambda function doesn’t attempt to alter your policy in any way.

  1. Get the ARN of the Amazon SNS topic created earlier.
  2. In the AWS Management Console, under Services, select Lambda to go to the Lambda Console.
  3. From the Dashboard, select Create Function. Or, if you were taken directly to the Functions page, select the Create Function button in the upper-right.
  4. On the Create function page:
    1. Choose Author from scratch.
    2. Provide a name for the function. We’re using AWSConfigOpenAccessResponder.
    3. The Lambda function we’ve written is Python 3.6 compatible, so in the Runtime dropdown list, select Python 3.6.
    4. Under Role, select Choose an existing role. Select the role you created in the previous section, and then select Create function.
       
      The "Create function" dialog

      Figure 10: The “Create function” dialog


       
  5. We’ll now add a CloudWatch Event based on the rule we created earlier.
    1. In the Add triggers section, select CloudWatch Events. A CloudWatch Events box should appear connected to the left side of the Lambda Function and have a note that states Configuration required.
       
      CloudWatch Events in the "Add triggers" section

      Figure 11: CloudWatch Events in the “Add triggers” section


       
    2. From the Rule dropdown box, choose the rule you created earlier, and then select Add.
  6. Scroll up to the Designer section and select the name of your Lambda function.
  7. Delete the default code and paste in the following code: 
    
import boto3
from botocore.exceptions import ClientError
import json
import os

ACL_RD_WARNING = "The S3 bucket ACL allows public read access."
PLCY_RD_WARNING = "The S3 bucket policy allows public read access."
ACL_WRT_WARNING = "The S3 bucket ACL allows public write access."
PLCY_WRT_WARNING = "The S3 bucket policy allows public write access."
RD_COMBO_WARNING = ACL_RD_WARNING + PLCY_RD_WARNING
WRT_COMBO_WARNING = ACL_WRT_WARNING + PLCY_WRT_WARNING

def policyNotifier(bucketName, s3client):
    try:
        bucketPolicy = s3client.get_bucket_policy(Bucket = bucketName)
        # notify that the bucket policy may need to be reviewed due to security concerns
        sns = boto3.client('sns')
        subject = "Potential compliance violation in " + bucketName + " bucket policy"
        message = "Potential bucket policy compliance violation. Please review: " + json.dumps(bucketPolicy['Policy'])
        # send SNS message with warning and bucket policy
        response = sns.publish(
            TopicArn = os.environ['TOPIC_ARN'],
            Subject = subject,
            Message = message
        )
    except ClientError as e:
        # error caught due to no bucket policy
        print("No bucket policy found; no alert sent.")

def lambda_handler(event, context):
    # instantiate Amazon S3 client
    s3 = boto3.client('s3')
    resource = list(event['detail']['requestParameters']['evaluations'])[0]
    bucketName = resource['complianceResourceId']
    complianceFailure = event['detail']['requestParameters']['evaluations'][0]['annotation']
    if(complianceFailure == ACL_RD_WARNING or complianceFailure == ACL_WRT_WARNING):
        s3.put_bucket_acl(Bucket = bucketName, ACL = 'private')
    elif(complianceFailure == PLCY_RD_WARNING or complianceFailure == PLCY_WRT_WARNING):
        policyNotifier(bucketName, s3)
    elif(complianceFailure == RD_COMBO_WARNING or complianceFailure == WRT_COMBO_WARNING):
        s3.put_bucket_acl(Bucket = bucketName, ACL = 'private')
        policyNotifier(bucketName, s3)
    return 0  # done
  8. Scroll down to the Environment variables section. This code uses an environment variable to store the Amazon SNS topic ARN.
    1. For the key, enter TOPIC_ARN.
    2. For the value, enter the ARN of the Amazon SNS topic created earlier.
  9. Under Execution role, select Choose an existing role, and then select the role created earlier from the dropdown.
  10. Leave everything else as-is, and then, at the top, select Save.

Step 5: Verify it Works

We now have the Lambda function, an Amazon SNS topic, AWS Config watching our Amazon S3 buckets, and a CloudWatch Rule to trigger the Lambda function if a bucket is found to be non-compliant. Let’s test them to make sure they work.

We have an Amazon S3 bucket, myconfigtestbucket that’s been created in the region monitored by AWS Config, as well as the associated Lambda function. This bucket has no public read or write access set in an ACL or a policy, so it’s compliant.
 

The "Config Dashboard"

Figure 12: The “Config Dashboard”


 
Let’s change the bucket’s ACL to allow public listing of objects:
 
Screen shot of "Permissions" tab showing Everyone granted access

Figure 13: Screen shot of “Permissions” tab showing Everyone granted list access


 
After saving, the bucket now has public access. After several minutes, the AWS Config Dashboard notes that there is one non-compliant resource:
 
The "Config Dashboard" shown with a non-compliant resource

Figure 14: The “Config Dashboard” shown with a non-compliant resource


 
In the Amazon S3 Console, we can see that the bucket no longer has public listing of objects enabled after the invocation of the Lambda function triggered by the CloudWatch Rule created earlier.
 
The "Permissions" tab showing access no longer allowed

Figure 15: The “Permissions” tab showing list access no longer allowed


 
Notice that the AWS Config Dashboard now shows that there are no longer any non-compliant resources:
 
The "Config Dashboard" showing zero non-compliant resources

Figure 16: The “Config Dashboard” showing zero non-compliant resources


 
Now, let’s try out the Amazon S3 bucket policy check by configuring a bucket policy that allows list access:
 
A bucket policy that allows access

Figure 17: A bucket policy that allows list access


 
A few minutes after setting this bucket policy on the myconfigtestbucket bucket, AWS Config recognizes the bucket is no longer compliant. Because this is a bucket policy rather than an ACL, we publish a notification to the SNS topic we created earlier that lets us know about the potential policy violation:
 
Notification about potential policy violation

Figure 18: Notification about potential policy violation


 
Knowing that the policy allows open listing of the bucket, we can now modify or delete the policy, after which AWS Config will recognize that the resource is compliant.

Conclusion

In this post, we demonstrated how you can use AWS Config to monitor for Amazon S3 buckets with open read and write access ACLs and policies. We also showed how to use Amazon CloudWatch, Amazon SNS, and Lambda to overwrite a public bucket ACL, or to alert you should a bucket have a suspicious policy. You can use the CloudFormation template to deploy this solution in multiple regions quickly. With this approach, you will be able to easily identify and secure open Amazon S3 bucket ACLs and policies. Once you have deployed this solution to multiple regions you can aggregate the results using an AWS Config aggregator. See this post to learn more.

If you have feedback about this blog post, submit comments in the Comments section below. If you have questions about this blog post, start a new thread on the AWS Config forum or contact AWS Support.

Want more AWS Security news? Follow us on Twitter.