AWS Security Blog
How to Receive Alerts When Specific APIs Are Called by Using AWS CloudTrail, Amazon SNS, and AWS Lambda
Let’s face it—not all APIs were created equal. For example, you may be really interested in knowing when any of your Amazon EC2 instances are terminated (ec2:TerminateInstance), but less interested when an object is put in an Amazon S3 bucket (s3:PutObject). In this example, you can delete an object, but you can’t bring back that terminated instance. So this begs the question, “Is there a way to be notified when certain APIs are called using my AWS account?” The answer is yes! But how, though, can you be notified if unexpected API calls are made to supported services in your AWS account?
This blog post will show you how to receive email notifications by using AWS CloudTrail, Amazon Simple Notification Service (SNS), and AWS Lambda when specific APIs that you are interested in monitoring are called in your AWS account. Specifically, this post includes step-by-step configuration instructions for implementing a solution, which combines:
- CloudTrail – As the source of log files for analysis.
- Lambda – To implement the parsing and filtering logic.
- SNS – To send notifications.
The following diagram shows how these AWS services work together.
The following is a breakdown of the process shown in the preceding diagram:
- CloudTrail generates a log entry for every API call made on your account. CloudTrail aggregates the log entries in JSON text files, zips the files, and copies them to the Amazon S3 bucket configured to receive CloudTrail log files.
- The log parsing logic is deployed as a Lambda function. The Lambda function is triggered when CloudTrail copies a new file to your S3 bucket.
- The Lambda function uses a configuration file that contains a list of specific API keywords that, when detected, will trigger a notification. The configuration file is stored in an S3 bucket.
- Whenever relevant keywords are detected in the CloudTrail logs, the Lambda function posts a notification to an SNS topic.
- SNS dispatches the notification to every topic subscriber via email, Amazon SQS, SMS, HTTP call, or mobile push.
The rest of this blog post walks through the step-by-step instructions to implement the flow shown in the preceding diagram. I will complete the AWS interactions in this walkthrough via the AWS Command Line Interface (CLI) on an Amazon Linux instance.
The prerequisites for these step-by-step configuration instructions are the following:
- An AWS account.
- Basic knowledge of AWS concepts such as regions and access keys.
- The AWS CLI installed on your computer and configured with administrative privileges (the commands will not run under an EC2 role—they require actual user credentials). You can follow instructions to install the AWS CLI and documentation to configure the AWS CLI with your access key, secret key, default output format, and default region. Be sure to use AWS CLI version 1.7.22 or more recent.
- To test the Lambda function on your laptop before deploying it on AWS, you must also install NodeJS4.3 and the npm tool and be familiar with a *nix-family shell.
This walkthrough will use us-west-2, US West (Oregon), as the default region. The full source code for this walkthrough is available on GitHub.
The Walkthrough
Step 1: Create an S3 bucket and configure CloudTrail
First, you will create an S3 bucket for CloudTrail to store the CloudTrail log files of your account’s API calls and to enable the CloudTrail service itself. If you have already enabled CloudTrail for your account, you can skip directly to Step 2.
The S3 bucket must have a unique name and requires an IAM bucket policy to ensure that the CloudTrail service has sufficient privileges to create files in the bucket. An Amazon S3 Bucket Policy provides a CloudTrail default S3 policy.
The following cloudtrail create-subscription command will automatically create the bucket, associate a bucket policy for CloudTrail access, and enable and configure CloudTrail for your account in that region.
Make sure you modify the $BUCKETNAME value, because S3 bucket names must be globally unique.
REGION=us-west-2 BUCKETNAME=cloudtrail-logs-abcd1234 TRAILNAME=security-blog-post aws cloudtrail create-subscription --region $REGION --s3-new-bucket $BUCKETNAME --name $TRAILNAME
From now on, all your AWS API calls from the console, AWS CLI, or the SDK will be recorded and delivered to you in a text file with content similar to the following.
{
"Records": [
{
"eventVersion": "2.0",
"eventSource": "aws:s3",
"awsRegion": "us-west-2",
"eventTime": "2014-11-27T21:08:30.487Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "AWS:ARxxxxxxxxxxSW:i-4fxxxxa5"
},
"requestParameters": {
"sourceIPAddress": "54.211.178.99"
},
"responseElements": {
"x-amz-request-id": "F104F805121C9B79",
"x-amz-id-2": "Lf8hbNPkrhLAT4sHT7iBYFnIdCJTmxcr1ClX93awYfF530O9AijCgja19rk3MyMF"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "quickCreateConfig",
"bucket": {
"name": "aws-cloudtrail-xxx",
"ownerIdentity": {
"principalId": "AHxxxxxxxxQT"
},
"arn": "arn:aws:s3:::aws-cloudtrail-xxx"
},
"object": {
nbsp; "key": "AWSLogs/577xxxxxxxx68/CloudTrail/us-west-2/2014/11/27/577xxxxxxxx68_CloudTrail_us-west-2_20141127T2110Z_JWcmX2IqHMUoBRIM.json.gz",
"size": 2331,
"eTag": "63d801bb561037f59f2cb4d1c03c2392"
}
}
}
]
}
Step 2: Create an SNS topic and subscribe an email address
You will now create an SNS topic that will send subscribers a copy of every message published to the topic. Subscribers can use different protocols: HTTPS, Mail, SQS, SMS (to US-based numbers only), and push notifications to mobile devices.
For the purpose of this walkthrough, you will subscribe an email address to the SNS topic. The Lambda function will post an SNS notification for every unexpected API call detected in the CloudTrail logs. Create the SNS topic and subscribe your email address to it (replace <YOUR EMAIL ADDRESS> with your actual email address).
EMAIL=<YOUR EMAIL ADDRESS>
TOPIC_ARN=$(aws sns create-topic --name cloudtrail_notifications --output text --region $REGION)
aws sns subscribe --protocol email --topic-arn $TOPIC_ARN --notification-endpoint $EMAIL --region $REGION
echo $TOPIC_ARN
Be sure to write down the topic ARN given as output of the previous command. You will need the topic ARN to configure the Lambda function later. The topic ARN should look similar to the following.
arn:aws:sns:us-west-2:577xxxxxxx68:cloudtrail_notifications
You will receive a confirmation email (check your spam folder, just in case). The email contains a link to confirm the subscription. You must click this link to complete the subscription process.
Step 3: Create IAM roles for your Lambda function
A Lambda function needs an IAM role to work correctly.
The IAM execution role refers to an IAM role that grants your custom code permissions to access the AWS resources it needs. Lambda assumes this role while executing Lambda functions on your behalf.
A role has two parts: a trust policy that defines which services or users can be authorized to assume the role, and one or more policies defining the permissions granted. Lambda assumes a role when your function is executed. Your function has only the permissions described in the role’s policies.
The following command creates the trust policy to authorize Lambda to assume the execution role.
cat > role_trust_policy.json <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
EOF
Then, create the execution role itself.
ROLE_ARN=$(aws iam create-role --role-name lambda-security-blog --assume-role-policy-document file://./role_trust_policy.json --query 'Role.Arn' --output text) echo $ROLE_ARN
Now that you have created the execution role, add the permissions that the Lambda function requires. The following commands assume that environment variables set by previous commands still exist.
cat > role_exec_policy.json <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:*"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::$BUCKETNAME/*"
]
},
{
"Effect": "Allow",
"Action": ["sns:Publish"],
"Resource": [
"$TOPIC_ARN"
]
}
]
}
EOF
The first statement in the preceding command gives the Lambda function authorization to publish its own logs to Amazon CloudWatch Logs. The second statement gives the Lambda function authorization to read from the CloudTrail log S3 bucket. And the last statement gives the Lambda function authorization to post messages on the SNS topic created earlier.
The following command will add these three statements to the role.
aws iam put-role-policy --role-name lambda-security-blog --policy-name lambda-execution-role --policy-document file://./role_exec_policy.json
Step 4: Create the Lambda function
Now that you have enabled CloudTrail, its associated S3 bucket, and an SNS topic, you are ready to create the CloudTrail log parsing function and deploy it to Lambda.
Create a bucket in which to store the configuration file
The Lambda function uses regular expressions to match unexpected API calls. It will use an external configuration file that is stored on S3 to store the regular expressions that you want the function to find. Using an external configuration file allows you to fine-tune the expression without having to redeploy the Lambda function with every change.
To simplify deployment, the configuration file will be stored in the same S3 bucket as the CloudTrail log files. The configuration file appears as follows.
{
"source" : "iam|signin|sts",
"regexp" : "Password|AccessKey|PutRolePolicy|User|Token|Login",
"sns" : {
"region" : "us-west-2",
"topicARN" : "arn:aws:sns:us-west-2:577xxxxxxx68:PushNotifications"
}
}
These are the parts of the configuration file:
- The source parameter defines the service APIs that you will monitor, which can be adjusted to match your requirements.
- The regexp parameter defines the list of API calls that will trigger notifications. This also can be adjusted to match your requirements.
- The sns section defines your SNS configuration. Adjust it to the region and topicARN created in Step 2 earlier in this blog post.
Edit the configuration file to meet your requirements, and then copy it to your S3 bucket by using the following command.
aws s3 cp filter_config.json s3://$BUCKETNAME/filter_config.json --region $REGION
Create the Lambda function
You can create the Lambda function by using any good JavaScript text editor. At the time of this writing, Lambda is using the Node.js framework. The core of your Lambda function will be a JavaScript function, defined as a Node.js handler. When writing your code, remember that Node.js is asynchronous. Every function call returns immediately. You can use callbacks or futures to synchronize your code execution.
When deploying the Lambda function, you will define the events that will trigger the execution of the Lambda function. Lambda can currently trigger functions in response to an API call, a change in a DynamoDB table, an S3 event, or an SNS notification.
This walkthrough uses the Q JavaScript Library. Q is abstracting asynchronous operations as promise objects (also known as futures). The Lambda function performs the following operations in sequential order:
- Download the Lambda function configuration file created at the beginning of Step 4.
- Download the CloudTrail log file of whose availability you have just been notified. Lambda passes a record object to the function. This record object contains the name of the file just created on S3.
- Unzip the CloudTrail log file to temporary storage.
- Filter the file, looking for log entries from selected AWS services only. In this example, it is looking for IAM, sign-in, and STS-generated records only.
- Search for given regular expressions.
- Send notifications to SNS when there is a match.
I will break the Lambda function into separate JavaScript functions, each subfunction implementing one of the preceding operations. The core of the Lambda function is included below. It implements the operations as described.
The full source code is available on GitHub. For clarity, snippets presented below do not include error handling, logging, or Q-related code.
exports.handler = function(event, context) {
var bucket = event.Records[0].s3.bucket.name;
var key = event.Records[0].s3.object.key;
// Download from S3, Gunzip, Filter and send notifications
download(bucket, key)
.then(extract)
.then(filter)
.then(notify)
.catch(function (error) {
// Handle any error from all above steps
})
.done(function() {
context.done();
});
};
The Q Library will ensure that each function is called sequentially.
Download the configuration file
Use the AWS JavaScript SDK to download the configuration file from S3.
var s3 = new aws.S3({apiVersion: '2006-03-01'});
var params = { Bucket:FILTER_CONFIG_BUCKET, Key:FILTER_CONFIG_FILE };
var body = s3.getObject(params)
.createReadStream()
.on('data', function(data) {
FILTER_CONFIG = JSON.parse(data);
});
This code downloads the configuration file and parses the text to create a JavaScript object. You will need to adjust the value of the global variables FILTER_CONFIG_BUCKET and FILTER_CONFIG_FILE to the name of your bucket and configuration file.
Download the CloudTrail log file from S3
Download the CloudTrail log file from S3 with the fs module provided by Node.js for the I/O operations to a local directory (/tmp) and the AWS Node.js SDK for the S3 API.
// extract file name from key name
var fileName = key.match(/.*/(.*).json.gz$/)[1]
var file = fs.createWriteStream('/tmp/' + fileName + '.json.gz');
// pipe from S3 to local file
var s3 = new aws.S3({apiVersion: '2006-03-01'});
var params = { Bucket:bucket, Key:key };
var stream = s3.getObject(params).createReadStream();
stream.pipe(file);
This will place a copy of the file in /tmp on the Lambda container.
Unzip the CloudTrail log file to temporary storage
To extract the text-based log file from the compressed archive, use the Node.js GZIP Library.
var gzFileName = file.path.match(/.*/(.*).json.gz$/)[1];
var jsFileName = '/tmp/' + gzFileName + '.json';
var zlib = require('zlib');
var unzip = zlib.createGunzip();
var inp = fs.createReadStream(file.path);
var out = fs.createWriteStream(jsFileName);
inp.pipe(unzip).pipe(out);
This will create an input stream to read the file from the local file system, and then pipe this input stream to an output stream attached to another file on the file system.
Filter for selected AWS services
Now that you have a JSON text file available locally, parse it to look for the AWS services and API for which you will generate an SNS notification.
var cloudTrailLog = require(file);
var records = cloudTrailLog.Records.filter(function(x) {
return x.eventSource.match(new RegExp(FILTER_CONFIG.source));
});
Search for regular expression
The last step loads an array of record objects in memory for parsing. For every matching record, build a message and pass it to the sendNotification() function.
for (var i = 0; i < records.length; i++) {
if (records[i].eventName.match(new RegExp(FILTER_CONFIG.regexp))) {
var message = "Event : " + records[i].eventName + "n" +
"Source : " + records[i].eventSource + "n" +
"Params : " +
JSON.stringify(records[i].requestParameters, null, '') + "n" +
"Region : " + records[i].awsRegion + "n"
var task = sendNotification(message,
FILTER_CONFIG.sns.topicARN, '');
}
}
Send SNS Notifications
Finally, send the SNS notification using the AWS JavaScript SDK.
var sns = new aws.SNS({
apiVersion: '2010-03-31',
region: FILTER_CONFIG.sns.region
});
var params = {}
params = {
Message: msg,
TopicArn: topicARN
};
sns.publish(params, function(err,data) {
if (err) {
// an error occurred
} else {
// successful response
}
});
Test the function locally
Before uploading the function to Lambda and testing it on AWS, we recommend testing the function on your local machine. This will considerably reduce your develop-deploy-test-debug cycle by quickly catching programming mistakes early in the development cycle. Testing Lambda functions locally before you deploy them should be considered a best practice because it allows you to test your code as you build it and customize it for your AWS account.
To test the Lambda function locally, I wrote a small piece of code that will invoke the Lambda function as it is invoked from the Lambda service itself. This stub code will provide two objects to your function:
- An input record, similar to the one Lambda will provide when hooked to S3 events. It will create a file (input.json) that will be used to store that record.
- A context object similar to the one Lambda is providing to your function.
The input record for this test looks like the following.
{
"Records": [
{
"eventVersion": "2.0",
"eventSource": "aws:s3",
"awsRegion": "us-west-2",
"eventTime": "2014-11-27T21:08:30.487Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "AWS:AROXXXXXXXXXXSW:i-4ffXXXXa5"
},
"requestParameters": {
"sourceIPAddress": "54.211.178.99"
},
"responseElements": {
"x-amz-request-id": "F104F805121C9B79",
"x-amz-id-2": "Lf8hbNPkrhLAT4sHT7iBYFnIdCJTmxcr1ClX93awYfF530O9AijCgja19rk3MyMF"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "quickCreateConfig",
"bucket": {
"name": "aws-cloudtrail-xxx",
"ownerIdentity": {
"principalId": "Ah5XXXXXXXQT"
},
"arn": "arn:aws:s3:::aws-cloudtrail-xxx"
},
"object": {
"key": "AWSLogs/577XXXXXXX68/CloudTrail/us-west-2/2014/11/27/577XXXXXXXX68_CloudTrail_us-west-2_20141127T2110Z_JWcmX2IqHMUoBRIM.json.gz",
"size": 2331,
"eTag": "63d801bb561037f59f2cb4d1c03c2392"
}
}
}
]
}
Be sure that the bucket name and key point to a real CloudTrail log in your bucket. The size and eTag values are not used by the function and do not need to be adjusted to your data.
You can obtain this input record object from a real Lambda function or from the documentation. The invocation function is in main.js and looks like the following code.
var lambda = require('./cloudtrail.js') //your lambda function
var event = require('./input.json') //your input record
var context = {}
context.done = function(arg1, arg2) {
console.log('context.done')
}
lambda.handler(event, context)
To test your code locally, first install dependencies on your local machine.
npm install aws-sdk q
The AWS SDK must be configured with an access key, secret key, and default region. If you are using the AWS CLI as described above, you are configured appropriately; otherwise, use aws configure to generate a default configuration file.
Then, invoke node from your laptop’s command line.
node main.js
Observe the output of the logs and fix any errors before deploying your code to Lambda.
Deploy the Lambda function
As soon as you’re confident the function works as expected, you’re ready to deploy it to Lambda. First, create a .zip file that contains the function (cloudtrail.js) and its dependencies (the Q library).
zip -r cloudtrail.zip cloudtrail.js node_modules -x node_modules/aws-sdk/*
You do not need to include the AWS JavaScript SDK because Lambda’s execution environment provides it by default.
Now, upload and create the function on Lambda ($ROLE_ARN must be set with the value returned by aws iam create-role in Step 3, earlier in this post.)
FUNCTION_ARN=$(aws lambda create-function --zip-file fileb://./cloudtrail.zip --function-name cloudtrail --runtime nodejs4.3 --role "$ROLE_ARN" --handler cloudtrail.handler --region $REGION --output text --query "FunctionArn")
Take note of the Function ARN returned because you will need it in the next step.
Hook the function to S3
The last thing to do is to configure S3 to trigger the Lambda function when a file is created in the bucket. This is a two-step process. You first need to add permission to the Lambda function to authorize your S3 bucket to trigger the Lambda function execution.
AWS_ACCOUNT_ID=$(echo $ROLE_ARN | sed 's/^arn:aws:iam::(.*):.*$/1/') aws lambda add-permission --function-name cloudtrail --statement-id Id-cloudtrail --action "lambda:InvokeFunction" --principal s3.amazonaws.com --source-arn arn:aws:s3:::$BUCKETNAME --source-account $AWS_ACCOUNT_ID --region $REGION
Be sure to replace $FUNCTION_ARN with the function ARN returned when you uploaded the code in Step 4, earlier in this post.
Next, configure the S3 notification system. You can perform this configuration either from the S3 web console or the command line, using the AWS CLI. When using the AWS CLI, a configuration file is required.
cat > s3_notifications_config.json <<EOF
{
"CloudFunctionConfiguration": {
"CloudFunction": "$FUNCTION_ARN",
"Id": "cloudtrail-lambda-notifications",
"Event": "s3:ObjectCreated:*"
}
}
EOF
Be sure to replace <FUNCTION_ARN> with the Lambda function ARN that was returned when you uploaded the function in the “Deploy the function” subsection, earlier in this post.
Finally, turn on event notification from S3 to Lambda.
aws s3api put-bucket-notification --notification-configuration file://./s3_notifications_config.json --bucket $BUCKETNAME --region $REGION
Step 5: End-to-end testing
If you have followed the instructions and completed the steps to this point, congratulations! You are now ready to do end-to-end testing of your deployment.
The good news is that only a few steps are required to test your deployment. You just need to generate a few unexpected API calls (the ones you have configured in your filter_config.json configuration file). If you are using the configuration file proposed above, sign in to the AWS Management Console, create a couple of roles, and then attach and remove policies from these roles. After a few minutes, CloudTrail will generate the log files, and S3 will trigger your Lambda function and will post messages to your SNS topic.
Step 6: Troubleshooting
If you don’t receive email notifications immediately, be patient. CloudTrail can take a few minutes to post the CloudTrail log files, and the email delivery depends on your email infrastructure.
If you have not received emails after 10–15 minutes, check the following:
- Are there CloudTrail log files in your bucket? If not, check your CloudTrail configuration.
- Has your Lambda function been invoked? If not, check your S3 notification settings, including Lambda function invocation permissions.
- Check Lambda logs on CloudWatch logs. The link is available on the Lambda console.
- From the logs:
- Is the event-filtered list empty? Check your regular expression filter in the configuration file.
- Does an error occur when posting to SNS? Check the Lambda function execution role policies.
- Still not receiving email notifications? Did you validate your email address when you subscribed to the topic? Did you check your junk mail folder?
Conclusion
This blog post walked through the steps required to invoke a Lambda function when CloudTrail makes log files available, and then to filter these files to generate notifications on unexpected API calls. As usual, do not forget to delete services and resources when you are not using them anymore to avoid any extra charges incurred by testing this scenario.
All source code presented in this blog post is available for you to download on GitHub.
If you have questions or suggestions, visit the CloudTrail forum and the Lambda forum.
– Sébastien