AWS Cloud Operations Blog
Keeping CloudWatch Dashboards up to date using AWS Lambda
With the launch of the new CloudWatch Dashboards API and CloudFormation support it is now easy to automate your CloudWatch Dashboards and make sure they monitor all the resources that you launched when creating your CloudFormation stacks.
Let’s now see how you can use the new CloudWatch Dashboards API to dynamically update your dashboard as EC2 instances are added or removed. When you use auto-scaled EC2 instances for example, EC2 instances may be launched or terminated at any time and if you have a CloudWatch Dashboard monitoring your EC2 resources it can suddenly be monitoring instances that do not exist anymore, and may be missing ones that do exist.
Use AWS Lambda to keep EC2 instances up to date
A simple solution is to run the script below periodically in AWS Lambda. The script loads your CloudWatch Dashboards that monitors your instances and updates the EC2 graph widgets if needed.
The script:
- Loads the specified CloudWatch Dashboard(s)
- Looks for all graph widgets displaying EC2 instance metrics
- Calls EC2 DescribeInstances API with configured parameters to discover the current EC2 instances for that graph in that region
- Updates the widget if needed
- Saves the CloudWatch Dashboards if any widget definition has changed
The script is configured via an environment variable AWS_DASHBOARDS whose value is a JSON array of the dashboard names that you want to update along with (optional) parameters for EC2 DescribeInstances API which sets which EC2 Instances should be displayed on each dashboard.
Here’s an example that will update a CloudWatch Dashboard called MyAutoScalingDashboard with all running instances in the AutoScaling group called MyAutoScalingGroup:
[
{
"dashboardName": " MyAutoScalingDashboard ",
"ec2DescribeInstanceParams": {
"Filters": [
{
"Name": "instance-state-name",
"Values": [ "running" ]
},
{
"Name": "tag:aws:autoscaling:groupName",
"Values": [ "MyAutoScalingGroup" ]
}
]
}
}
]
To create the Lambda function:
- In the Lambda console, choose Create a Lambda function
- Select Blank Function and choose Next when asked to configure a trigger.
- For Name, enter
ec2DashboardUpdater - For Code entry type choose Upload a file from Amazon S3.
- For S3 link URL enter
https://s3.amazonaws.com/ec2-dashboard-updater/ec2DashboardUpdater.zip - You’ll need to specify the
AWS_DASHBOARDSenvironment variable with a JSON array for the dashboards you want to update, e.g. with the JSON in the example earlier (no newlines:[{"dashboardName":"AutoUpdateEC2Dashboard","ec2DescribeInstanceParams":{"Filters":[{"Name":"instance-state-name","Values":["running"]},{"Name":"tag:aws:autoscaling:groupName","Values":["test-asg-01"]}]}}] - Set the Handler as
ec2DashboardUpdater.handler - For Role select Create a custom role
- In the IAM Console window that opens choose IAM Role of Create a new IAM Role
- Set Role Name of
ec2DashboardUpdaterRole - Choose View Policy Document, Edit, Ok.
- Set the role to:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudwatch:GetDashboard", "cloudwatch:PutDashboard", "ec2:DescribeInstances" ], "Resource": "*" } ] } - Choose Allow
- In Advanced settings set Timeout to a value that is enough to describe all instances for all EC2 graph widgets and load and save all dashboards – set it to 5 min if unsure
- Select Next then Create function
To call the Lambda function periodically:
- In the CloudWatch console, choose Rules in the left navigation pane
- Select Create rule
- Choose the Schedule radio button and set Fixed rate of
15Minutes - Select Add target, make sure Lambda function is active and for Function select the previously created
ec2DashboardUpdaterfunction - Choose Configure details
- Enter Name as
ec2DashboardUpdaterCron - Select Create rule
And that’s it, you’re done. Your selected dashboards will now be kept up to date with your changing EC2 instances hourly. If your EC2 instances do not change from hour to hour then your dashboards will not be updated.
Please note some limitations of the script:
- It will only update graphs where the first metric is an EC2 instance id metric, such as CPUUtilization, NetworkIn or DiskReadBytes, in the AWS/EC2 namespace.
- It assumes EC2 metrics on a graph all use the same MetricName field.
- All EC2 graphs on a particular dashboard are for the same list of EC2 instances (as specified in the ec2DescribeInstanceParams configuration parameter.
The (Javascript) code for the script is given below. Feel free to use it as the basis for keeping your CloudWatch Dashboards up to date with other resources.
//
// EC2 Dashboard Updater
// This script will keep EC2 graphs on a chosen list of CloudWatch Dashboards up to date. In the CloudWatch Events
// console create a rule to call this Lambda whenever and EC2 instance changes state.
//
// Configure this script with the AWS_DASHBOARDS environment variable. The content of AWS_DASHBOARDS indicates
// which dashboards to update with which EC2 instances. The value is an array in JSON format. Each array element
// is an object with two properties, dashboardName and ec2DescribeInstanceParams. dashboardName is the dashboard to
// update and ec2DescribeInstanceParams are parameters to pass to EC2 DescribeInstances API, to determine which
// instances to update the graphs with.
//
// For example, set AWS_DASHBOARDS to the following to keep 'MyDashboard' up to date with the instances in the
// AutoScalingGroup 'MyAutoScalingGroup':
//
// [
// {
// dashboardName: 'MyDashboard',
// ec2DescribeInstanceParams: {
// Filters: [
// {
// Name: 'tag:aws:autoscaling:groupName',
// Values: [ 'MyAutoScalingGroup' ]
// }
// ]
// }
// }
// ]
//
// Limitations:
// - it will only update graphs whose first metric is an EC2 instance metric, all other metrics on the graph
// will be replaced with these metrics
// - metrics can not have custom periods or statistics, the graph defaults will be used
//
'use strict'
var _ = require('lodash'),
co = require('co'),
AWS = require('aws-sdk'),
EC2_NAMESPACE = 'AWS/EC2',
INSTANCE_ID_DIM = 'InstanceId',
REPEAT_PREVIOUS = '...',
NAME_TAG = 'Name',
METRIC_WIDGET_TYPE = 'metric';
function getDashboards() {
let dashboardDef = process.env.AWS_DASHBOARDS,
dashboards;
if (! dashboardDef) {
throw 'Environment variable AWS_DASHBOARDS is not set. It should be set with JSON array, e.g [{"dashboardName": "myDashboard"}]';
}
try {
dashboards = JSON.parse(dashboardDef);
} catch (err) {
throw 'Error, AWS_DASHBOARDS is not valid JSON';
}
if (! Array.isArray(dashboards)) {
throw 'Expecting AWS_DASHBOARDS to be an array';
}
return dashboards;
}
function ec2MetricsFromDescribeInstances(ec2DescribeInstances, metricName) {
return _.chain(ec2DescribeInstances.Reservations).
map('Instances').
flatten().
map(function(instance) {
let idAndLabel = {
id: instance.InstanceId,
label: instance.InstanceId
};
_.each(instance.Tags, (tag) => { // Use Name tag instead of InstanceId as label, if it exists
if (tag.Key === NAME_TAG) {
idAndLabel.label = tag.Value;
return false;
}
});
return idAndLabel;
}).
sortBy(['label']).
map((idAndLabel, i) => {
var metric = i == 0 ?
[ EC2_NAMESPACE, metricName, INSTANCE_ID_DIM, idAndLabel.id ] :
[ REPEAT_PREVIOUS, idAndLabel.id ];
if (idAndLabel.id != idAndLabel.label) {
metric.push({label: idAndLabel.label});
}
return metric;
}).
value();
}
function* getEC2Metrics(dashboardContext, region, metricName) {
var params = dashboardContext.dashboard.ec2DescribeInstanceParams || {},
ec2Cache = dashboardContext.ec2Cache,
ec2DescribeInstances;
// ec2Cache is a cache of instance responses per region for the current dashboard, to limit calls to EC2
if (!ec2Cache) {
dashboardContext.ec2Cache = {};
} else if (ec2Cache[region]) {
ec2DescribeInstances = ec2Cache[region];
}
if (!ec2DescribeInstances) {
// Not in cache, call EC2 to get list of regions
let ec2Client = new AWS.EC2({region: region});
ec2DescribeInstances = yield ec2Client.describeInstances(params).promise(),
dashboardContext.ec2Cache[region] = ec2DescribeInstances;
}
return ec2MetricsFromDescribeInstances(ec2DescribeInstances, metricName);
}
function* updateDashboard(cloudwatchClient, dashboardContext, dashboardBody) {
var newDashboard = _.cloneDeep(dashboardBody),
widgets = newDashboard.widgets,
dashboardName = dashboardContext.dashboard.dashboardName;
for (let i = 0; i < widgets.length; i++) {
let widget = widgets[i],
metrics = widget.properties.metrics,
region = widget.region;
// Check to see if it's a metric widget and first metric is an EC2 Instance metric
if (widget.type === METRIC_WIDGET_TYPE && metrics && (metrics[0].length === 4 || metrics[0].length === 5) &&
metrics[0][0] === EC2_NAMESPACE && metrics[0][2] === INSTANCE_ID_DIM) {
let ec2Metrics = yield getEC2Metrics(dashboardContext, region, metrics[0][1]);
widget.properties.metrics = ec2Metrics;
}
}
if (JSON.stringify(dashboardBody) !== JSON.stringify(newDashboard)) {
console.log(`Updating dashboard ${dashboardName} to:`, JSON.stringify(newDashboard));
// Save updated dashboard
yield cloudwatchClient.putDashboard({
DashboardName: dashboardName,
DashboardBody: JSON.stringify(newDashboard)
}).promise();
} else {
console.log(`Dashboard ${dashboardName} is unchanged, update skipped`);
}
}
exports.handler = (event, context, callback) => {
var dashboards = getDashboards(),
cloudwatchClient = new AWS.CloudWatch(),
dashboard,
dashboardContext;
try {
co(function* () {
for (let i = 0; i < dashboards.length; i++) {
dashboard = dashboards[i];
dashboardContext = { dashboard: dashboard };
if (dashboard.dashboardName) {
let dashboardResponse = yield cloudwatchClient.getDashboard({
DashboardName: dashboard.dashboardName }).promise(),
dashboardBody = JSON.parse(dashboardResponse.DashboardBody);
yield updateDashboard(cloudwatchClient, dashboardContext, dashboardBody);
}
}
callback(null, 'Update complete');
});
} catch (err) {
callback(err);
}
};