AWS Open Source Blog
Creating simple AWS Cost and Usage charts with D3 JavaScript library
Web applications interacting with AWS in a number of ways may need to represent and display sets of information in the form of charts, diagrams, or graphs. Common examples of that information includes small amounts of data coming from AWS Costs & Usage Reports or Amazon Elastic Compute Cloud (Amazon EC2), either historical or real-time.
Web developers who are trying to keep it simple may want to build a basic chart via JavaScript, without managing additional policies, AWS Management Console access, service authorizations and permissions, HTML objects embeds, additional software tools, or AJAX calls. D3 JavaScript library is one popular open source JavaScript library that is able to quickly and simply draw a variety of charts. D3 JavaScript library not only helps users easily create clear and readable charts, but also helps manipulate and transform data from one format to another, supports dynamic updates, and supports animations.
In this post I’ll explain how to create simple charts with D3 from AWS data. As examples, I’ll describe how to generate:
- A bar chart showing costs or usage over time, with data coming from AWS Cost Explorer
- A pie chart showing active Amazon EC2 instances states, with data coming from Amazon EC2
In both cases, I’ll use Boto3 Python APIs to retrieve data. The end of each example includes a demo on jsfiddle.net.
Note: This post focuses on simple charts generated from small amounts of data. For more complex charts and datasets, you might want to use AWS QuickSight.
Components and solution overview
- AWS Cost & Usage Report contains a comprehensive set of AWS cost and usage data, including additional metadata about AWS services, pricing, and reservations (e.g., Amazon EC2 Reserved Instances [RIs]). The AWS Cost & Usage Report lists AWS usage for each service category used by an account and its IAM users in hourly or daily line items, as well as any tags that we have activated for cost allocation purposes. We also can customize the AWS Cost & Usage Report to aggregate our usage data to the daily or hourly level.
- AWS Cost Explorer has an interface that lets us visualize, understand, and manage our AWS costs and usage over time.
- D3.js is a JavaScript library for manipulating documents based on data. D3 helps us visualize data using HTML, SVG, and CSS. D3’s emphasis on web standards gives us the capabilities of modern browsers without tying us to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation.
Getting data
Our first step in building charts is to retrieve the data (i.e., the set of values we want to visualize). Data can come not only from Cost Explorer, but also from other AWS services, such as Amazon EC2, Amazon Simple Storage Service (Amazon S3), AWS CloudFormation, or from outside AWS. Each data source may imply partial values, limitations, API definitions, security, confidentiality, formats, and time constraints, so each source must be chosen—and data must be collected—carefully and efficiently. More examples of data include our AWS IAM users list, the number of alarms coming from Amazon CloudWatch each month, the amount of budget assigned to each AWS account for the current year, and a set of AWS CloudTrail entries in a specific time range.
Once we’ve determined how to get data from its source and how to query it, we also should keep in mind that data can come in several formats, such as CSV (comma-separated values), TSV (tab-separated values), JSON, and YAML. Having data in a standard format helps developers to decouple it from the plotting or transformation code, share and integrate it with other tools and software, and limit any future update needs. D3 can read data expressed in CSV, TSV, and JSON directly, so in this post I’ll use CSV format because it also is widespread, and reading and manipulating it is quick and simple.
An introductory example of D3 plotting CSV data is provided in a post on the Observable site called “Learn D3: Data“.
Note: I assume the methods and queries I’ll use in this document are executed with the necessary AWS permissions and policies, granted, for example, by related roles, API keys, or credentials.
Bar chart example: Costs and usage over time
In this first example, we will get costs and usage data from AWS Cost Explorer APIs, and we will generate a couple of bar charts displaying AWS costs and usage over time.
Getting data
We will query AWS Cost Explorer to get data with the following characteristics:
- Time range: From day a to day b
- Accounting: Amazon EC2 usage and costs in terms of daily running hours
- Related to a specific group or project
The AWS Cost Explorer Python method we are going to use is get_cost_and_usage(**kwargs). The time range will reduce to have a starting time and an ending time, passed asTimePeriodparameter: Start(string)End(string). The accounting will be read as UnblendedCost and UsageQuantity Metrics, related to EC2: Running Hours Dimension. Our data has a daily Granularity.
The third characteristic, range, is custom and strictly related to our environment, thus it will refer to a related group Tag applied to all of the AWS resources we want to take into account.
Note: If using tags, please remember they must be attached to each AWS resource we want to monitor (for example, Amazon EC2 instances) and activated via AWS Cost Allocation Tags. Also note that Cost Explorer data is populated after a minimum time of one hour.
get_cost_and_usage() method returns a data in JSON format, but it includes a lot of data we don’t need. So once its invocation has been successfully completed, we can pick the values we’re interested in from the JSON results and output them in a CSV format. For complex JSON structures or bigger amounts of data, there are specific Python libraries to convert formats. In our example, data is simple and limited, so we’ll generate the CSV with standard Python print() calls (e.g., yyyy-mm-dd,value).
After we have the code to get the values, we must integrate it into the web application so our JavaScript code will be able to access it and plot the chart using D3 methods. Because this is out of the scope of this post, we’ll just have our CSV data wrapped into HTML tags, and assume they’ll be part of the HTML DOM produced by our web application.
Our Python script invoking get_cost_and_usage will be:
#!/usr/bin/python3
import boto3
ce = boto3.client('ce')
# my filtes, you might want to adapt them
starting = "2020-06-23"
ending = "2020-06-27"
group = "system"
request = {
'TimePeriod' : {
'Start': starting,
'End': ending
},
'Filter' : {
'And': [
{ 'Dimensions' : {
'Key' : 'USAGE_TYPE_GROUP',
'Values' : [ 'EC2: Running Hours' ]
}
},
{ 'Tags': {
"Key": "group",
"Values": [ group ]
}
}
]
},
'Granularity' : 'DAILY',
"Metrics" : [ 'UnblendedCost', 'UsageQuantity' ]
}
response = ce.get_cost_and_usage(**request)
# wrap cost values into specific <div>:
print ('<div class="cost-explorer-csv-cost">time,value')
# print date, cost
for results in response['ResultsByTime']:
print(results['TimePeriod']['Start'] + \
',' + results['Total']['UnblendedCost']['Amount'])
# close tag
print ('</div>')
# wrap usage values into specific <div>:
print ('<div class="cost-explorer-csv-usage">time,value')
# print date, usage
for results in response['ResultsByTime']:
print(results['TimePeriod']['Start'] + \
',' + results['Total']['UsageQuantity']['Amount'])
',' + results['Total']['UsageQuantity']['Amount'])
# close tag
print ('</div>')
To run this Python script successfully, we or our web application must have the necessary permissions. This means, for example, that your AWS security credentials are set accordingly, or you run the script from inside an instance with equivalent IAM roles granted, or inside AWS Cloud9.
If everything is set correctly, the above script should produce something like:
Generating chart
From the previous step, we have a web page with two <div> tags containing our CSV data to plot. Now our JavaScript code will read the data, parse it with D3 calls, and replace each with its bar chart. For simplicity and portability, we are also using a JQuery library.
Our final JavaScript code is:
jQuery(document).ready(function () {
myChart.timeBarChart('cost-explorer-csv-cost');
myChart.timeBarChart('cost-explorer-csv-usage');
});
var myChart = {
timeBarChart: function (target) {
if (typeof(target) == "undefined" || target=="") return;
target = "." + target;
var svgTarget = target + ' .chart';
// load CSV data
var csvData = jQuery(target).html();
// define svg tag that will contain the chart
jQuery(target).html('<svg class="chart">' +
'<div class="chartLoading">Loading...</div></svg>');
// set chart size and margins
var availX = jQuery('.cost-explorer').width() - 240;
var margin = {top: 20, right: 30, bottom: 130, left: 40},
width = availX - margin.left - margin.right - 4,
height = 500 - margin.top - margin.bottom;
var x = d3.scaleBand()
.rangeRound([0, width])
.padding(0.1);
var y = d3.scaleLinear()
.rangeRound([height, 0]);
var chart = d3.select(svgTarget)
.attr("width", width + margin.left + margin.right)
.attr("height", height + margin.top + margin.bottom)
.append("g")
.attr("transform", "translate(" + margin.left + "," + margin.top + ")");
var div = d3.select("body")
.append("div")
.style("opacity", 0)
.attr("class", "d3-tip");
// parse CSV data
console.log(csvData)
var parsedData = d3.csvParse(csvData, function(d) {
return {
time:d.time,
value:+d.value
};
});
// set x=time and y=value
var color = d3.scaleOrdinal(d3.schemeCategory10);
x.domain(parsedData.map(function(d) { return d.time; }));
y.domain([0, d3.max(parsedData, function(d) {
if (d.value == 0) { return 1 } else { return d.value } })]);
var xTicksLimit = Math.floor(x.domain().length / 36);
var xAxis = d3.axisBottom(x)
.tickValues(x.domain().filter(function(d, i) {
return !(i % xTicksLimit); }));
var yAxis = d3.axisLeft(y);
chart.append("g")
.attr("class", "x axis")
.attr("transform", "translate(0," + height + ")")
.call(xAxis)
.selectAll("text")
.attr("y", 12)
.attr("x", -12)
.attr("dy", ".30em")
.attr("transform", "rotate(-45)")
.style("text-anchor", "end");
chart.append("g")
.attr("class", "y axis")
.call(yAxis)
.append("text")
.attr("transform", "rotate(-90)")
.attr("y", 6)
.attr("dy", ".71em")
.style("text-anchor", "end")
// draw data bars
var bar = chart.selectAll(".bar")
.data(parsedData)
.enter().append("rect")
.attr("class", "bar")
.attr("x", function(d) { return x(d.time); })
.attr("y", function(d) { return y(d.value); })
.attr("height", function(d) { return height - y(d.value); })
.attr("width", x.bandwidth())
.style("fill", color(d3.scaleQuantize()))
.on("mouseover", function(d) {
div.transition()
.duration(200)
.style("opacity", .9)
div.html(d.value)
.style("left", (d3.event.pageX) + "px")
.style("top", (d3.event.pageY - 28) + "px");
})
.on("mouseout", function(d) {
div.transition()
.duration(500)
.style("opacity", 0);
});
jQuery(".chartLoading").remove();
}
}Our generated charts are:
A full example is available as JSFiddle.
Pie chart example: Amazon EC2 instance states
In this example, we will create a pie chart displaying the status of our active Amazon EC2 instances. By active, I mean each Amazon EC2 instance in starting, running, shutting down, stopping, stopped, or recently terminated status.
Getting data
Because Amazon EC2 usage data is available in AWS Costs & Usage after a minimum of one hour, we will not use its methods; rather, we will query Amazon EC2 directly using Boto3 describe_instances(**kwargs), which will allow us to get the current, real-time status of our instances.
We are interested in cutting the pie according to the various possible Amazon EC2 instance states: pending, running, shutting down, terminated, stopping, and stopped. For each one, we just have to get the number of instances in that state;: those will be the values to plot.
Let’s also keep the tag-based group or project filter described in the previous example. Note that describe_instances() produces a lot of information, so you might want to add more filters.
Our Python code will be:
#!/usr/bin/python3
import boto3
counters = {
'pending': 0,
'running': 0,
'shutting-down': 0,
'terminated': 0,
'stopping': 0,
'stopped': 0
}
filters = [
{'Name': 'tag:group', 'Values': [ 'mygroup' ]}
]
ec2 = boto3.client('ec2')
# get relevant values
paginator = ec2.get_paginator('describe_instances')
response_iterator = paginator.paginate(Filters=filters)
for response in response_iterator:
for reservation in response['Reservations']:
for instance in reservation['Instances']:
state = instance['State']['Name']
counters[state] += 1
# output values in CSV format
print('status,count')
print('pending,' + str(counters['pending']))
print('running,' + str(counters['running']))
print('shutting-down,' + str(counters['shutting-down']))
print('terminated,' + str(counters['terminated']))
print('stopping,' + str(counters['stopping']))
print('stopped,' + str(counters['stopped']))
An example output will so be:
status,count
pending,2
running,15
shutting-down,0
terminated,0
stopping,0
stopped,29Generating chart
We want to couple our pie chart with additional information displayed to the user:
- A legend of the various states, each with different colors
- The total number of active instances printed in the center of the pie chart
In this example, we also will cover other possibilities, such as passing the CSV data directly to the JavaScript method as parameter, and position the chart inside a target DOM object.
Our CSV data will be contained in the mycsvData variable, and our target DOM object will be a <svg class=“chart”/> tag that our web application should emit.
Our JavaScript code will be:
// target will contain the DOM object reference that will contain my chart
// e.g. if having <div class="chart" />, target will be ".chart"
mytarget = ".chart";
mycsvData = `status,count
pending,2
running,15
shutting-down,0
terminated,0
stopping,0
stopped,29`;
function myPieChart(target, csvData) {
if (typeof(target) == "undefined" || target=="") target=".chart";
jQuery(target).children().remove();
jQuery(target).html('<div class="chartLoading">Loading...</div>');
// define sizes and slices colors
var width = 250,
height = 300,
radius = Math.min(width, height) / 2;
var color = d3.scaleOrdinal()
.range(["#FC0", "#39C", "#98abc5", "#7b6888", "#a05d56", "#ff8c00"]);
var arc = d3.arc()
.outerRadius(radius - 10)
.innerRadius(radius - 80);
var svg = d3.select(target)
.attr("width", width * 2)
.attr("height", height)
.append("g")
.attr("transform", "translate(" + width / 2 + "," + height / 2 + ")");
// parse data
var parsedData = d3.csvParse(csvData, function(d) {
return {
status:d.status,
count:+d.count
};
});
// calculate the total number of active instances
var sum = d3.sum(parsedData, function(d) { return d.count; });
if (sum == 0)
parsedData = [{ status: "none", count: "100"}];
var pie = d3.pie()
.sort(null)
.value(function(d) { return d.count });
var g = svg.selectAll(".arc")
.data(pie(parsedData))
.enter().append("g")
.attr("class", "arc");
// build pie
g.append("path")
.style("fill", function(d) { if (sum==0) { return "#dddddd" }
else {return color(d.data.status); }})
.transition().delay(function(d, i) { return i * 500; }).duration(500)
.attrTween('d', function(d) {
var i = d3.interpolate(d.startAngle+0.1, d.endAngle);
return function(t) {
if (d.value!=0) {
d.endAngle = i(t);
return arc(d);
}
};
});
if (sum != 0) {
// add legend: color box plus status label and number
var legend = svg.selectAll(".legend")
.data(parsedData)
.enter().append("g")
.attr("class", "legend")
.attr("transform", function(d, i) {
return "translate(0," + i * 20 + ")"; });
legend.append("rect")
.attr("x", width - 18)
.attr("width", 18)
.attr("height", 18)
.style("fill", function(d) { return color(d.status); });
legend.append("text")
.attr("x", width - 24)
.attr("y", 9)
.attr("dy", ".35em")
.style("text-anchor", "end")
.text(function(d) {
if (d.count != 0) { return d.status + " (" + d.count + ")" }
else return d.status
});
}
// print the total number of instances in the pie center
svg.append("svg:text")
.attr("dy", ".70em")
.attr("text-anchor", "middle")
.style("font-size","14px")
.style("font-weight","bold")
.text(sum + " Instances")
}
myPieChart(mytarget, mycsvData);Our final pie chart will be displayed as:
A full example is available as JSFfiddle.
Conclusion
This post walked through a process for displaying data coming from AWS Costs & Usage or other AWS sources using the D3 JavaScript library. These procedures can be quickly added to any web applications requiring simple charts and custom diagrams. Further customization and renderings are also possible by using a wide range of D3 capabilities and other third-party libraries, such as JQuery UI or D3 plugins.