How can I push custom metrics to CloudWatch?

Last updated: 2021-06-21

How can I push custom metrics to Amazon CloudWatch?

Short description

AWS services push data points to CloudWatch by default. However, there might be instances where you must calibrate the performance of your resources based on metrics that aren't supported by AWS services. In these cases, you can push custom metrics to CloudWatch using the unified CloudWatch agent or the API.

Important: Custom metrics are charged according to their storage and API use.

Resolution

Push custom metrics using the CloudWatch agent

The unified CloudWatch agent collects system-level metrics and sends them to CloudWatch as custom metrics. You can use the agent to push custom metrics from:

  • Linux or Windows servers
  • Amazon Elastic Compute Cloud (Amazon EC2) instances or on-premises servers

For Linux, refer to the list of supported metrics.

Sample agent configuration file metric block for disk metrics (Linux):

"disk": 
    {
        "measurement": [
          
          "used_percent"
        ],
        "resources": [
          
          "*"
        ],
        "drop_device": 
        true
      }

For Windows, you can reference any counter mentioned in the Windows Performance Monitor in the agent configuration file.

Sample agent configuration file metric block for Processor Counter (Windows):

"Processor": {
        "measurement": [
          {"name": "% Idle Time", "rename": "CPU_IDLE", "unit": "Percent"},
          "% Interrupt Time",
          "% User Time",
          "% Processor Time"
        ],
        "resources": [
          "*"
        ],
        "append_dimensions": {
          "d1": "win_foo",
          "d2": "win_bar"
        }
      }

You can also use "StatsD" and "collectd" protocols to retrieve custom metrics from your applications or services. Then, the metrics are pushed through the agent. StatsD is supported on both Linux and Windows servers. collectd is supported only on Linux servers. To use these protocols:

1.    Install the CloudWatch unified agent.

2.    Assign a role or credentials to the instance with CloudWatch permissions.

3.    Create the CloudWatch agent configuration file.

4.    Start the agent.

Push custom metrics using PutMetricData

If your use case doesn't support the use of the CloudWatch unified agent, you can use the PutMetricData API to push custom metrics to CloudWatch.

For example, to push connections for a specific port as a custom metric, you can retrieve the values locally and transfer them in the API:

total_conn=$(netstat -an | grep <port> | wc -l) 

aws cloudwatch put-metric-data   --namespace "totalconn"   --metric-name <port> --dimensions Instance=<InstanceId> --value $  total_conn

Similarly, you can use AWS SDKs to use the PutMetricData API and send custom metrics to CloudWatch. Create a local script that runs periodically to send custom metrics. The API is optimal for pushing different metrics and multiple values for the following reasons:

  • You can use up to twenty different metrics in a single API.
  • Using the Values and Counts method, you can publish up to 150 values per metric with one PutMetricData request.
  • You can use up to ten dimensions per metric.
  • Each PutMetricData request is limited to 40 KB for HTTP POST requests.

For example, instead of making a separate API call for each metric, you can push multiple metrics using a single API call:

aws cloudwatch put-metric-data --namespace "Usage Metrics" --metric-data file://metric.json

where metric.json is:

[
  {
    "MetricName": "DiskMetric",
    "Value": <value_derived_by_some_calculation>,
    "Unit": "Count"
  },
  {
    "MetricName": "MemoryMetric",
    
    "Value": <value_derived_by_some_calculation>,
    "Unit": "Count"
  }
]

This approach typically reduces costs because custom metrics are charged based on the number of APIs and storage. This approach might also help reduce throttling on the PutMetricData API.


Did this article help?


Do you need billing or technical support?