Amazon CloudWatch Prometheus metrics now generally available

Imaya Kumar Jagannathan, TP Kohli, and Michael Hausenblas

In Using Prometheus Metrics in Amazon CloudWatch we showed you how to use the beta version of the Amazon CloudWatch supporting the ingestion of Prometheus metrics. Now that we made this feature generally available we explore its benefits in greater detail and show you how to use Prometheus in the context of Amazon ECS, our native container orchestrator.

While Prometheus has its root in the Kubernetes ecosystem, it’s by no means limited to it. The community around Prometheus has found interesting use case: from monitoring temperature on a Raspberry Pi to its role in service meshes; we encourage you to learn more about its origins and applications areas.

For a general introduction to CloudWatch Container Insights (CI) monitoring for Prometheus, we recommend perusing our docs. Before we dive into the topic of how to use Prometheus metrics in CloudWatch from ECS, let us first discuss why you would want to do that.

Benefits of Prometheus metrics with CloudWatch

No matter the container orchestrator you’re running your containerized microservices on, be it Amazon EKS or ECS, using the CloudWatch CI support for Prometheus metrics offers a number of benefits, some of which we will demonstrate in the following:

• You get a range of out-of-the-box dashboards.
• You can use custom metrics.
• The ability to create alarms with a deep integration with other AWS services such as SNS.
• The ability to configure, add and drop new metrics and dimensions.
• Especially for EKS, easily set up control plane monitoring.

In addition, more generally speaking, using Prometheus enables you a smooth and flexible migration path from on-premises deployments. You benefit from a rich open source ecosystem in terms of exporters and have the freedom of choice where, how, when to consume your metrics based on an open industry standard.

Out-of-the-box dashboards and more …

You can also make use of the out-of-the box dashboards provided by CloudWatch Container Insights for workloads such as AWS AppMesh and Java/JMX. Using this, you are now not only able to quickly setup Prometheus metric monitoring on CloudWatch but also get deeper insights into these workloads without having to create a dashboard from the scratch.

You can also add the widgets shown on the built-on dashboards to a custom dashboard of your choice as well by simply clicking on the Add to dashboard button.

The screenshot below shows the built-on dashboard for AWS AppMesh workload hosted on an ECS cluster with CloudWatch Container Insights enabled.

In action: using Prometheus in an ASP.NET app

In this following setup we will instrument an ASP.NET Core application using Prometheus client libraries with the goal to expose custom metrics and ingest these metrics into CloudWatch. We will do this using the CloudWatch Prometheus agent with a custom configuration.

Instrument app for custom metrics

First, clone the sample application from aws-samples/amazon-cloudwatch-prometheus-metrics-sample and have a look at the HomeController.cs file:

// Prometheus metrics:
private static readonly Counter HomePageHitCounter = Metrics.CreateCounter("PrometheusDemo_HomePage_Hit_Count", "Count the number of hits to Home Page");

private static readonly Gauge SiteVisitorsCounter = Metrics.CreateGauge("PrometheusDemo_SiteVisitors_Gauge", "Site Visitors Gauge");

public IActionResult Index() {
            HomePageHitCounter.Inc();
            SiteVisitorsCounter.Set(rn.Next(1, 15));
            return View();
}

As well as the ProductsController.cs file:

// Prometheus metric:
private static readonly Counter ProductsPageHitCounter = Metrics.CreateCounter("PrometheusDemo_ProductsPage_Hit_Count", "Count the number of hits to Products Page");

public IActionResult Index(){
            ProductsPageHitCounter.Inc();
            return View();
}

The code snippets shown above instrument three different metrics to track the number of visitors to each page and overall visitors in general using an open source Prometheus client library.
Next, for local testing and preview, navigate to the directory where the Dockerfile is located. Build the container image and run it using the following commands:

docker build . -t prometheusdemo
docker run -p 80:80 prometheusdemo

Now navigate to localhost where you should be able to see a screen like the one below. Click on the Home and Products links a few times to generate some traffic:

Next, navigate to the http://localhost/metrics where you should see all the Prometheus metrics the app is exposing via the /metrics endpoint:

Set up CloudWatch agent to discover Prometheus metrics

Open the ecs-prom-cwagent-config.yaml file under /ecs folder in the repo. You will find that the config file creates an SSM parameter with a JSON as its value. The CloudWatch Prometheus agent can be configured to perform service discovery either using Docker labels or using ECS Service APIs or the combination of both at the same time. In this example application, we will be configuring the agent to perform service discovery using ECS Service. Look for the service discovery configuration under task_definition_list section in the configuration JSON. As shown below, we are configuring the agent to scrape metrics from /metrics URL on port.

{
    "sd_job_name": "petsite-webapp",
    "sd_metrics_ports": "80",
    "sd_task_definition_arn_pattern": ".*:task-definition/Servicespetsite.*:[0-9]+",
    "sd_metrics_path": "/metrics"
}

The Prometheus scrape configuration below shows the list of metrics that we want to scrape and the dimensions under which these metrics will to be aggregated. The metrics will be gathered from log events that have the label container_name matching with the value container.

{
    "source_labels": ["container_name"],
    "label_matcher": "container",
    "dimensions": [["ClusterName","TaskDefinitionFamily"]],
    "metric_selectors": [
    "^process_cpu_seconds_total$",
    "^process_open_handles$",
    "^process_virtual_memory_bytes$",
    "^process_start_time_seconds$",
    "^process_private_memory_bytes$",
    "^process_working_set_bytes$",
    "^process_num_threads$"
    ]
 },
 {
    "source_labels": ["container_name"],
    "label_matcher": "container",
    "dimensions": [["ClusterName","TaskDefinitionFamily"]],
    "metric_selectors": [
    "^dotnet_total_memory_bytes$",
    "^dotnet_collection_count_total$",
    "^dotnet_gc_finalization_queue_length$",
    "^dotnet_jit_method_seconds_total$",
    "^dotnet_jit_method_total$",
    "^dotnet_threadpool_adjustments_total$",
    "^dotnet_threadpool_io_num_threads$",
    "^dotnet_threadpool_num_threads$",
    "^dotnet_gc_pinned_objects$",
    "^dotnet_gc_allocated_bytes_total$"
    ]
 },
 {
    "source_labels": ["container_name"],
    "label_matcher": "container",
    "dimensions": [["ClusterName","TaskGroup"]],
    "metric_selectors": [
    "PrometheusDemo_SiteVisitors_Gauge",
    "PrometheusDemo_HomePage_Hit_Count",
    "PrometheusDemo_ProductsPage_Hit_Count"
    ]
 }

OK, time to move on to the agent setup.

Deploy CloudWatch Prometheus agent

We want to deploy the CW agent into an ECS cluster and for that, create an ECS on Fargate as per our docs.

Ensure you replace the values for the environment variables used to suit your needs. Execute the following commands from the /ecs folder on the repo. The following commands will enable CloudWatch Container Insights on the ECS cluster and also setup the CloudWatch Prometheus agent to scrape metrics from the application’s metrics endpoint.

export AWS_PROFILE=<your_aws_config_profile_eg_default>
export AWS_DEFAULT_REGION=<your_aws_region_eg_ap-southeast-1>
export ECS_CLUSTER_NAME=<your_fargate_ecs_cluster_name>
export ECS_LAUNCH_TYPE=FARGATE
export CREATE_IAM_ROLES=True
export ECS_CLUSTER_SECURITY_GROUP=<your_security_group_eg_sg-xxxxxxxxxx>
export ECS_CLUSTER_SUBNET=<your_subnet_eg_subnet-xxxxxxxxxx>

aws ecs update-cluster-settings --cluster ${ECS_CLUSTER_NAME} --settings name=containerInsights,value=enabled

aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-${ECS_LAUNCH_TYPE}-awsvpc \
    --template-body file://ecs-cwagent-config.yml \
    --parameters ParameterKey=ECSClusterName,ParameterValue=${ECS_CLUSTER_NAME} \
                 ParameterKey=CreateIAMRoles,ParameterValue=${CREATE_IAM_ROLES} \
                 ParameterKey=ECSLaunchType,ParameterValue=${ECS_LAUNCH_TYPE} \
                 ParameterKey=SecurityGroupID,ParameterValue=${ECS_CLUSTER_SECURITY_GROUP} \
                 ParameterKey=SubnetID,ParameterValue=${ECS_CLUSTER_SUBNET} \
                 ParameterKey=TaskRoleName,ParameterValue=CWAgent-Prometheus-TaskRole-${ECS_CLUSTER_NAME} \
                 ParameterKey=ExecutionRoleName,ParameterValue=CWAgent-Prometheus-ExecutionRole-${ECS_CLUSTER_NAME} \
    --capabilities CAPABILITY_NAMED_IAM \
    --region ${AWS_DEFAULT_REGION} \
    --profile ${AWS_PROFILE}

Verify that the services are installed correctly by executing:

aws ecs list-services --cluster ${ECS_CLUSTER_NAME}

You should see a result similar to the one below which indicates that there are two services running on the cluster:

{
 "serviceArns": [
 "arn:aws:ecs:us-west-1:001234567890:service/promdemo",
 "arn:aws:ecs:us-west-1:001234567890:service/cwagent-prometheus-replica-service-FARGATE-awsvpc"
 ]
}

Prometheus metrics in the CloudWatch Metrics console

Go to CloudWatch Metrics and you will be able to see two custom namespaces called ECS/ContainerInsights and ECS/ContainerInsights/Prometheus. The namespaces contain performance metrics and Prometheus metrics from the ECS cluster:

Inside the ECS/ContainerInsights/Prometheus namespace you will find two dimensions as depicted in the below screen shot. Based on our agent configuration, 18 metrics are being collected from the application environment:

The following screenshot shows how you can graph custom Prometheus metrics from the application:

You can also navigate to the Container Insights page and see the default dashboard automatically created for the ECS cluster:

As we pointed out in the beginning, the CloudWatch support for Prometheus metrics is both relevant for ECS and EKS. While we spent most of the time in this post on ECS, here are two more topics around EKS that may be of interest for you:

• Automating the scaling of your Kubernetes workloads can be tricky and if you want to learn more about good practices in this realm we recommend to read Viji Sarathy’s blog post about Autoscaling Amazon EKS services based on custom Prometheus metrics using CloudWatch Container Insights.
• Another EKS topic that often is a requirement, especially for regulated verticals, is Kubernetes control plane metrics scraping.

Conclusion

In this article we discussed the benefits of the CloudWatch support for Prometheus metrics. We demonstrated some of the key benefits in the context of ECS clusters, such as out-of-the-box dashboards or customizing agent configurations to scrape metrics from an app emitting metrics in the Prometheus exposition format. You can leverage this feature by ingesting Prometheus metrics from ECS or EKS clusters into CloudWatch and make use of the powerful observability features that CloudWatch offers while relying on an open industry standard.