Monitoring AWS Elemental MediaStore with Amazon CloudWatch metrics

Many customers have built live streaming workflows using AWS Elemental MediaStore as a high performant live video origin in the cloud. For example, FOX recently used MediaStore as a low latency origin for its AWS-enabled production workflow for Super Bowl LIV.

In a previous blog post, I explained how to use MediaStore access logs to build monitoring dashboards. As MediaStore now supports Amazon CloudWatch metrics, this post describes how you can use CloudWatch metrics to build live streaming dashboards to monitor MediaStore performance.

The CloudWatch metrics used in this solution include RequestCount, 4xxErrorCount, 5xxErrorCount, BytesUploaded, BytesDownloaded,TotalTime and TurnaroundTime.

Set up

Review the following resources to familiarize yourself with AWS Elemental MediaStore, Amazon CloudWatch metrics and access logs, and how to send live video to MediaStore.

Solution walkthrough

First, you will create a live channel using MediaStore as an origin and start a player to play the live stream directly from MediaStore to generate egress traffic. Next, you attach the MediaStore metric policy to your container and make sure the folder names are match your configuration correctly.

Prerequisites

For this walkthrough, you should have the following prerequisites:

An AWS account
An Encoder such as AWS Elemental Live that can push live stream to MediaStore, or an AWS Elemental MediaLive service to stream to MediaStore
An HLS player that can play HLS streams

MediaStore and Live Stream Setup

Choose the MediaStore service from the management console, and create a container named ‘mediastore_live’. Under Containers, select the ‘mediastore_live’ container to display the container details property interface.

To play live stream directly from MediaStore and enable HTTP/S access, scroll to the Container policy section and click Edit policy. Copy and paste the following code in the editing text box. Be sure to modify the Resource ARN and region to match your container settings. Once finished, click Save.

{
“Version” : “2012-10-17”,
“Statement” : [ {
“Sid” : “MediaStoreFullAccess”,
“Effect” : “Allow”,
“Principal” : {
“AWS” : “arn:aws:iam::xxxxxxxxxxxx:root”
},
“Action” : “mediastore:*”,
“Resource” : “arn:aws:mediastore:us-west-2:xxxxxxxxxxxx:container/mediastore_live/*”,
“Condition” : {
“Bool” : {
“aws:SecureTransport” : “true”
}
}
}, {
“Sid” : “PublicReadOverHttpOrHttps”,
“Effect” : “Allow”,
“Principal” : “*”,
“Action” : [ “mediastore:GetObject”, “mediastore:DescribeObject” ],
“Resource” : “arn:aws:mediastore:us-west-2:xxxxxxxxxxxx:container/mediastore_live/*”,
“Condition” : {
“Bool” : {
“aws:SecureTransport” : [ “true”, “false” ]
}
}
} ]
}

To enable MediaStore to send CloudWatch metrics, scroll down to Metric policy, and click Edit metric policy. Copy and paste the following metric policy and click Save button.

{

“ContainerLevelMetrics”: “DISABLED”,

“MetricPolicyRules”: [

{

“ObjectGroup”: “msmetric1/”,

“ObjectGroupName”: “msmetric1”

{

“ObjectGroup”: “msmetric2/”,

“ObjectGroupName”: “msmetric2”

}

]

}

Note the prefix name mnmetric1 and msmetric2. These prefix names are used in the AWS Elemental MediaLive configuration in the output path.

Scroll to the Container CORS policy section and click Edit CORS policy. Copy and paste the following CORS policy and click Save.

[
{
“AllowedHeaders”: [
“*”
],
“AllowedMethods”: [
“GET”,
“HEAD”
],
“AllowedOrigins”: [
“*”
],
“ExposeHeaders”: [
“*”
],
“MaxAgeSeconds”: 3000
}
]

Next, configure a live stream from AWS Elemental MediaLive to AWS Elemental MediaStore. Use the following URLs to configure HLS group destination A and HLS group destination B in MediaLive.

HLS group destination A: mediastoressl://<MediaStore data end point URL>/msmetric1/master, HLS group destination B: mediastoressl://<MediaStore data end point URL>/msmetric1/master

To learn more, review this blog post: How to Send Live Video to AWS Elemental MediaStore

Stream Ingest Dashboard

One of the most important aspects when monitoring a live stream from an origin perspective is the health status of the stream ingest into the origin. Now you will create dashboards by using MediaStore ingest CloudWatch metrics.

From your AWS Account Console, choose the CloudWatch service. Select Metrics in the left-side menu. Under the All metrics tab, choose MediaStore and then select Data Transfer Metrics by Object Group.

Under the ContainerName column, choose mediastore_live container. Under the ObjectGroupName column, choose msmetric1. For Metric Name, choose BytesUploaded.

Repeat this step and select the following:

ContainerName: mediastore_live
ObjectGroupName: msmetric2
Metric Name: BytesUploaded

Select MediaStore CloudWatch metric BytesUploaded for ingest dashboard

Select the Graphed metrics tab. Change the Statistic value to Average and change the Period value to 5 Minutes. On the top left corner of the diagram, change the name to Bytes Upload.

Change dashboard metric graphic options and create a dashboard name

Click the Actions button and select Add to dashboard from the dropdown menu. In the Add to dashboard popup that appears, choose Create new under the Select a dashboard section. Name the dashboard mediastore_live_cw. Then click Add to dashboard at the bottom of the window.

Next, choose Metrics from left menu. Select the All metrics tab, click MediaStore followed by Request Metrics by Object Group. Input the metrics referenced in the following table:

ContainerName	ObjectGroupName	RequestType	Metric Name
mediastore_live	msmetric1	DeleteRequests	5xxErrorCount
mediastore_live	msmetric1	PutRequests	5xxErrorCount
mediastore_live	msmetric1	DeleteRequests	4xxErrorCount
mediastore_live	msmetric1	PutRequests	4xxErrorCount
mediastore_live	msmetric2	DeleteRequests	5xxErrorCount
mediastore_live	msmetric2	PutRequests	5xxErrorCount
mediastore_live	msmetric2	DeleteRequests	4xxErrorCount
mediastore_live	msmetric2	PutRequests	4xxErrorCount

Choose Graphed metrics tab. Set the Statistic value to Sum and set the Period value to 5 Minutes.

To show a value of 0 in the dashboard when there are no errors, choose Math expression in the header of the table. Select All Functions and select FILL. This will add a math function to the top of the table.

Choose the Graph options tab, and, in Widget type section, select Number. Input ‘Ingest Error’ in the top left corner of the dashboard. Click the Actions button and select Add to dashboard from the dropdown menu. In the Add to dashboard dialog that appears, under the Select a dashboard section, choose mediastore_live_cw from the dropdown menu. When finished, click Add to dashboard at the bottom of the window.

This dashboard will help keep track of ingest errors. An increasing number of 4xx and 5xx errors should warrant a further investigation on stream ingest.

A screenshot of MediaStore CloudWatch dashboard monitor ingest 5xx and 4xx error counts

Using a similar process, you can create dashboard to monitor MediaStore PUT requests counts and delete request counts. These number should be relatively similar between pipelines and stay stable.

A screenshot of MediaStore CloudWatch dashboard monitor put request counts

A screenshot of MediaStore CloudWatch dashboard monitor delete request counts

The last metrics for MediaStore ingest monitoring are the PutRequest TotalTime and PutRequest TurnaroundTime. In this example, you can set the Statistic value to p99, and set the Period value to 1 Minute.

A screenshot of MediaStore CloudWatch dashboard monitor PutRequest TotalTime and PutRequest TurnaroundTime

Next, rearrange your dashboard to have a clear view of the MediaStore ingest operations based on CloudWatch metrics.

A screenshot of MediaStore CloudWatch ingest monitoring dashboard

Following the same principles, you can also monitor MediaStore egress performance. Choose the following MediaStore metrics to graph your egress monitoring dashboard: BytesDownloaded, GetRequests TotalTime, GetRequests TurnaroundTime, GetRquest RequestCount and GetRequests 4xxErrorCount.

Unify your dashboards into one view with the ingest dashboard to the left and the egress dashboard to the right. We now have a full high-level view of your origin’s performance.

A screenshot of MediaStore CloudWatch ingest and egress monitoring dashboard

Clean up

To avoid incurring future charges, delete the MediaStore container and MediaLive services you created as part of this example.

Conclusion

This blog explained how to use CloudWatch metrics to create live streaming dashboards to track MediaStore performance and operations. Using CloudWatch metrics to monitor MediaStore origin performance provides comprehensive information on both ingress and egress operations. When MediaStore is used as a live streaming origin, these metrics help you create custom dashboards to monitor your MediaStore services, assess operational health and performance, and easily to identify potential issues in the workflow. Moreover, with support for object groups, you can pull unique metrics for each channel within the same MediaStore container, as long as each channel has a unique prefix.

To learn more about MediaStore dashboarding, check out this blog: Create a Live Streaming CloudWatch Dashboard using AWS Elemental MediaStore Access Logs

AWS for M&E Blog