Getting started with the workflow monitor for AWS Media Services

Introduction

One common feature request from customers using AWS Media Services is the ability to visualize and monitor live video workflows. To address this, Amazon Web Services (AWS) introduced the workflow monitor for AWS Media Services and Amazon CloudFront. The workflow monitor provides a visual representation of services and connectivity in a video workload. In addition, the workflow monitor provides a template-based approach to alarms and notifications, including metric alarms based on best practices from AWS engineering and support teams. You can use existing notification mechanisms (via Amazon EventBridge) to alert your team to issues automatically.

Today, the workflow monitor includes support for AWS Elemental MediaConnect, AWS Elemental MediaLive, AWS Elemental MediaPackage, Amazon Simple Storage Service (Amazon S3), and Amazon CloudFront.

In this blog post, we describe how to set up a workflow monitor signal map and alarming for a simple live video streaming workflow. The account we use contains three workflows. Each workflow has a MediaLive channel with two inputs: a file input from Amazon S3 and a live MediaConnect input. There are two MediaConnect flows in use: one flow feeds two of the MediaLive inputs and the second flow feeds the remaining MediaLive input. For simplicity, single pipeline MediaLive channels are used, and each channel creates an adaptive bitrate (ABR) output that is sent to MediaPackage. Each MediaPackage channel has two endpoints for HLS and DASH, which are fronted by a CloudFront distribution.

Access the workflow monitor

The workflow monitor is located in the left navigation pane of AWS Elemental MediaLive in the AWS Management Console, or by going directly to its console landing page. Before you start, make sure you are working in a region and AWS account where you have Media Services resources deployed.

Choose “Workflow monitor” and the welcome screen opens in a new tab:

Build a signal map

First, create a signal map. which is the object that describes a complete workflow. Choose “Create signal map” and supply the required information, including Name and Description:

From the list of resources in the “Discover new signal map” box, select the one that you want to use as your starting point and choose “Create” to begin the discovery process. After a few moments, a signal map appears:

Note that the map includes two of the three MediaLive channels, because both are supplied from the same MediaConnect flow.

Close-up of one channel

The indication by the signal map name shows that the monitor is pending deployment. None of the work done at this point becomes active until the monitor is deployed in a later step.

Define alarm templates

Next, select some appropriate alarm templates. The workflow monitor includes several templates you can import, but you can also define your own alarm thresholds based on any metric emitted by the services in your workflow. Under “Template groups” choose “CloudWatch alarms” and “Create alarm template group”. Then, provide a Group name and optionally a Description for your template group:

Once the group is created, we can either create or import templates. For this exercise, the alarm group includes several pre-defined templates provided with the workflow monitor. Choose “Import” to move to the Import alarm templates page and expand the drop-down menu. The drop-down lists all the AWS provided template groups followed by any groups you have created. For this workflow we select several alarm templates:

1. AWS CloudFront general workflows
2. AWS MediaPackage general workflows
3. AWS MediaConnect ARQ protocol workflows (our MediaConnect flows use Zixi sources)
4. AWS MediaLive to MediaPackage workflows

Each time you select one of the groups from the drop-down, the list of alarm templates is displayed. Choose the alarm templates to include in your new template group and choose “Add”. Repeat for each template group until you have all the alarms you want to use. Then, choose “Import” to add them to the new template group. For now, we include all of the templates in each of the previously listed groups.

Choose “Done” to return to the summary list of your new template group.

Attach template to signal map

Now we can return to the signal map display and attach our alarm template. In the right-hand panel, choose “Attach CloudWatch alarm template groups” and from the following screen select the newly created group, choose “Add” and then “Save”. This returns you to the signal map page, but now the alarm group now appears in the right-hand panel.

EventBridge rules

In addition to metrics-driven alarms, the workflow monitor can help set up targets for EventBridge messages that are emitted by the services in your workflows as well as the alarms generated by the signal map. You can define different targets for different types of messages. In this example, we create a Simple Notification Service (SNS) topic that sends an email whenever the signal map goes into alarm, configured in the “Event type” drop-down:

Following is another example of a rule template that sends EventBridge messages about MediaLive channel input switches to a CloudWatch log group:

Again, we attach this rule template to the signal map as we did for the alarm group:

Deployment

Up to this point, none of the alarms or rules have been deployed, and the signal map isn’t actively monitoring the workflow. With our configuration complete, we are ready to deploy the signal map. From the Actions drop-down menu, choose “Deploy monitor” to begin. Review the information about the resources that will be created in the confirmation box:

Selecting the “Deploy” button starts the process. Behind the scenes, you can see AWS CloudFormation templates being executed to create the resources needed. When complete, the status for the map will change to “Monitor in sync.”

Using the monitor view your workflows

The current view of the signal map is configuration mode, which does not display the active status and alarms for the workflow. To enter monitor mode, you can either choose “Monitor signal map” from the configure page, or you can choose “Overview” from the left-hand menu to see a list of all signal maps and their status:

From this overview, we can see the workflow-1 signal map is in alarm state. Choosing the radio button provides a metrics and logging summary view for the workflow. Choosing the link brings up the detail page for the signal map and allows us to see what service(s) are in alarm:

The CloudWatch alarms panel displays the source of the issue:

The two alarms in MediaLive from the lack of timecode (InputTimecodesPresent) are expected because they are running on a looped file input that does not have timecode. There are also two alarm for pipelines (PipelinesLocked) showing in MediaLive:

This is also to be expected as these are single-pipeline channels. We can remove the noise here by excluding these two templates in the alarm template group. Return to the CloudWatch Alarm template group and delete the individual template items:

Once deleted, re-deploy the signal map to pick up the updates. Once complete, we now see that the alarms are cleared.

EventBridge notifications

Earlier, we configured two EventBridge rule templates to handle signal map alarms and MediaLive input switch events. While we have been operating the channel, the log group target has been collecting entries:

SNS topics can also be used to cause other actions against the messages that match the defined rules. In this case, the SNS topic used sent the EventBridge messages to this log group as well.

Cleaning up

While the workflow monitor itself is provided without charge, the CloudFormation templates it creates and uses will in turn create objects that may have costs associated with them. To avoid ongoing charges for these items, it’s important to clean up any signal maps and templates that aren’t needed.

From the Signal Maps view, choose the radio button next to the map you wish to delete, and choose “Delete”.

1. A prompt appears to confirm deletion of the monitoring resources. Choose “Delete.” The status changes to “NOT_READY” and deployment to “DELETE_IN_PROGRESS”
2. After a few moments and when you refresh the view, the status changes back to “READY” and deployment to “DELETE_COMPLETE”. It’s now possible to again select the radio button for the signal map and choose “Delete” to delete the signal map itself.

At this point, any CloudFormation templates created by the workflow monitor should have been deleted from your account. All EventBridge rules and CloudWatch alarms will now also be deleted. Note that the log streams delivered to CloudWatch Logs will not be deleted by this process. They need to be manually deleted in the CloudWatch console.

Conclusion

This blog post walked through the steps to configure a workflow monitor signal map and create alarm and EventBridge templates. It described and illustrated an alarm condition and diagnosis. After determining the alarm was not due to any actionable issue, we demonstrated how to remove the noisy alarms from the alarm template. We also demonstrated how EventBridge rule templates can be used to capture transient information either from service-generated EventBridge traffic or from alarm states in the signal map. Finally, we demonstrated how to delete templates and signal maps from an account.

To learn more about the workflow monitor, read the User guide. To get started, visit the workflow monitor console.