Enhancing video operations with Datazoom and Amazon CloudFront

This post was co-authored by Kerry Travilla, VP Technology, Datazoom.

This blog explores how to gain video player and CDN observability by integrating Datazoom’s video data platform with Amazon CloudFront. The solution enables real-time monitoring of client-side player data correlated with CDN performance data, helping video operations reduce time to resolution.

The first step towards achieving video observability is to measure the end-user quality of experience (QoE) in order to understand if users are being negatively affected by system performance. Then add the capability to monitor and manage the performance of processing functions in the workflow and understand the factors that impact the streaming experience. The processing functions include transcoding, packaging, and content delivery networks (CDN).

Datazoom provides a Data-as-a-Service platform which continually gathers data from endpoints (including video players, CDNs, Origin Servers, and server-side ad insertion (SSAI) services) to create a normalized event-stream which adheres to standardized data definitions. Real-time collection of client application behavior and video events can be achieved by integrating pre-scripted Collectors (scripts, SDKs, tags). Amazon CloudFront, a CDN from Amazon Web Services (AWS), offers near real-time logging to Amazon Kinesis. Kinesis allows you to gain detailed insights and process requests in near real-time.

To enable observability, establish a join key between data sets—in this case, between the video player and CDN. Both Datazoom and CloudFront support the Common Media Client Data (CTA 5004) standard, which logs CDN data from the player including a session ID (SID). Datazoom uses the SID to create a join key when the datasets are combined (Figure 1). By routing both datasets to the same destination, and implementing the Datazoom platform, customers using a single CDN like CloudFront can achieve near real-time observability between CDN performance and viewing experience impacts.

Customers in multi-CDN scenarios can leverage Datazoom to actively capture, standardize, and provide smart-sampling of logs across multiple CDNs before analysis. Combining these two datasets is an important step towards observability across the complete delivery infrastructure.

Figure 1: CMCD Block Diagram.

In a previous blog we described how to set up active data collection for the application and video player to achieve near real-time QoE observability. The example used uses Amazon Interactive Video Service (Amazon IVS) video player with data sent to Kinesis-Athena-Grafana for visualization. To enhance observability, we now need to collect additional information from CDN. We use the media request events emitted by the video player to correlate upstream CDN logs. Instructions on how to instrument media object requests and CloudFront CDN log files can be found in “Amazon CloudFront Log Collector”.

By capturing correlated video player events and CDN log files, engineering teams gain better insights into user-impacting failures. Figure 2 illustrates three time-correlated spans obtained from real-time data: player events (play request, stall/buffering, and so on), media object requests (manifest, video object, audio object), and CDN log responses. The figure shows a tracing view created in an analytics tool using queries on the two datasets ingested. Each media request contains a request ID (RID) which is attached to the CDN log lines. This information, along with timestamps, are used to create time correlated visualizations to quickly perform a root cause analysis.

Let’s break down what is shown in the Figure 2 example. The top row are events emitted by the video player. The second row are the media object requests coming from the video player to the CDN (these have been color coded by object type like audio segment or video segment). The third row are the CDN log lines showing the time taken to respond to the media object requests. The yellow buffering line in the top row indicates a video player stall (usually indicated by a spinning circle waiting for data). We can see that the CDN received a request for a video segment (the blue line in the second row) and the resulting response (the gray line in the third row) shows a delayed object response from the CDN. This delayed response is the cause of the buffering for the user.

Figure 2 Player, Media Object, and CDN trace.

Beyond a distributed tracing view, other signals and metrics are used by engineers to further diagnose poor QoE for a user.

These dimensions are automatically collected by the Datazoom video player SDK:

• Bitrate (Average) – Insight into current bitrate, maximum bitrate, constraints from network, device, geography, ISP, content titles, or origin servers.
• Time to First Frame (Average) – Insight into the time to first frame, including potential causes such as slow CDN processing, content served from mid-tier or origin instead of the edge, and any relationship to geography, ISP, content title, device type, or bitrate.
• Stall Ratio (Average) – Visibility into stall ratio, enabling analysis of potential causes such as changes in bandwidth, throughput, or shifts in the location serving the content chunks.

If there’s a state change present in both correlated data sets, we can determine the root cause of the degraded experience. If a correlated state change is not present, more data is needed, potentially from other parts of the workflow to establish causation.

By bringing player and CDN data together we are linking the quality of service (performance of the video streaming infrastructure) with the quality of experience (what the user is perceiving). This moves from inference to root cause analysis (RCA). Correlating data from each contributing element of the video streaming workflow results in actionable insights. These insights can be shared both with internal stakeholders and with external partners, such as your CDN partner(s), to reduce time to resolution (TTR). Reducing the TTR leads to less disruption for the customer—creating a better perception of the service by the customer and data-driven decisions for improving the video streaming workflow.

To improve video streaming service observability using real user data together with CDN log files visit Datazoom on the AWS Marketplace. Or visit www.datazoom.io to learn more about their industry leading data solutions.

Contact an AWS Representative to learn how we can help accelerate your business.

Further reading

• Amazon CloudFront Logging and monitoring in Amazon CloudFront
• Improving video observability with CMCD and CloudFront blog
• Troubleshooting CloudFront performance
• Web Application Video Ecosystem – Common Media Client Data

AWS Partner spotlight

Datazoom is an enterprise technology company offering a novel approach to the collection, processing, and delivery of consumer application and digital experience data. Our cloud-based Data-as-a-Service Platform offers unique capabilities to support industries that leverage media and advertising, and is fully built on AWS. Publishers, media platforms, measurement companies and other technology vendors are our Customers.