AWS for M&E Blog
Grabyo optimizes live cloud production with Amazon CloudFront Origin Shield
This blog was coauthored by Mun Wai Kong, CTO, Grabyo and Isaac Martin, VP Engineering, Grabyo.
Live cloud production requires the lowest latency possible to deliver a smooth experience for content producers. Receiving a high quality, jitter-free reference stream is crucial to providing timely edits and ensuring a high-quality end-viewer experience in sports event, live news, and other live clipping use-cases.
Grabyo, an Amazon Web Services (AWS) for Media & Entertainment software competency partner, operates the live production software for some of the major broadcasters and news networks. Their Cloud Video Production and Live Clipping and Editing products run on AWS and are architected to deliver resilient, low-latency ingestion, editing, and distribution.
We’ll explores how Grabyo leverages Amazon CloudFront to provide scalable, low-latency reference feeds to video editors distributed globally with predictable and reliable performance.
Solution overview—cloud production from anywhere
A core value proposition of the Grabyo solution is that a content producer or editor can be anywhere, geographically dispersed from the content they are producing. This is in contrast to traditional production which is done on premises, often at the site of the event. The high-level architecture for the Grabyo Live Clipping and Editing product is shown in Figure 1.
The key components of this architecture are:
- Live video streams are ingested from remote sources through AWS Elemental MediaConnect or Amazon Elastic Compute Cloud (Amazon EC2).
- These video segments are stored in an Amazon Simple Storage Service (Amazon S3) bucket.
- The Grabyo web UI is used by the remote editor to retrieve a near real-time, live feed using the HTTP live streaming (HLS) video segments from Amazon S3 to perform live editing.
- The edited content is delivered through multiple mechanisms (such as Amazon CloudFront and MediaConnect), depending on the distribution channel:
- Social media platforms (such as YouTube, Facebook, X, TikTok).
- Online Video platforms (such as Brightcove, Comcast, JW Player).
- Custom destinations and integrations through Amazon S3, FTP or Media RSS (MRSS).
The challenge
One challenge the Grabyo team had to solve was how to deliver HLS live feeds to their editors located around the globe with low latency and at a low cost. As editors are working with live content, this feed must be delivered with as low latency as possible. Grabyo also had to continue providing a stream of a certain resolution (such as 1080p) and bitrate (between 6 Mbps and 20 Mbps). So, implementing a low rendition in the adaptive bitrate streaming (ABR) stack wouldn’t fulfill this requirement.
Newer technologies like low-latency HLS (LL-HLS) couldn’t be used as editors need both low-latency live viewing and the ability to quickly seek through recent content to create clips. LL-HLS optimizes for live playback latency but doesn’t address the seeking requirements that are crucial for editors’ work.
A content delivery network (CDN) plays a dual role in this workflow. For seeking through recent content, CDN caching is beneficial – editors can quickly access already-cached segments when scrubbing through the timeline. However, for the live viewing portion, the cache-hit ratio is low. This is because live segments are unique and constantly changing – when multiple editors in different locations are viewing the latest live content, each request is for a new segment that hasn’t been cached yet, resulting in origin fetches.
This combination of requirements – low-latency live viewing and quick seeking through recent content – creates a complex delivery challenge. When editors are viewing live content, each new segment requires a round trip to the origin, and the network path variations between different global locations can lead to inconsistent latency. This workflow is illustrated in the following diagram (Figure 2).
Grabyo’s aim is to deliver two second 1080p HLS segments, in under two seconds, to the editor. Using this architecture, different editors saw latency between 2–4 seconds. This was not suitable for their low-latency editing requirements.
CDN optimization with Amazon CloudFront Origin Shield
Grabyo and AWS designed a solution that optimizes Amazon CloudFront performance with Amazon CloudFront Origin Shield.
Amazon CloudFront Origin Shield is a CloudFront feature that adds an additional layer between CloudFront edge locations and the origin. It acts as a regional cache aggregation point, reducing the number of requests that reach your origin while improving cache hit ratios. For live streaming workloads like Grabyo’s, Origin Shield helps by:
- Consolidating requests from multiple edge locations through a single regional shield location. Instead of each edge location making separate requests to the origin, requests are funneled through a regional shield location. For live HLS segments, this means only one request for each segment reaches the origin, regardless of how many users are viewing.
- Reducing the load on the origin (Amazon S3 in this case). Origin Shield creates a hierarchical caching architecture that significantly reduces origin requests. Origin protection is valuable during spike events or when multiple regions access content simultaneously.
- Providing a consistent network path to the origin. Origin Shield establishes and maintains persistent connections to the origin. These connections use optimized AWS network paths, reducing latency and jitter. Network optimization includes transmission control protocol (TCP) connection reuse and keep-alive settings tuned for streaming workloads.
- Improving cache efficiency. The regional consolidation of Origin Shield improves cache hit ratios, even for live content.
- Geographic optimization. Origin Shield locations can be strategically selected based on editor locations and origin placement. This allows for optimal network paths and reduced latency for global deployments.
The consistent network path is key for the editor’s reference stream. Here a long-lived connection with a high keep-alive is maintained between the CloudFront point of presence (PoP) and the Origin Shield. This reduces the TCP connection overhead and the variability in establishing the connection from the PoP to the origin (Figure 3).
For AWS origins, the optimal Origin Shield location is the same region as your origin to minimize latency between the shield and the origin. For example, if the origin S3 bucket is in us-east-1 and editors are primarily in North America and Europe, enabling Origin Shield in us-east-1 provides optimal performance.
Multiple Origin Shield locations can be configured for global redundancy, but this needs to be balanced against the potential for reduced cache hit rates.
This solution led to improved glass-to-glass latency for Grabyo editors, reducing the p99 segment retrieval time to under two seconds. To monitor the results, Grabyo created a daily dashboard through Amazon QuickSight, using the time-taken (time-to-last-byte) metric from the CloudFront access logs (Figure 4).
The results
The results show that in all regions over 99 percent of requests were delivered in less than two seconds, with the majority of regions delivering 99.99 percent of requests under this threshold. This allows Grabyo to meet their two second target for each two second 1080p video segment, and make a significant improvement over the direct fetches from the origin.
To confirm optimal performance of the Origin Shield implementation, several key metrics are monitored using Amazon CloudWatch and CloudFront access logs. Critical metrics include cache hit ratio and origin latency (Time-to-first-byte, HTTP status codes, and request count).
Setting up CloudWatch dashboards to track these metrics helps Grabyo identify potential issues before they impact an editor’s experience. Grabyo implements automated alerting based on these metrics to verify a rapid response to any performance degradation.
Overall, by implementing CloudFront with Origin Shield, Grabyo achieved the following improvements in their live production workflow:
- Reduction in p99 to less than two seconds.
- More predictable performance with consistent network paths.
- Better scaling capabilities for global editors.
- Reduced origin load and associated costs.
- Improved editor experience with lower glass-to-glass latency.
By moving to CloudFront, Grabyo can now also take advantage of edge-compute such as CloudFront Functions to perform additional authorization and routing logic.
Conclusion
The implementation of Amazon CloudFront with Origin Shield has enhanced Grabyo’s cloud-based live editing platform. It enables them to deliver high-quality, low-latency video streams to editors worldwide
As the media and entertainment industry continues to evolve towards more distributed and cloud-based production models, solutions like Grabyo’s showcase the potential for innovation in this space. By continually optimizing their architecture and leveraging the latest AWS technologies, Grabyo is well-positioned to meet the growing demands of modern content creation and delivery.
Check out more AWS Partners or contact an AWS Representative to know how we can help accelerate your business.