Master control and distributed remote production on AWS with GV AMPP from Grass Valley

This blog was co-authored by Mark Stephens – Senior Partner SA, Media & Entertainment at Amazon Web Services, Mike Cronk – Vice President, Advanced Technology at Grass Valley, and Boromy Ung – Vice President, Product Marketing at Grass Valley.

Live event and video production in the cloud is here. Grass Valley’s GV AMPP, the Agile Media Processing Platform, is a virtual control room and master control solution that runs on Amazon Web Services (AWS). Since February, Blizzard has used AMPP to deliver global professional esports tournaments for the Overwatch League (OWL) and the Call of Duty League (CDL), and since the end of March, they have done so in a completely remote fashion. All necessary functions for production and master control are virtualized, allowing the entire production crew managing these events to work from home, without any physical switchers or audio consoles.

Grass Valley enables its customers a high level of sophistication for live production in the cloud, while providing purpose-built interfaces for every operator function, even as these functions are fully distributed. There is no required upfront investment in hardware infrastructure, the solution is fully elastic, and the result is a broadcast-grade live production, remotely operated and up and running in minutes.

An accelerated timeframe

As California took steps to keep residents and businesses safe, Blizzard started to think differently while Grass Valley accelerated several features to assist a remote workflow. In a traditional home and away model, production is performed locally (using trucks, for example) and the program feed for each location is sent to the cloud for switching using GV AMPP Master Control. In a new model that adheres to new California safety guidelines, all feeds are sent straight to the cloud where the production is performed for every match-up using GV AMPP for distributed remote production. Then, the GV AMPP Master Control is used for master control switching and regionalization

How AMPP works

GV AMPP is a cloud-based SaaS platform for broadcast that fully leverages the power of elastic compute. Built for the cloud, GV AMPP is designed to overcome broadcasters’ reliance on hardware-based media systems. Using just a browser on a laptop, the solution gives customers the flexibility to spin up or spin down applications as needed — and only pay for the services they use — even in live production environments.

With the GV AMPP Master Control application, customers can create configurable virtual master control rooms accessible via web-based interface from anywhere in the world. GV AMPP meters usage and each tool—switcher, audio mixer, multiviewer, clip player—has a different metered rate associated with it. The user is charged only for the features activated and the amount of time each feature is used. When an event is over, the user can take a snapshot of their setup so that they no longer incur cost while retaining the ability to spin it up before the next event.

GV AMPP allows multiple distribution streams with separate graphics and languages for each audience region produced by a single operator. Monitoring and local program-distribution processes take place completely in the cloud.

Master control and virtual control rooms

Master control is the technical hub of a broadcast operation.  A master control room is used to switch between feeds coming from different production control rooms (PCRs) and other pieces of content, such as clips (e.g. commercials). Moreover, the function of master control may also include insertion of other branding elements (logos, lower thirds) and regionalization of content. For example, Blizzard broadcasts in four languages: English, French, Korean, and Chinese. Each broadcast contains localized commercials, branding and graphics. Master control leverages four G3.16xl GPU-based Amazon Elastic Compute Cloud (Amazon EC2) instances for full redundancy.

A PCR is where the activities such as switching from camera to camera, audio mixing, and production graphics are generated. Blizzard has several PCRs, all completely virtualized for casters, analysts, and observer stations. The user interfaces for these PCR stations are distributed to each operator, who is working from their residence. These PCR stations run on G3.16xl instances as well.

The GV AMPP AWS architecture

GV AMPP was designed from the ground up for high security and availability. It requires authentication, uses private subnets, and is behind an Elastic Load Balancer. In the following diagram, the orchestration layer of the system runs across three AWS Availability Zones. Currently the system ties into native AWS services such as Amazon Elasticsearch Service and Amazon ElastiCache. Other services such as MongoDB, RabbitMQ and Consul are deployed via Amazon Elastic Kubernetes Service (Amazon EKS) to EC2 instances (Linux T2 and C5s).

From a data plane perspective, Grass Valley has qualified G3.16xl EC2 instances. Customers deploy GV AMPP apps via the GV AMPP resource manager on these instances to perform the video and audio processing required for live broadcast production. These instances provide substantial CPU and GPU cores to handle the intensive real-time video processing required. For example, on a G3.16xl, depending on system configuration, customers can typically run up to eight decodes of 1080p H.264 at 59.94 frames per second while encoding multiple outputs. This also includes the capacity to support multiviewing, two players for clip playback, multiple graphics renderers, low latency web monitoring, and video egress – all on a single instance.

The production crew

The job roles for Blizzard’s at-home productions are the same as onsite with a truck, with one major difference: a single producer takes on the roles of director, producer, and technical director, as well as monitoring the audio levels within the GV AMPP interface. Each Overwatch League and Call of Duty League production also features one or two graphics operators, one or two replay operators, and an operator for the clip player (for playback of music or video clips). There is also a six-person observer team (five observers and one in-game director), along with casters and talent. Blizzard uses GV AMPP for OWL matches in North America and Asia, and the operation runs around the clock.

With GV AMPP on AWS, an entire production, including production control rooms and master control rooms, can be ready to go live in a matter of minutes. All of the GV AMPP applications and their state are preserved when the G3 instances spin down, and when they spin up again, the apps, their configuration, and their signal flows come right back to life.

Click here to learn more about Grass Valley’s GV AMPP platform.