Deluxe enables same day global theatrical delivery using AWS Snowcone

Deluxe Entertainment Services Inc. (Deluxe) is a world-leading video creation and distribution company offering global, end-to-end services and technology for the media and entertainment market. The largest content creators, broadcasters, OTTs, and distributors around the globe rely on Deluxe’s scale, technology, and capabilities to enable the worldwide market for premium content.

Deluxe’s latest innovation for theatrical experiences comes with the launch of One VZN, a new cloud-based IP content delivery solution leveraging AWS Snowcone for edge compute, data storage, and transport. Snowcone is a portable, ruggedized, and secure edge computing and data transfer device, and it is the smallest member of the AWS Snow Family. One VZN also uses other AWS services to quickly and securely distribute major studio and independent content to movie theaters and exhibitors worldwide. One VZN provides a fully AWS cloud-based management and delivery system for Digital Cinema Packages (DCP) by building a dedicated network to theaters. This enables Deluxe to distribute DCPs directly to Snowcones at theatrical endpoints, bypassing satellites and eliminating the need for hard drives.

One VZN is an enhancement to the Deluxe One platform, the company’s flagship cloud-based solution that unifies the media supply chain. One VZN leverages Deluxe’s 100+ years of experience providing high-quality, reliable theatrical services to the media and entertainment industry. The updated platform represents a continued investment in theatrical content delivery and support for the resurgence of theatrical viewing by consumers.

In this blog, we discuss the technical transformation of the movie industry and why an evolution needed to occur. We also cover the need to effectively move content to the edge using One VZN, AWS Snowcone, and AWS DataSync to enable same day global theatrical delivery.

Background

As the backbone of the entertainment industry, Deluxe has been providing finishing and distribution services for theatrical experiences since 1915. The early 2000s marked a milestone for the adoption of DCPs (features and ancillary asset packages like trailers and ads). This included delivery of DCPs to 95% of the world’s theaters (5,500+ theaters in the US; 20,000+ internationally), moving the industry from classic film reels into the digital world.

Currently, delivery is done either via satellite or hard drives shipped by Deluxe to theaters. As years have passed, theater chains and studios are eager for an evolution from satellite-based methods to the cloud to grow the capabilities to feed an ever-demanding consumer market.

One VZN by Deluxe marks the first at-scale innovation for the industry by offering cloud-based IP delivery to exhibitors. The updated platform not only solves challenges, but delivers new benefits and support for new theatrical consumer experiences.

Why evolve?

Satellite solved many issues over the last 10 years; however, satellite has reached the peak of its capability for this use case. It creates barriers to scalability, cost-effectiveness, and accessibility. With satellite transmission requiring significant capital expenditure, the speed to innovate has been slow in coming, as those costs are amortized over many years. In addition, the current state of the workflow is antiquated – it presents obstacles to advancing the theatrical experience, such as providing compelling alternative content in an ever-growing streaming world. Now, those limitations are no longer a roadblock.

In particular, movie studios and exhibitors wanted to work with Deluxe to overcome many of these challenges and apply specific improvements to deliver additional benefits. Studios want to shorten the time between feature lock and global release at theaters, to reduce the risk of piracy or pre-leak of content. Studios and exhibitors struggled to respond to unforeseen events due to content delivery windows and infrastructure storage constraints, limiting their ability to increase monetization via the addition of “late bookings.” Exhibitors have struggled with procurement and management of the IP network in the satellite-based system – a common barrier due to cost and complexity that slowed down adoption of new IP-based solutions. Exhibitors also sought easier ways to receive and unlock content. Finally, exhibitors lacked the ability to view long-term performance of product releases via analytics tools, so they could adjust theatrical bookings against demand for more content.

The ever-evolving obstacles were so significant that it required a new platform to ensure successful achievement of the desired benefits and outcomes for the industry. This became the foundation for One VZN, which combines the rich history and experience of the Deluxe One supply chain with the power and flexibility of AWS to deliver a new architecture. The combination of Deluxe and AWS brings together cloud content via a transformational elastic-scale consumption model, instead of an always-on but inefficient service model.

Serverless design and architecture

One VZN moves content to the edge using a cloud native serverless architecture, based on key design patterns to drive and optimize reliability, performance, and security across the stack.

Digital cinema delivery as a business process requires that content always be available for delivery to target locations. Any transport failure scenario should always have a backup, even if the primary path is redundant. Traditionally, this has come in the form of being able to deliver physical media in the use case where a redundant set of delivery mechanisms may have failed. In looking at replacing satellite with internet-based delivery, redundant paths and managed circuits were naturally the first of these challenges to address. In keeping with the existing delivery model, we must also support the ability to ship content securely on a separate device if a network or component experienced a failure. Additionally, the delivery system must be able to operate without any dependencies on upstream services such as supply chain and storage pipelines. At the core, delivery must happen – no matter what the scenario impacting the delivery chain.

One core and guiding design principle for the One VZN system was to have it couple efficiently into the existing Deluxe One platform and ecosystem for servicing content delivery to the edge. Another was for the platform to be able to function entirely independently, using a minimally complex architecture. By lowering architecture complexity and fully decoupling the backend systems, maintenance and code complexity would also be reduced, including TechOps and DevOps support requirements in failure situations across the entire workflow.

In essence, the model of a highly reliable and production grade distribution system would have to be adopted, taking the same design approach as is typically seen with satellite communication services.

In order to address this challenge, three key design decisions were made to ensure consistent, reliable, timely, and maintenance-free deliveries.

First, the ability to deliver content under a tight SLA is of utmost importance. One VZN leverages the Deluxe One microservice infrastructure to ensure that the delivery mechanism supports the same resiliency as the rest of the supply chain. The challenge becomes one of how to drive content through the system at the scale and size to service an entire global market with same day releases. This was solved by using Amazon S3 in combination with delivery package manifest files, and making the infrastructure event-based with the ability to cope with and recover through the use of patterns. Such patterns include dead letter queues and intelligent retry mechanisms. Similarly, ‘closing the loop’ for deliveries back to the upstream supply chain uses an event-based mechanism. The mechanism is composed of Amazon CloudWatch Events and Lambda functions to drive delivery metrics into SQS queues linked back into the supply chain.

Second, One VZN fully embraces a serverless infrastructure model for all service components. A serverless model eliminates the need to have to manage operating systems, schedule downtime, and perform security patching and associated maintenance tasks. This model is key in simplifying security requirements for movement of content to the edge, since content is not readable or manipulatable by any Lambda function. Only specific AWS data movement services are able to access the content, tightly controlled by AWS IAM policies and AWS CloudTrail logs. AWS DataSync is used to transport distribution content from S3 content source buckets to targeted edge locations running Snowcone devices. Lambda acts as glue and decision logic, validating manifests, configuring jobs for DataSync, and readying content for delivery to Snowcone devices.

Third, we decided to eliminate the need for database and storage management by driving the system via the concept of event-based contracts to facilitate data movement to the edge. We did so through use of structured manifest sidecar files. Delivery of content and receipt of delivery statuses and metrics are essentially unidirectional data transfer patterns, and as such, S3 and delivery manifests can be effectively used as a persistence layer. A single bucket event for manifest reception drives event pipelines that validate DataSync locations (tied to Snowcones) and readies content with DataSync. The single bucket event also aggregates metrics coming back from devices in the field to close an end-to-end delivery loop. Deliveries are ephemeral, and state is tracked upstream in the supply chain system and database tiers. This approach makes manual servicing of content possible if the service were ever to hit a failure state. Human operators must simply drive deliveries by crafting delivery manifests and upload deliveries to an S3 content delivery bucket.

Delivery to the edge

AWS Snowcone plays a key role in enabling Deluxe One VZN to move content securely and rapidly to digital cinema edge networks. When combined with DataSync and the ability to package and deploy a custom AMI on the Snowcone device, it presents a compelling solution for moving content to edge locations that require customized processing to interface with downstream storage and control services.

Snowcone devices provide an environmentally hardened, customizable, secure, and programmatically shippable solution. The devices allow Deluxe to extend the One VZN virtual network into edge cinema location facilities.

Using Snowcone’s custom AMI functionality, we were able to bake services and automate the build and snapshot of a custom AMI for inclusion on our Snowcone deployments. This provided us an ability to tamper-proof our software through Snowcone’s secure shipping with remote unlock model. It also enabled us to easily build and test mock deployments using EC2.

Having the ability to run a custom AMI also enabled us to eliminate the need of storing state and synchronize DataSync deliveries using proprietary logic. In the DCP to edge delivery model, a custom service running in the Snowcone AMI periodically checks DataSync for deliveries matching the Snowcone’s DataSync agent. As we have access to the local storage subsystem, we are able to compare targeted deliveries against content on the local device. This feature enables for content to be pre-loaded at a Deluxe facility before the Snowcone is shipped to a cinema location. This is a critical requirement for bringing a facility back online or providing redundancy with fast turnaround time.

Delivery flexibility plays an important part in being able to deliver content successfully to thousands of edge locations, each with differing bandwidth and capacity options. DataSync allowed us to solve this challenge without having to build a custom delivery service, by providing a number of advantages on top of being a fully managed service. First, content can be sourced from S3 and directly transferred to Snowcone storage for downstream SMB, FTP, and other protocol store and forward delivery mechanisms. Second, content access and data movement is entirely controlled by DataSync. This is critical from a security perspective, as no Lambda code is thus able to access DCP packages. IAM roles and credentials can be scoped and optimized for zero-touch content access. Third, DataSync provides a comprehensive set of data filtering, exclusion/inclusion, encryption, in-flight and after-arrival integrity verification, and event/logging options. This allowed us to simplify the complexity of the Lambda functions required to manipulate content flowing through the delivery pipeline, and implement a fully automated service free of manual operations. Last, we were able to optimize network transfer by making use of several DataSync features. These features include the ability to recover from transfer failures, tune each delivery task for edge location fixed bandwidth and capacity, and queue deliveries in an ordered FIFO fashion.

Snowcone-based delivery, combined with a thin control layer, enabled us to reduce our edge engineering effort to that of a single service supported by a small engineering team. As such, it completely eliminating the undifferentiated and complex heavy lifting commonly seen with traditional satellite edge delivery receivers.

Summary

Deluxe One VZN reaches beyond a migration of theatrical content distribution to the cloud by completely transforming the industry. The platform embraces Deluxe’s strongly established footprint in addition to its large strategic partnerships with both AWS and Echostar. As a next-gen distribution network, One VZN embraces cloud frameworks and optimizes business process via an evolution of system and platform architecture. It is the combination of AWS Snowcone, AWS DataSync, and Amazon S3 that is essential, and the collective services distinguish themselves from any other cloud offerings today.

The collaboration with AWS on this service offering is a fundamental addition and big step forward within Deluxe’s cloud strategy and efforts to migrate all distribution workloads and servicing to the cloud. The Deluxe solution epitomizes the opportunity to advance the art of enabling theatrical experiences worldwide from new release, back catalogue, and repertoire content to live events. The secure, compact Snowcone device, with Deluxe’s powerful Deluxe One platform, creates a truly differentiated solution in the industry. The enhanced Deluxe One platform changes not only the economics of film distribution for exhibitors and studios around the globe, but enables new theatrical experiences for viewers as well.

Thanks for reading this blog and learning more about how Deluxe enables same day global theatrical delivery using AWS Snowcone. Check out this video to learn more: