Thinking outside the office: Nature’s catalyst for innovation

Authored by Andy Shenkler, Chief Product & Technology Officer, Deluxe. The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post. Illustration by Cecilia Betsill. 

On March 11, 2011, a 9.1-magnitude undersea earthquake strikes off the coast of Japan at 2:46 p.m. local time. The force from the fourth-largest earthquake ever recorded moved the main island of Japan and shifted the axis of the Earth itself. It also produced one of the most devastating tsunamis in history, killing over 15,000 people and leaving millions displaced or without power and water. In addition to the tragic human toll, the quake caused massive destruction of infrastructure and left hundreds of thousands of buildings in ruin. It was in this infrastructure disruption that the impact on broadcast, production, and post-production industries was felt far beyond the shores of Japan.

Suddenly the supplies of HDCAM, Betacam, Digi-Beta, Blu-Ray, and Linear Tape Open (LTO) became incredibly scarce commodities as Japanese manufacturing plants had to cease operations for an indeterminate amount of time. The industry began to scramble with the realization that, with regard to business continuity and disaster recovery, a single point of failure existed in virtually every production workflow: the global supply of tape stock.

File-based workflows were still nascent and amidst continued investment in legacy technology, large capital expenditures had hampered many organizations’ ability to change their processes to move to a mostly digital supply chain. The industry at that time was still highly dependent on tape and was reluctant to change, especially due to the depreciation of large tape-based assets.

Thus, it wasn’t a pronouncement of a consortium or standards body that triggered a change, but rather Mother Nature herself that ultimately forced an accelerated migration to a file-based world. While such a calamitous event was tragic, it would be hard to imagine how the explosion of global streaming services would have been possible in the following years had video tape continued to be the dominant format by which to operate. The first “kick” the industry needed to get moving was at hand, and while we traded out video tape for large-scale LTO tape robots, at least progress was underway.

While consumer-focused services rapidly adopted cloud computing as the only cost-effective way to scale, post-production environments continued to entrench in large and costly on-premises workflows. Capital flowed quickly during the expedited transformation to on-premises digital. Unfortunately, the environments and workflows were not re-imagined to become truly non-linear. Those who ran such facilities took great pride in showing off their new “all-in-one facility” showrooms. No meeting would ever be complete without a tour of a facility, to show how many tapes, badged doors, and bays were available at any given time. Post-production companies had created what was effectively a digital assembly line, but was often misrepresented as a digital supply chain.

Workflows had been only slightly modified by effectively crossing out the word tape and replacing it with the word file, but fundamentally very little had changed. Conversations about storing three copies of an LTO tape, on-premises at a second location with a third at an off-site storage vault were a common solution to disaster recovery. Additionally, there was often a dependency on a single robotic library in a particular location, which rendered the multiple copies pointless except for a bad copy of a tape. The deployment of multiple same-sized robotic infrastructures, in geographically dispersed locations, with exact duplication of content would most certainly have been cost prohibitive, especially for an industry that viewed digital as both faster and cheaper.

Fast forward a few years to the advent of cloud computing and distributed workforces. After having faced the unexpected reality in 2011 of industry-wide tape stock shortages, one would have hoped that the advent of cloud storage and computing would have been a welcome replacement to protect against similar supply disruptions. Yet here we are in 2020, nine years later, and the industry’s cloud adoption rate has only just begun to truly accelerate. While early movers have migrated their media supply chains to the cloud, even now there are still post-production facilities that continue to invest in on-premises LTO infrastructure, only furthering their dependency at a time when alternatives are needed the most.

As time evolves, once again Mother Nature brings new challenges and our society is forced to rapidly adapt to the impact felt across global markets from Entertainment to Education and all verticals in between. The potential for disruptions of supply chains and workforces again brings into focus the topics of disaster recovery and business continuity, and the role of remote operations and technology throughout the M&E industry. As the entertainment industry responds to the closure of events, suspension of national sports leagues, and delayed box office releases, to name just a few of the consequences, businesses are reacting quickly to the harsh realities of the world today as they create a plan to sustain tomorrow.

Conversations are already occurring across organizations and divisions regarding protecting employees as well as protecting the business. Serious thought is being given to health and safety and some companies have no choice but to restrict working conditions to minimize risk for employees. Many organizations currently encourage personnel, if possible, to work from home for this reason.

Today there is great reliance on large workforces in single environments, whether based in the United States, India, or Europe. Looking across the various post-production companies, could those organizations continue to effectively operate if an entire site was inaccessible for weeks? In the type of scenarios being modeled, what if multiple facilities became physically inaccessible at the same time? Suddenly the dependency on tape and large scale on-premises infrastructure— which has historically been seen as the pillar of stability—is now a very tangible business risk.

The key to true progress in business continuity and business risk reduction for M&E is a cloud-based, distributed workforce leveraging a true digital supply chain orchestration system with built-in security and automation. Such an approach must take into account continuity-of-service challenges faced by post-production operations, including complex on-premises SAN fabrics intertwined with hardware-based equipment and security protocols that have been put in place to completely isolate networks from any outside access to those environments. There are certain functions that will require specialty environments for the purposes of particular QC or high-end workflows. The amount of services that can be controlled and operated remotely is quite significant, but it also requires tools that support that type of working model.

There are many organizations that believe they have this in place already, but those claims are already being put to the test: Closure of a facility for at least one week. Phone, Slack, email, and online conference meetings only. No ability for employees and contractors to go into the office, no quick sidebar conversations, no grabbing a group for an impromptu discussion. Can your company receive POs from customers? Can your teams enter them into a financial system remotely? Do they have access to every subsystem necessary? Even that legacy database that the IT team wasn’t aware of but turns out to be critical to parts of your operation that you hadn’t even realized? What of the multitude of Final Cut Pro or other content workflow stations that haven’t been upgraded in three to five years, for fear of the upgrades making that version of software obsolete? Is your VPN and external connectivity appropriately sized or even capable of supporting the entire office to concurrently access content all day?

A multitude of think tanks and scholarly studies have predicted this type of scenario. Not many of us were focused on this as we prepared to ensure stable servicing of our industry. Now, M&E businesses are undoubtedly discussing how to expedite core functionality to the cloud and reduce the industry’s reliance on so many single points of failure, even the ones not previously seen as such.

It’s now more critical than ever that we move our workflows to the cloud and geographically dispersed infrastructure, pushing forward to embrace remote editing and virtualized workstations. All content must be in the cloud and not stored in the loose directory structures of the past, but in an organized hierarchy that can be shared between organizations with appropriate security and authorization protocols. We may not be able to move every function to the cloud, such as services provided on sound mixing stages or high-end 4K QC, but that is only a matter of time. In particular, security protocols need to be reviewed to enable secure content without impeding the ability to operate. The legacy Fort Knox model of excluding production networks from remote access doesn’t seem as valuable if you simply can’t perform the needed work. All functions of the organizations need to collaborate and think about the problem through a new lens.

When the inherent risks of the old ways of conducting business threaten future viability, we have no other option than to aggressively move towards a more resilient and nimble way of operating. Change can be difficult for some people, but we can’t allow those who are comfortable to prevent us from doing what is necessary. Embracing highly distributed cloud-based workflows is the key to that future.

About the Author: Andy Shenkler, Chief Product & Technology Officer, Deluxe

Andy leads Deluxe’s global product & technology team while additionally leading Deluxe’s worldwide fulfillment and platform division. He focuses on strategically developing tools and applications to better serve our clients’ needs to create and distribute content around the world more efficiently. He is a pioneer of cloud-based content management services and has over 20 years of experience in product development and innovation with a track record of success in scaled, global roles.

About Deluxe Entertainment Services Inc.

Deluxe Entertainment Services (Deluxe) is the world’s leading video creation to distribution company offering global, end-to-end services and technology. Through unmatched scale, technology and capabilities, Deluxe enables the worldwide market for premium content. The world’s leading content creators, broadcasters, OTTs and distributors rely on Deluxe’s experience and expertise. With headquarters in Los Angeles and New York and operations in 38 key media markets worldwide, the company relies on the talents of more than 7,500 of the industry’s premier artists, experts, engineers and innovators.