Using OffShoot Pro by Hedge to offload camera footage to Amazon S3
Amazon Web Services (AWS) is aligned with the MovieLabs 2030 Vision as a north star in understanding and solving industry challenges for the adoption of content production workloads in the cloud. Principle 1 of the MovieLabs 2030 Vision states that:
“All assets are created or ingested straight into the cloud and do not need to be moved.”
Working backwards from this principle, we set out to solve the challenge of moving large Original Camera Files (OCF) to the cloud from on-set and studio locations, often with limited connectivity.
In this blog post, we describe how to use OffShoot Pro along with Amazon S3 to build a workflow that enables ingest of OCF data straight to the cloud, as well as integrating with FoolCat to create camera reports and EditReady to create proxies. We also discuss how customers can benefit from this workflow to accelerate and secure their content production workloads.
Why offload to the cloud?
Offloading footage to the cloud can be beneficial for many reasons. It is important to decide which benefit is important to your production so you can build your workflow accordingly. Three reasons we focus on in this blog post are:
- Backing up footage to the cloud
- Using the cloud for secure and fast collaboration
- Automating downstream tasks
Backing up footage to the cloud
OCF data is the output of a day’s shooting or recording, and holds enormous value. This data is often the sum total of months of production planning and includes the costs of assembling crew, talent, and numerous services. Award-winning performances, stunts and moments of great significance may be captured in this data, so a good data management plan is of paramount importance to ensure the durability and security of this data.
The reliability of a cloud backup solution requires secure and error-free data transfer, and safe and durable storage in the cloud.
The combination of Offshoot Pro by Hedge and Amazon S3 is a backup solution that excels in all these areas. Communication with S3 is performed over secure HTTPS connections, trusted by the most security-sensitive workloads in the world. Offshoot Pro uses the checksum functionality of Amazon S3 to verify that all data is received by AWS without transmission errors. Finally, Amazon S3 durably stores data across multiple devices in different data centers along with error correction data. The service is designed to sustain concurrent device failures by quickly detecting and repairing any redundancy loss. It also runs periodic checks to verify the integrity of your data using checksums.
The media and entertainment (M&E) industry standard for data backups is known as the ‘3:2:1 model’, requiring 3 copies of the data on 2 different types of storage, with one off-site storage.
OffShoot Pro can offload data from a single source device to multiple destinations at once. You can configure OffShoot Pro to create multiple local copies, as well as sending a copy to Amazon S3.
Secure and fast collaboration
The second key benefit of offloading camera footage to the cloud is that other members of your team can access it right away. This is in line with the third principle from the MovieLabs 2030 Vision, which states that:
“Propagation and distribution of assets is a ‘publish’ function.”
Using Amazon S3 as a single repository for assets, customers can employ a ‘store once, use many’ approach to their content, by publishing their content to stakeholders and external vendors on an as-needed basis rather than creating and distributing many copies of this precious data.
You can use EditReady to generate smaller proxy files for upload along with raw OCFs. These proxies can then be used by editors to make edits even while the shoot continues on set.
To keep track of the media ingested into S3 and to find the right clips quickly, FoolCat can generate reports that can also be uploaded, along with data for downstream use.
The security of the media is extremely important. AWS provides multiple layers of security for your data. First, all data in Amazon S3 is encrypted at rest by default using one of the strongest block ciphers available (AES-256). Second, S3 integrates with the AWS Identity and Access Management (IAM) service, allowing fine-grained access control and monitoring that adheres to the principle of least privilege.
Automation of downstream tasks
Configure Amazon S3 to send event notifications when certain actions are taken on a bucket or an object. These events can be used to trigger automated workflows such as:
- Sending notification emails
- Adding media to Media Asset Management software
- Transcoding media
- Using Amazon Transcribe to generate a transcription text for your media
- Using Amazon Rekognition for automatic tagging
- Extracting thumbnails from video clips
Film and TV sets may be in remote locations where internet access is limited. More and more studios are leveraging technologies such as AWS Direct Connect in order to bring these locations closer to the cloud with managed, high-bandwidth connectivity. In addition, technologies such as 5G bring the internet to harder to reach locations.
Many of the initial phases of production also only need access to proxy copies of data. By using transcoding tools such as EditReady by Hedge, we can, for example, create and upload proxy copies of OCF data and make them immediately available to editorial teams.
For the OCF data itself, in a scenario where a high-bandwidth connection isn’t possible, data could be placed on a device such as an AWS Snowball or Snowcone and reconciled with the proxy copy later in the production process.
Try it out yourself
To set up the infrastructure in AWS, we need to perform the following steps. The hyperlinks point to instructions for each step.
- Create an S3 bucket
- Choose a bucket name
- Choose a region that is close to you
- Leave the rest as default
- Create an IAM user
- Create access keys for the user
- Create and attach the following policy to the user. (Replace [CONTENT-UPLOAD-BUCKET] with the name of your bucket.)
To set up OffShoot Pro:
- Connect OffShoot Pro to AWS
- Provide the bucket name
- Provide the access credentials
- Optionally configure OffShoot to create a custom folder structure
- The following screenshots show an example configuration that conforms to the Netflix Folder and File Naming guidelines
At this point, you are ready to test your setup. Choose a source location and set the destination to the AWS cloud endpoint you set up previously. Click the “Add Transfers” button to start the upload.
Once the transfer is complete, you can log in to the AWS S3 console, and browse your S3 bucket. You should see the files just uploaded by OffShoot Pro.
Hedge‘s products are renowned for their simplicity, efficiency, and excellent customersupport. With over 50,000 customers worldwide, from Hollywood productions to wedding shoots and everything in between. Hedge’s mission is to make complex media workflows simple and accessible to all. Hedge’s product offerings cover all aspects of the post-production workflow from Offload and Ingest, Transcode and Prepare, Collaborate and Share, Check and Review, to Archive & Backup.Hedge’s mission is to remove friction and close all time-consuming and error-prone workflow gaps.