Using OffShoot Pro by Hedge to offload camera footage to Amazon S3

Amazon Web Services (AWS) is aligned with the MovieLabs 2030 Vision as a north star in understanding and solving industry challenges for the adoption of content production workloads in the cloud. Principle 1 of the MovieLabs 2030 Vision states that:

“All assets are created or ingested straight into the cloud and do not need to be moved.”

Working backwards from this principle, we set out to solve the challenge of moving large Original Camera Files (OCF) to the cloud from on-set and studio locations, often with limited connectivity.

At NAB 2023, Hedge announced the new name for its offload software, now called OffShoot. The Pro version of the software ships with Amazon Simple Storage Service (Amazon S3) native integration.

In this blog post, we describe how to use OffShoot Pro along with Amazon S3 to build a workflow that enables ingest of OCF data straight to the cloud, as well as integrating with FoolCat to create camera reports and EditReady to create proxies. We also discuss how customers can benefit from this workflow to accelerate and secure their content production workloads.

Figure 1: The architecture starts with a camera on the left. The next step in the workflow is shown by an arrow from the camera to an icon with “OffShoot Pro” label. From OffShoot Pro, 2 arrows lead to 2 separate icons of hard disks with the label “backup” on the arrows. Two other arrows lead from OffShoot Pro to icons for FoolCat and EditReady. Arrows from each of these tool icons lead into another icon labeled OffShoot Pro. This second OffShoot Pro icon has an arrow pointing to an Amazon S3 bucket icon inside a box labeled AWS Cloud

Figure 1: Architecture diagram of Hedge tools (OffShoot Pro, FoolCat, and EditReady) used to create a package to upload to Amazon S3

Why offload to the cloud?

Offloading footage to the cloud can be beneficial for many reasons. It is important to decide which benefit is important to your production so you can build your workflow accordingly. Three reasons we focus on in this blog post are:

Backing up footage to the cloud
Using the cloud for secure and fast collaboration
Automating downstream tasks

Backing up footage to the cloud

OCF data is the output of a day’s shooting or recording, and holds enormous value. This data is often the sum total of months of production planning and includes the costs of assembling crew, talent, and numerous services. Award-winning performances, stunts and moments of great significance may be captured in this data, so a good data management plan is of paramount importance to ensure the durability and security of this data.

The reliability of a cloud backup solution requires secure and error-free data transfer, and safe and durable storage in the cloud.

The combination of Offshoot Pro by Hedge and Amazon S3 is a backup solution that excels in all these areas. Communication with S3 is performed over secure HTTPS connections, trusted by the most security-sensitive workloads in the world. Offshoot Pro uses the checksum functionality of Amazon S3 to verify that all data is received by AWS without transmission errors. Finally, Amazon S3 durably stores data across multiple devices in different data centers along with error correction data. The service is designed to sustain concurrent device failures by quickly detecting and repairing any redundancy loss. It also runs periodic checks to verify the integrity of your data using checksums.

The media and entertainment (M&E) industry standard for data backups is known as the ‘3:2:1 model’, requiring 3 copies of the data on 2 different types of storage, with one off-site storage.

Figure 2: There are 3 icons. From left to right their labels read “3 copies of your data”, “2 types of storage”, and “1 off-site storage

Figure 2: The M&E industry standard “3:2:1” backup model

OffShoot Pro can offload data from a single source device to multiple destinations at once. You can configure OffShoot Pro to create multiple local copies, as well as sending a copy to Amazon S3.

Figure 3: OffShoot Pro user interface

Secure and fast collaboration

The second key benefit of offloading camera footage to the cloud is that other members of your team can access it right away. This is in line with the third principle from the MovieLabs 2030 Vision, which states that:

“Propagation and distribution of assets is a ‘publish’ function.”

Using Amazon S3 as a single repository for assets, customers can employ a ‘store once, use many’ approach to their content, by publishing their content to stakeholders and external vendors on an as-needed basis rather than creating and distributing many copies of this precious data.

You can use EditReady to generate smaller proxy files for upload along with raw OCFs. These proxies can then be used by editors to make edits even while the shoot continues on set.

To keep track of the media ingested into S3 and to find the right clips quickly, FoolCat can generate reports that can also be uploaded, along with data for downstream use.

Figure 4: Sample FoolCat report

The security of the media is extremely important. AWS provides multiple layers of security for your data. First, all data in Amazon S3 is encrypted at rest by default using one of the strongest block ciphers available (AES-256). Second, S3 integrates with the AWS Identity and Access Management (IAM) service, allowing fine-grained access control and monitoring that adheres to the principle of least privilege.

Automation of downstream tasks

Configure Amazon S3 to send event notifications when certain actions are taken on a bucket or an object. These events can be used to trigger automated workflows such as:

Sending notification emails
Adding media to Media Asset Management software
Transcoding media
Using Amazon Transcribe to generate a transcription text for your media
Using Amazon Rekognition for automatic tagging
Extracting thumbnails from video clips

Connectivity considerations

Film and TV sets may be in remote locations where internet access is limited. More and more studios are leveraging technologies such as AWS Direct Connect in order to bring these locations closer to the cloud with managed, high-bandwidth connectivity. In addition, technologies such as 5G bring the internet to harder to reach locations.

Many of the initial phases of production also only need access to proxy copies of data. By using transcoding tools such as EditReady by Hedge, we can, for example, create and upload proxy copies of OCF data and make them immediately available to editorial teams.

For the OCF data itself, in a scenario where a high-bandwidth connection isn’t possible, data could be placed on a device such as an AWS Snowball or Snowcone and reconciled with the proxy copy later in the production process.

Try it out yourself

To set up the infrastructure in AWS, we need to perform the following steps. The hyperlinks point to instructions for each step.

Create an S3 bucket
- Choose a bucket name
- Choose a region that is close to you
- Leave the rest as default
Create an IAM user
Create access keys for the user
Create and attach the following policy to the user. (Replace [CONTENT-UPLOAD-BUCKET] with the name of your bucket.)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": [
                "arn:aws:s3:::[CONTENT-UPLOAD-BUCKET]"
            ]
        },
        {
            "Sid": "PutObject",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": [
                "arn:aws:s3:::[CONTENT-UPLOAD-BUCKET]/*"
            ]
        }
    ]
}

To set up OffShoot Pro:

Connect OffShoot Pro to AWS
- Provide the bucket name
- Provide the access credentials
Optionally configure OffShoot to create a custom folder structure
- The following screenshots show an example configuration that conforms to the Netflix Folder and File Naming guidelines

Figure 5: The user interface shows a configuration page with the following options: “Do not copy files”, “ignore bundles”, “ignore folders”, “Auto Label”, “Rename Files”, “Create Folders”, and “Counter”. All the options are left empty except for “Create Folders” that has the text “Year YYYY, Month Day/Camera Media/Source Name” next to it

Figure 5: OffShoot Pro user interface (settings)

At this point, you are ready to test your setup. Choose a source location and set the destination to the AWS cloud endpoint you set up previously. Click the “Add Transfers” button to start the upload.

Figure 6: The user interface of OffShoot looks similar to figure 3 with a source hard drive on the left and a destination drive with AWS on it on the right. There is now a new panel in the middle that has input fields for Date and Source Name

Figure 6: OffShoot Pro transfer with presets

Figure 7: The user interface shown is similar to figure 6. In the middle section there is a progress bar that is filled with green. Below the progress bar is the word “Completed”

Figure 7: Completed transfer to Amazon S3

Once the transfer is complete, you can log in to the AWS S3 console, and browse your S3 bucket. You should see the files just uploaded by OffShoot Pro.

About Hedge

Hedge‘s products are renowned for their simplicity, efficiency, and excellent customersupport. With over 50,000 customers worldwide, from Hollywood productions to wedding shoots and everything in between. Hedge’s mission is to make complex media workflows simple and accessible to all. Hedge’s product offerings cover all aspects of the post-production workflow from Offload and Ingest, Transcode and Prepare, Collaborate and Share, Check and Review, to Archive & Backup.Hedge’s mission is to remove friction and close all time-consuming and error-prone workflow gaps.