Transferring asset files, caching, and storage-to-render on AWS

INTRODUCTION

One of the many goals of Deadline 10 is to ensure that customers can render on AWS as easily as possible. An essential part of moving to a hybrid or fully AWS-based rendering workflow is dealing with transferring asset files to AWS, and downloading resulting renders from AWS. Fortunately, customers can accomplish this job with Deadline 10 painlessly by using the new features that make dealing with asset files in AWS a breeze!

SIMPLIFY YOUR AWS RENDER PIPELINE

In almost all cases, a rendering job will depend on many external files and textures to produce its renders. When moving to the cloud, these files will not exist in cloud storage prior to rendering. Previously, artists had to manually transfer the required files to cloud storage prior to submitting a render. Artists would also have to manually download the resulting renders after the rendering job was complete. This presented an extra hurdle in cloud-based workflows. Deadline 10 helps clear this hurdle by enabling the AWS render nodes to utilize your on-premises network file system.

USE YOUR ON-PREMISES NETWORK FILESYSTEM IN AWS

Deadline 10 offers a service that allows rendering nodes to see your on-premises network filesystem as if it were in the cloud. The files themselves do not need to exist in the cloud in order for the rendering application to use them.

Asset files that are needed get copied to cloud storage prior to the render beginning. Any files that were missed in the initial copy get copied to cloud storage when they are first accessed. This workflow removes the manual asset gathering steps that an artist would need to do previously. No asset files are missed, and everything renders as if it were local.

As soon as files are uploaded, the files are cached in cloud storage. Once a file is initially cached, that file will never be re-uploaded unless its source has been modified on-premises. This allows the user to render an entire sequence, while the required files are only transferred to cloud storage once.

AUTOMATIC DOWNLOADING OF RENDERED IMAGES FROM AWS

In addition to automatically uploading required assets from the on-premises network, Deadline 10 also automatically downloads the resulting rendered images back to the on-premises network storage. This happens without any artist intervention and begins as soon as a render task is complete. These render files also remain cached in cloud storage which allows future Deadline jobs to use the results of a render without having to upload the files from on-premises.

HOW IT WORKS

All Deadline 10 AWS-based render nodes have the option to mount a directory which can mirror an on-premises network share. Using Deadline’s built-in Path Mapping options, users are able to path map this mounted directory to their on-premises directory root. When this option is enabled, all of the communication with the on-premises network, as well as the caching, transfer, and storage of files are handled behind the scenes.

In order for this to work properly, Deadline 10 provides client software which must be installed on the on-premises side. This client software is able to connect with the AWS render nodes and provide access to the on-premises files. Our client software does not require the use of a VPN, and does not require any modifications to the on-premises firewall. Only your AWS-based rendering nodes will be able to connect to it.

WRAPPING UP

We hope you enjoy Deadline 10’s new cloud rendering features. We are working hard to make Deadline renders on AWS as easy as possible! Let us know what you think of our new filesystem features on the Deadline Forums.