AWS Thinkbox Deadline adds support for Rez

Today Amazon Web Services (AWS) announces AWS Thinkbox Deadline 10.2.0.10 support for Rez.

“Rez is an open source, cross-platform package manager that creates standalone configured environments for third-party and proprietary digital content creation software. Rather than installing packages directly into environments like most other package managers, Rez installs all package versions into repositories on disk and references them in dynamically created standalone environments. Rez is widely used in the pipeline community and solves countless package management problems for visual effects and animation production developers.” – Academy Software Foundation (ASWF.io)

Figure 1: Rez installs packages once, and configures environments dynamically (via GitHub)

Rez solves the problem of ensuring an artist’s workstation has all the correct packages present in a dynamically resolved environment when they are working on a particular task. For example, these packages can be used to set aliases for specific tools, append file paths, and set environment variables. This powerful combination allows Rez to configure an environment where applications can be launched, custom software can be sourced, and licensing environment variables can be set, allowing for granular control over the tools exposed to an artist in the VFX (Visual Effects) pipeline.

From the perspective of a render farm, it is likely any Job submitted from a Rez environment configured for the artist should run with the same configuration and include the same file paths, custom software, and licensing.

Previous methods

Prior to today’s announcement, customers implemented their own solutions to integrate Rez into Deadline. Two common approaches are:

• On the submitting workstation, inject environment variables from the submitting Rez-resolved environment into a Deadline Job object at submission time. The environment variables help set file paths for specific applications, libraries, and Python packages, which should allow the right packages to load as part of the Job. The environment variables are automatically applied at runtime on the Deadline Worker. This is the simplest implementation but has limitations: only environment variables are applied onto the Worker, the Worker can’t dynamically resolve a Rez context, and there is no easy method to configure each Deadline render plugin to use a specific Rez tool and subsequent render executable.
• On the submitting workstation, inject a list of all non-implicit Rez resolved packages within the submitting Rez-resolved environment as an Extra Info Key Value into a Deadline Job via an OnJobSubmitted event. Then, on the Worker, use the ModifyCommandLineCallback to prefix all render commands with the rez-env command to dynamically resolve the Extra Info Value list of Rez packages. This is a clever use of the Deadline Scripting API but it still doesn’t provide an easy mechanism to configure the render executable to be used per Deadline render plugin.

Both solutions require customers to write custom code using the scripting API built into Deadline but that’s undifferentiated heavy lifting as we say at AWS and we can do better.

Deeper integration

The following is an overview of the newly announced integration of Rez into AWS Thinkbox Deadline.

Figure 2: Rez integration in Deadline via a VFX Platform CY2022 configured environment example

Let’s see how this works in practice by configuring the open-source Blender application in Deadline.

Digital content creation (DCC) example

Prior to using Rez with Deadline, make sure that Rez is installed and configured on your workstations and Worker machines. Deadline assumes that Rez is on your system PATH as per its standard install instructions. Once Rez is installed, the Rez event plugin can be used. To access the Rez event plugin settings, enter Super User mode in the Deadline Monitor and select Tools -> Configure Events from the menu. From there, select the Rez entry from the list on the left. Once enabled, the Rez event plugin will use by default the Rez derive mode: REZ_USED_RESOLVE.

Figure 3: Rez: Configure Event Plugin dialog in Deadline

Job submission

During Job submission, Deadline will automatically attach the contents of the environment variable REZ_USED_RESOLVE that was set by Rez on the workstation, minus the Rez defined implicit packages taken from REZ_USED_IMPLICIT_PACKAGES, to the Deadline Job as an Extra Info Key Value: DEADLINE_REZ_REQUEST_PACKAGES. Additionally, Deadline sets the DEADLINE_REZ_TOOLS Extra Info Key Value on the Job by running rez-context –tools as a subprocess and then parsing the output to compile a list of Rez tools available to the Job. If Deadline cannot derive the package or tool list, it will log a warning and complete the Job submission without Rez. This is done in all Rez plugin modes.

Post job submission

After a Job is submitted with Rez, you can view the Extra Info Key Values added from the Job properties panel. Available by double-clicking the Job in the Monitor; DEADLINE_REZ_REQUEST_PACKAGES is the list of package-versions the Worker will attempt to resolve, and DEADLINE_REZ_TOOLS is the list of tool names available in the Rez resolved context.

Figure 4: Rez: Blender request packages and tools present in Deadline Job, post-submission

Render plugin configuration

To use Rez in conjunction with existing Deadline render plugins, you must add the Rez tool for the appropriate renderer to that render plugin’s executable list. For example, if you have a blender Rez package that contains a tool called blender, then you will need to add the tool name to the list of Blender Executables in the Blender plugin. Render Plugin configurations can be found by entering Super User mode and selecting Tools -> Configure Plugins from the Monitor’s menu. When running the Job on a Worker, the executable list is checked for Rez tools before any full executable path.

Figure 5: Rez: Blender executable list with Rez blender tool listed, in Deadline

Job execution

A Worker dequeues a task for a Deadline Job and then checks if this is a Rez enabled Job via the DEADLINE_REZ_TOOLS Job Extra Info key. If DEADLINE_REZ_TOOLS has a value, then the DeadlinePlugin.GetRenderExecutable() scripting function returns the first Rez tool found in both the DEADLINE_REZ_TOOLS list and the Deadline render executable list. If no Rez tool is found, then it searches the render executable list in order, as usual. The Worker then invokes the rez-env executable as an inline prefix command to the render command using this example syntax:

rez-env DEADLINE_REZ_REQUEST_PACKAGES — $RENDERER $RENDER_ARGUMENTS

So, in the case of our Blender example:

/path/to/rez-env python-3.7.12 hello_world-1.0.0 blender-3.3.1 numpy-1.21.6 — blender -b /path/to/<scene-file>.blend -t 0 -s 1 -e 1 -a

where rez-env will attempt to resolve a context using the Rez packages defined in the Job Extra Info value: DEADLINE_REZ_REQUEST_PACKAGES. Then, within that context, continue to execute the Blender render command and immediately exit the Rez context after the command runs.

Success! Rez now dynamically resolves a configured environment on-demand, locates the exact render tool to be invoked, and renders a frame of work on a Deadline Worker identically to the workstation the Job was submitted from.

Extra notes

• The Rez event plugin can be configured to “Opt-In” instead of “Global Enabled” allowing you to selectively choose which Deadline Jobs trigger the Rez workflow at Job submission. This could be useful for early adopters, migration, or debugging in a studio pipeline.
• The default REZ_USED_RESOLVE derive mode during Job submission removes implicit packages such as ~platform==windows, which is important if you wish to submit a Deadline Job from a Windows workstation and render on a Linux render node.
• Other Rez derive package modes are supported. Please see the Rez event plugin documentation for more details.
• If a Deadline Job has the two Extra Info Keys present (DEADLINE_REZ_REQUEST_PACKAGES and DEADLINE_REZ_TOOLS), then the Worker will attempt to launch the compute process with a resolved Rez context. To disable Rez for a particular Deadline Job, you should remove/temporarily rename these Extra Info Keys in the Job properties.
• If multiple Rez tools are listed in a Deadline render plugin configuration, the Worker will select the first tool from DEADLINE_REZ_TOOLS found in the ordered executable list.

Summary

In this post, we explained that Rez is an open source, cross-platform package manager that creates standalone configured environments for third-party and proprietary digital content creation software. Rez is now deeply integrated into AWS Thinkbox Deadline, allowing customers to build dynamically resolved pipelines that can be executed in an identical way on their render farm, whether that be on-premises or in the cloud.

Further information is available in the ASWF Rez documentation and Deadline’s Rez event plugin documentation and FAQs.