AWS for M&E Blog

Using Open Job Description to Customize Submitter Workflows

Are you an Artist or Technical Director (TD) who’d like to streamline the process of defining and submitting jobs to your render farm? This blog describes how to use Open Job Description (OpenJD) to customize the Amazon Web Services (AWS) Deadline Cloud integrated submitters, and concludes with a code sample of a custom submitter written from scratch.

OpenJD is an open specification that allows users to define jobs to submit to a render farm. Job submission takes place toward the end of content creation workflows, and OpenJD provides opportunities for TDs and technical artists to customize submitter workflows to minimize workflow interruptions. Submitter customizations can save time and effort for users submitting jobs to AWS Deadline Cloud.

The Deadline Cloud integrated Maya 2024 Submitter is shown in Figure A. It provides users with core functionality for submitting Maya jobs.

Figure 1: The Deadline Cloud integrated Maya 2024 Submitter.

Figure A: The Deadline Cloud integrated Maya 2024 Submitter.

Customization Use Cases

The FuzzyPixel custom Maya submitter was developed by the FuzzyPixel team. We are an AWS team working on real world productions to help design, prototype, and test AWS features before they are released. In this blog, we’ll focus on the customizations we developed during our most recent production.

The use cases we’ll cover are:

  1. Updating Project and Output Paths: When project and output paths are automatically populated based upon the opened scene, it saves time and reduces cognitive load for users.
  2. Image Resolution: A job might have a final render resolution of 8K. Allowing users to scale down image resolution for test renders saves time and reduces rendering cost.
  3. GUI Re-Organization: The integrated submitter organizes GUI controls using four different tabs. Placing frequently used controls under a RenderSettings tab helps to streamline job submission workflows.
  4. Fleet Custom Attribute: When users are allowed to choose whether their jobs run on a Spot or On-Demand Instance, it helps them to avoid job interruptions for longer renders.
  5. Render Layer Control: Giving users explicit control of render layers in the submitter allows them more control over render layer visibility and frame ranges.

We fulfilled all of the use case requirements using OpenJD, the AWS Deadline Cloud API, the Maya API, PySide2, and Python scripting. Let’s take a closer look at OpenJD before exploring the first use case.

Prerequisites

All of the following prerequisites are required before continuing:

  1. A Windows workstation.
  2. An installation of Maya 2024 and the AWS Deadline Cloud Submitter.
    1. Note that Maya 2024 has its own fee structure and this is not associated with AWS
  3. A farm, with a fleet and queue, plus the Deadline Cloud monitor and a profile created for your farm.
  4. An installation of Python-3.10.8 (For best results, use this specific version of python to match Maya 2024’s python version)
    1. https://www.python.org/downloads/
    2. pip install deadline and gui dependencies into this version of python
      1. For example, “C:\Program Files\Python-3.10.8\PCBuild\amd64\python.exe” -m pip install deadline[gui]
  5. Download the FuzzyPixel custom Maya submitter.

Note: All examples provided within this blog run only on Windows workstations.

A Look at OpenJD with Job Bundles

In the image of the Deadline Cloud integrated submitter (Figure A), there’s an Export bundle button at the lower right. A job bundle is a complete description of a job that runs on a Deadline Cloud render farm. In this section, we’ll take a look at a job bundle exported from a Maya scene as a first step toward understanding how the previously stated use cases were fulfilled.

Export a Job Bundle

  1. Open Maya and load a scene to submit to your render farm.
  2. After the scene loads, launch the submitter. If you have installed the submitter correctly, you should see an AWS Deadline tab as shown in Figure B (on the far-right side of the screen). If you don’t see the submitter, go to Windows, then Settings/Preferences, and lastly Plug-in Manager. In the popup window that appears, type Deadline, and then select Loaded and Auto load before exiting the window.
Figure 2 : The AWSDeadline tool shelf in Maya 2024.

Figure B: The AWSDeadline tool shelf in Maya 2024.

  1. When you see the Deadline Cloud icon appear on the AWS Deadline shelf, click on it and the Maya integrated Submitter will launch. Login to your farm by following these instructions.
Figure 3: Initial Launch of the Maya 2024 integrated submitter.

Figure C: Initial Launch of the Maya 2024 integrated submitter.

  1. After launching the submitter, click the Export bundle button. After exporting, a file browser and confirmation modal will pop up. The file browser will show a folder with three files containing the job bundle.

 

Figure 4: The Maya 20204 integrated submitter after clicking Export bundle.

Figure D: The Maya 20204 integrated submitter after clicking Export bundle.

 

A Description of the Job Bundle

Each job bundle folder exported from Maya contains three files: one for asset references, one for parameter values, and one template file to describe the exported job.

  1. The asset_references.yaml file contains asset paths, the output folder, and any referenced paths for the job. If you are using a Service-Managed Fleet (SMF), this is the vehicle for uploading your assets to the render farm. If you are using a Customer-Managed Fleet (CMF) with storage profiles, the lists specified in asset_references.yaml can be empty.
  2. The parameter_values.yaml file conveniently contains any values you intend to reference in your job template. For our Image Resolution use case, an example of a set of parameter values would be ImageWidth and ImageHeight.
  3. The template.yaml file contains a description for the job. It can reference the asset_references.yaml and parameter_values.yaml files to fully represent the job. While it’s convenient to use all three files, a job can be completely described using only the template.yaml file.

We’ll use and build on the job bundle to fulfill our first use case: Updating Project and Output Paths.

Use Case #1 – Updating Project and Output Paths

The FuzzyPixel team wanted to populate the project and output paths using python code that derives them from the currently opened Maya scene.

Here’s how we approached this:

  1. Perform Maya queries,
    • There are a variety of ways to get the current path of a Maya file. We used the following: project_path = cmds.workspace(q=True, active=True)
    • For the output path, we concatenated the project path with a folder as shown in Figure G: output_path = f’{project_path}/images’
  2. We then used the query results to populate the project and output fields in the submitter.
    • Once we have values for the project and output paths, we can inject the data into the parameter_values.yaml file described in the previous section.
    • Open the parameter_values.yaml file you exported earlier and find the line in the file that refers to the project and output paths. The exported bundle with these lines highlighted can be seen in Figure E. The appropriate lines for our exported bundle .yaml file are:
- name: ProjectPath
value: W:/Prod2/prod/sequences/sq0300/sh0050/light
- name: OutputFilePath
value: W:/Prod2/prod/sequences/sq0300/sh0050/light/maya/images
Figure 5: The exported parameter_values.yaml file.

Figure E: The exported parameter_values.yaml file.

 

  1. You can update the parameter_values.yaml file programmatically using the Python pyyaml package. For now, modify the file by hand. For example, change line 13 to the project_path and line 15 to the output_path (as seen in Figure E).

Submitting a Job from a Modified Bundle

In this section, we’ll launch a GUI that displays the changes we made to our exported job bundle.

1. From a command shell on a workstation, with the pre-requisites above, type the following command: deadline bundle gui-submit C:\path_to_your_exported_and_modified_bundle

    • The submitter launches:
Figure 6:  The deadline bundle gui-submit GUI.

Figure F: The deadline bundle gui-submit GUI.

 

  1. Click on the Job-specific settings tab to see your modified project and output paths.
Figure 7:  Deadline bundle gui-submit: Job-specific settings.

Figure G: Deadline bundle gui-submit: Job-specific settings.

 

Use Case #2 – Image Resolution

You can save time and reduce rendering costs by following the pattern of modifying bundle data to achieve the Image Resolution use case. A description of all GUI controls is available.

Use Cases #3, #4 and #5 – Custom Submitter Written from Scratch

The Tab Reorganization, Fleet Custom Attribute, and Render Layer Control use cases fell outside of the customization pattern from the previous section in four areas:

  1. Tab Reorganization: You can’t add or remove tabs from the Deadline Cloud Maya integrated submitter through .yaml editing.
  2. QTree Widget: We wanted to use a QTree Widget control to display render layers, and this control is not available as a job template UI control.
  3. Custom Handlers: The integrated submitter doesn’t yet support updating template.yaml after clicking the Submit button. Custom handlers are required for querying Maya, or modifying the custom submitter GUI, and then updating template.yaml after the user clicks the Submit button.
  4. Surfacing Fleet Attributes: To give users control over the type of instances jobs would run on, we defined ‘spot’ and ‘onDemand’ fleet attributes in our farm and surfaced these attributes in our submitter.

For Tab Reorganization, the QTree Widget implementation, and Custom Handlers, we leaned into publicly available QT controls and documentation. We used the AWS Deadline Cloud API to Surface Fleet Attributes.

The FuzzyPixel Maya custom submitter streamlines user workflows by consolidating frequently used functionality under the Render Settings tab as shown in Figure H. The Render Layers tab extends submitter functionality by allowing users to control the visibility and frame ranges of render layers directly in the submitter as shown in Figure I. All of the prior use cases were implemented through Python code written from scratch and thereby provide practical exposure and context for exploring the entire OpenJD schema.

 

FuzzyPixel MayaSubmitter Render Settings tab consolidating commonly used functionality.

Figure H: FuzzyPixel MayaSubmitter Render Settings tab.

 Figure 9: FuzzyPixel MayaSubmitter Render Layers tab.

Figure I: FuzzyPixel MayaSubmitter Render Layers tab.

Conclusion

In this blog we hand edited and used Python code to modify job bundles to customize Maya submitter workflows. These customizations helped us to fulfill the use cases—minimizing interruptions to creative workflows, which saves time and ultimately reduces costs.

Contact an AWS Representative to know how we can help accelerate your business.

Further Reading

Gregory Dismond

Gregory Dismond

Gregory Dismond is a Design Technologist for Creative Tools at AWS with 35+ years of experience in areas including pipeline development at DreamWorks and EA, narrative R&D at telltale games, product development at Wavefront Technologies and computational geometry at an NSF ERC.