AWS for M&E Blog
Using Open Job Description to Customize Submitter Workflows
Are you an Artist or Technical Director (TD) who’d like to streamline the process of defining and submitting jobs to your render farm? This blog describes how to use Open Job Description (OpenJD) to customize the Amazon Web Services (AWS) Deadline Cloud integrated submitters, and concludes with a code sample of a custom submitter written from scratch.
OpenJD is an open specification that allows users to define jobs to submit to a render farm. Job submission takes place toward the end of content creation workflows, and OpenJD provides opportunities for TDs and technical artists to customize submitter workflows to minimize workflow interruptions. Submitter customizations can save time and effort for users submitting jobs to AWS Deadline Cloud.
The Deadline Cloud integrated Maya 2024 Submitter is shown in Figure A. It provides users with core functionality for submitting Maya jobs.
Customization Use Cases
The FuzzyPixel custom Maya submitter was developed by the FuzzyPixel team. We are an AWS team working on real world productions to help design, prototype, and test AWS features before they are released. In this blog, we’ll focus on the customizations we developed during our most recent production.
The use cases we’ll cover are:
- Updating Project and Output Paths: When project and output paths are automatically populated based upon the opened scene, it saves time and reduces cognitive load for users.
- Image Resolution: A job might have a final render resolution of 8K. Allowing users to scale down image resolution for test renders saves time and reduces rendering cost.
- GUI Re-Organization: The integrated submitter organizes GUI controls using four different tabs. Placing frequently used controls under a RenderSettings tab helps to streamline job submission workflows.
- Fleet Custom Attribute: When users are allowed to choose whether their jobs run on a Spot or On-Demand Instance, it helps them to avoid job interruptions for longer renders.
- Render Layer Control: Giving users explicit control of render layers in the submitter allows them more control over render layer visibility and frame ranges.
We fulfilled all of the use case requirements using OpenJD, the AWS Deadline Cloud API, the Maya API, PySide2, and Python scripting. Let’s take a closer look at OpenJD before exploring the first use case.
Prerequisites
All of the following prerequisites are required before continuing:
- A Windows workstation.
- An installation of Maya 2024 and the AWS Deadline Cloud Submitter.
- Note that Maya 2024 has its own fee structure and this is not associated with AWS
- A farm, with a fleet and queue, plus the Deadline Cloud monitor and a profile created for your farm.
- An installation of Python-3.10.8 (For best results, use this specific version of python to match Maya 2024’s python version)
- https://www.python.org/downloads/
- pip install deadline and gui dependencies into this version of python
- For example, “C:\Program Files\Python-3.10.8\PCBuild\amd64\python.exe” -m pip install deadline[gui]
- Download the FuzzyPixel custom Maya submitter.
Note: All examples provided within this blog run only on Windows workstations.
A Look at OpenJD with Job Bundles
In the image of the Deadline Cloud integrated submitter (Figure A), there’s an Export bundle button at the lower right. A job bundle is a complete description of a job that runs on a Deadline Cloud render farm. In this section, we’ll take a look at a job bundle exported from a Maya scene as a first step toward understanding how the previously stated use cases were fulfilled.
Export a Job Bundle
- Open Maya and load a scene to submit to your render farm.
- After the scene loads, launch the submitter. If you have installed the submitter correctly, you should see an AWS Deadline tab as shown in Figure B (on the far-right side of the screen). If you don’t see the submitter, go to Windows, then Settings/Preferences, and lastly Plug-in Manager. In the popup window that appears, type Deadline, and then select Loaded and Auto load before exiting the window.
- When you see the Deadline Cloud icon appear on the AWS Deadline shelf, click on it and the Maya integrated Submitter will launch. Login to your farm by following these instructions.
- After launching the submitter, click the Export bundle button. After exporting, a file browser and confirmation modal will pop up. The file browser will show a folder with three files containing the job bundle.
A Description of the Job Bundle
Each job bundle folder exported from Maya contains three files: one for asset references, one for parameter values, and one template file to describe the exported job.
- The asset_references.yaml file contains asset paths, the output folder, and any referenced paths for the job. If you are using a Service-Managed Fleet (SMF), this is the vehicle for uploading your assets to the render farm. If you are using a Customer-Managed Fleet (CMF) with storage profiles, the lists specified in asset_references.yaml can be empty.
- The parameter_values.yaml file conveniently contains any values you intend to reference in your job template. For our Image Resolution use case, an example of a set of parameter values would be ImageWidth and ImageHeight.
- The template.yaml file contains a description for the job. It can reference the asset_references.yaml and parameter_values.yaml files to fully represent the job. While it’s convenient to use all three files, a job can be completely described using only the template.yaml file.
We’ll use and build on the job bundle to fulfill our first use case: Updating Project and Output Paths.
Use Case #1 – Updating Project and Output Paths
The FuzzyPixel team wanted to populate the project and output paths using python code that derives them from the currently opened Maya scene.
Here’s how we approached this:
- Perform Maya queries,
- There are a variety of ways to get the current path of a Maya file. We used the following:
project_path = cmds.workspace(q=True, active=True)
- For the output path, we concatenated the project path with a folder as shown in Figure G:
output_path = f’{project_path}/images’
- There are a variety of ways to get the current path of a Maya file. We used the following:
- We then used the query results to populate the project and output fields in the submitter.
- Once we have values for the project and output paths, we can inject the data into the parameter_values.yaml file described in the previous section.
- Open the parameter_values.yaml file you exported earlier and find the line in the file that refers to the project and output paths. The exported bundle with these lines highlighted can be seen in Figure E. The appropriate lines for our exported bundle .yaml file are:
- You can update the parameter_values.yaml file programmatically using the Python pyyaml package. For now, modify the file by hand. For example, change line 13 to the project_path and line 15 to the output_path (as seen in Figure E).
Submitting a Job from a Modified Bundle
In this section, we’ll launch a GUI that displays the changes we made to our exported job bundle.
1. From a command shell on a workstation, with the pre-requisites above, type the following command: deadline bundle gui-submit C:\path_to_your_exported_and_modified_bundle
-
- The submitter launches:
- Click on the Job-specific settings tab to see your modified project and output paths.
Use Case #2 – Image Resolution
You can save time and reduce rendering costs by following the pattern of modifying bundle data to achieve the Image Resolution use case. A description of all GUI controls is available.
Use Cases #3, #4 and #5 – Custom Submitter Written from Scratch
The Tab Reorganization, Fleet Custom Attribute, and Render Layer Control use cases fell outside of the customization pattern from the previous section in four areas:
- Tab Reorganization: You can’t add or remove tabs from the Deadline Cloud Maya integrated submitter through .yaml editing.
- QTree Widget: We wanted to use a QTree Widget control to display render layers, and this control is not available as a job template UI control.
- Custom Handlers: The integrated submitter doesn’t yet support updating template.yaml after clicking the Submit button. Custom handlers are required for querying Maya, or modifying the custom submitter GUI, and then updating template.yaml after the user clicks the Submit button.
- Surfacing Fleet Attributes: To give users control over the type of instances jobs would run on, we defined ‘spot’ and ‘onDemand’ fleet attributes in our farm and surfaced these attributes in our submitter.
For Tab Reorganization, the QTree Widget implementation, and Custom Handlers, we leaned into publicly available QT controls and documentation. We used the AWS Deadline Cloud API to Surface Fleet Attributes.
The FuzzyPixel Maya custom submitter streamlines user workflows by consolidating frequently used functionality under the Render Settings tab as shown in Figure H. The Render Layers tab extends submitter functionality by allowing users to control the visibility and frame ranges of render layers directly in the submitter as shown in Figure I. All of the prior use cases were implemented through Python code written from scratch and thereby provide practical exposure and context for exploring the entire OpenJD schema.
Conclusion
In this blog we hand edited and used Python code to modify job bundles to customize Maya submitter workflows. These customizations helped us to fulfill the use cases—minimizing interruptions to creative workflows, which saves time and ultimately reduces costs.
Contact an AWS Representative to know how we can help accelerate your business.