Enhanced Autodesk VRED render scheduling with AWS Thinkbox Deadline
This blog was co-authored by Mike Jupp, Manager of Design Realisation & Technology, Nissan Design Europe and Rusil Patel, Sr. Manager of Cloud Infrastructure, Rivian.
With the release of AWS Thinkbox Deadline 10.3, AWS partnered with Nissan Design Europe and Rivian to overhaul the user experience and functionality of Deadline’s support for Autodesk VRED. The resulting updates allow Deadline to offer the broad set of rendering options available within VRED — enabling CPU and GPU workloads to easily be offloaded from artists and prioritized within a larger application environment. The purpose of this blog post is to explore these updates as well as the larger use-case of how Deadline can benefit automotive design studios.
“By leveraging the AWS Thinkbox Deadline solution, we are looking to expand our rendering capabilities to the cloud, providing more capacity and flexibility, which ultimately enables Rivian to deliver our amazing products faster.” – Rusil Patel, Sr. Manager, Cloud Infrastructure, Rivian.
Automotive design teams typically leverage applications such as Autodesk VRED to create high-quality media content to support design reviews and marketing of their products. Historically, on-premises hardware has been used for artist’s workstations and rendering machines, providing the ability to batch render media and to perform real-time CPU and/or GPU-distributed raytracing. Artists typically expand their workflows with other applications such as Autodesk Maya and Chaos V-Ray where appropriate (thus creating greater flexibility for content generation at the expense of a more complex software environment).
As the range of media deliverables increases alongside the complexity of the underlying assets and their respective visual quality, the desire to optimize and maximize existing resources is paramount. Studios are not only looking to aggregate the efficiency of their compute power, but they also seek to ensure that the management and accessibility of that compute is simple for both artists and administrators alike.
AWS Thinkbox Deadline is a hassle-free hybrid administration and compute management software for Windows, Linux and macOS-based render farms. It allows the scheduling of media content tasks such as rendering to be spread across studio resources. These resources are capable of running appropriate Deadline tasks—maximizing the compute power available to artists and their studio. Deadline supports more than 80 applications out-of-the-box, enabling a common environment for design teams to submit, prioritize, monitor, and process their tasks. When project requirements exceed the capability of on-premises render farms, Deadline can be used to automatically add (or scale) additional Amazon Elastic Compute Cloud (Amazon EC2) Compute Nodes (“Workers”) to accelerate the processing of Deadline tasks (including Rendering).
The main benefits of implementing a Deadline-based Render Farm in a design environment are:
- Treat performant machines in a local office as ‘Workers’ to a collective ‘Farm’ of resources—allowing tasks such as rendering to use all available resources efficiently.
- Prioritize tasks within a team, so that resources focus on the render jobs needed first.
- Artists can submit render jobs and tasks to a queue directly from their Digital Content Creation (DCC) application UI with ease. Deadline automatically distributes and executes processing of render jobs and tasks across available resources—allowing for processing completion with no human intervention.
- Since August 2022, the Deadline product has been free to use on premises.
Deadline itself comprises two main components: a database for holding the render jobs (plus related metadata), and a file system repository that contains all of the code to submit and process render jobs and tasks via respective plugins. The latter component is transparent—utilizing Python for the majority of plugins, which allows for customizations, or new plugins to be developed with a low barrier to entry.
Visualization of render jobs, their progress, resultant log information, plus Deadline configurations, etc. are done via the Deadline Monitor application. This allows Artists and Render Farm Managers alike to have an information-rich view of the current state of their render jobs, resources, etc. at a quick glance.
In addition to scheduling render jobs to be processed for batch rendering, Deadline can also assign Workers as resources to real-time distributed render engines. This allows machines to be reserved and prioritized to support Autodesk VRED Cluster or Chaos V-Ray Spawner workflows—in the same environment as other off-line tasks—enabling a single environment to control a variety of applications and their differing methods of usage. A render job is made for each clustering request, with an appropriate number of tasks dependent on the number of machines that need to participate in the rendering process. Tasks are then fulfilled by Workers, and the relevant cluster application is executed. You can allocate cluster fleets directly from the Deadline Monitor—e.g. to allocate a fleet of cluster machines to Autodesk VRED, simply click on Submit > AEC > VRED Cluster.
Deadline 10.3 updates for Autodesk VRED
Deadline 10.3 now supports Autodesk VRED 2020.0 – 2024.0 and includes new features and UI improvements for a better user experience. These changes are visible from within the Submit VRED to Deadline dialog window.
The viewing perspective of output frames is influenced by the Render View / Camera setting (1). This setting lists active camera names and viewpoints names (which take precedence). A selection of popular options from the stock Render Options dialog are now centralized, including support for Image Size Presets (2).
One new feature is support for GPU-based raytracing (3), which can be toggled in this dialog window. This eliminates the need to use custom scripting to spawn GPU raytracing upon scene loading. Please make sure that you have appropriate hardware and operating system compatibility (including GPU support and up-to-date GPU drivers) per Autodesk’s VRED version-specific System Requirement Guidelines, located here.
New Animation Clip (4) support ensures that during a render job submission, each frame of an animation within a specified frame range will be distributed to render nodes for rendering. Entire frame ranges for Animation Clips can also be automatically calculated by enabling Render Animation and Use Clip Range (5) options. If you disable the Use Clip Range setting, then you can manually specify one or more ranges (a sequence) via ‘number-hyphen-number, comma’ notation—e.g. “1-20,30-40”.
Values for the settings in the VRED Options tab are now retained between accessing this dialog window within a scene file’s session. Those settings include: Image Size Preset, Animation Clip, Animation Type, Use Clip Range, Render Job Type, Render Animation, and GPU Raytracing. While the Submit VRED to Deadline dialog window and/or the Script Editor are visible, they will now remain as topmost windows and they won’t block VRED from performing other tasks, which was not the case in previous Deadline versions; you can therefore interact with other windows in VRED in parallel.
Note: The Render Quality setting is set to Raytracing by default and Render Output format now defaults to PNG.
Deadline also stores the VRED Options settings (from the Submit VRED to Deadline interface) within a submitted render job. Those settings are then re-applied to their corresponding scene file that the Worker receives (prior to rendering). This negates the need to save the scene file for different render submissions; only functional changes to the scene file itself will require it to be saved.
For VRED 2024 (and future releases), Autodesk improved VRED’s security posture by isolating Python scripts into Sandboxes. To ensure that Deadline functions correctly, you must follow either of these two steps: disable the Python Sandbox option (a security-related option found in the main Preferences dialog under the Script tab) or alternatively, you can add to an approved list all imported modules for Deadline by their module name. Note: The Python Sandbox option is designed to disallow imported custom (or non-standard) Python modules (including Deadline); disabling Python Sandbox would allow Deadline to function correctly; however, you should first assess any related security concerns you may have in advance.
Please review the latest guidance for Deadline’s support of VRED.
Peak rendering with AWS
Deadline also offers the ability to provision extra compute resources to scale (and possibly replace) on-premises Worker resources. Fleets of cloud-based Workers scale upward to execute more tasks, and can then be shut down essentially using an “on demand” pay-as-you-go model. This brings many benefits:
- Align render resources with project requirements as needed—accommodating peak deliverable scenarios such as team growth, overlapping projects, changes to production, etc., which would otherwise tax limited on-premises resources to a breaking point.
- See cloud-based Workers in the same Deadline Monitor pane as on-premises Workers. Treat them as a simple extension of the current resource fleet.
- Use Deadline to automatically and repeatedly spawn Cloud-based Workers within the limits of the fleets that you define. Automatically terminate these Workers once their processing tasks are complete.
- In hybrid-Worker scenarios (where Workers are a mix of on-premises and cloud-based Workers), use Deadline’s Pools and Groups to prioritize on-premises Workers over cloud Workers. This ensures that cloud-based Workers are relinquished as soon as prudently possible.
AWS offers over 630+ different kinds of Amazon EC2 instances (at the time of writing). These include an array of GPU and CPU-based machines that accommodate distributed real-time raytracing and/or rendering workloads. For example, the fifth-generation Amazon EC2 graphics instance class (G5), boasts an impressive arsenal of hardware features—enabling it to perform GPU shader-based parallel processing, NVIDIA OptiX workloads, and machine learning—all of which are used by VRED to create fast, accurate, ray-traced imagery. Additional use cases for this instance class include: spawning graphics workstations to author Autodesk VRED scenes, joining a VRED Cluster Service fleet for distributed rendering, and serving as a Worker within a headless (non-UI based) GPU fleet for rendering projects via Autodesk VRED Core.
As an alternative, CPU-based cluster fleets can use VRED’s rendering service on an array of cost-effective Amazon EC2 compute-based instance classes (with different resource sizes), such as the M7a—using the AMD 4th generation EPYC (Genoa) processors (up to 192 vCPUs).
For more information on this topic, please review Virtual Prototyping with Autodesk VRED on AWS.
Deadline AWS Portal is a recommended solution for studios that seek to burst render jobs to the cloud for rendering, while seamlessly maintaining asset synchronization (both to and from the cloud). This solution enables Deadline users who have SuperUser access (Deadline administrative privileges), to securely deploy AWS cloud infrastructure directly within the Deadline UI. Asset synchronization allows one to have dependent data stored in the cloud automatically (for when that data is needed). Further, that data is cached securely in cost effective Amazon Simple Storage Service (S3) for use in cloud-based Workers.
For customers seeking more advanced integrations with AWS (e.g. via a dedicated connection such as AWS Direct Connect) and for more explicit control of asset synchronization, Deadline’s Spot Event Plugin can integrate into custom solutions. This can facilitate the creation of AWS cloud-based fleets in an elastic capacity—i.e. to auto-scale cloud Workers when required, within the limits of the fleet type and size that you define, along with the ability to automatically terminate those Workers when their processing effort is complete.
In the following architectural diagram, you will notice a custom solution for a Deadline Worker fleet that extends into the AWS cloud. Custom fleet requests can be associated with Deadline Groups, allowing for a customization of instance types to accommodate the required processing needs of a render job. To facilitate rendering, a high-performance file system such as Amazon FSx for OpenZFS can serve to support Workers.
Automotive design studios and AEC related businesses have found Deadline to be incredibly valuable in supporting their workloads, both in efficiency and scalability. This blog post describes some of those opportunities, including how compute resources can be aggregated and scaled into larger fleets for artists and studios—offering a single (merged) environment to manage a variety of DCC applications as well as methods for resource utilization. AWS is committed to the continued development of workflows supporting products such as Autodesk VRED within Deadline and to working with customers to ensure that features evolve and align with the needs of their businesses.
We look forward to working with our customers! As always, if you have any questions or feedback, please reach out to your account team or the authors of this blog to continue the conversation.
More information (including Deadline 10.3 documentation), can be found at https://www.awsthinkbox.com/deadline.
Brief tutorial videos can be found on the following AWS Thinkbox Resources website; these tutorials include demonstrations of the steps to deploy Deadline, and cover extended functionality such as AWS Portal. https://www.awsthinkbox.com/resources/deadline/aws-portal
The Deadline Forum is a great resource for sharing inquiries and feedback with the Deadline community: https://forums.thinkboxsoftware.com/.
For additional Deadline-related blogs from AWS, including details on support for Hammerspace’s global file system technology, and the recent inclusion of Unreal Engine into Deadline, please visit: https://aws.amazon.com/blogs/media/category/media-services/aws-thinkbox/
As a world leader in 3D design, engineering, and entertainment software, Autodesk delivers the broadest product portfolio, helping over 10 million customers, including 99 of the Fortune 100, to continually innovate through the digital design, visualization, and simulation of real-world project performance.