RedToast custom pipeline integration enhances rendering capabilities with AWS Deadline Cloud

Authored by Lisa Brodie, Founder and Producer at  RedToast. The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

RedToast started as a small production consulting company run by founder and producer Lisa Brodie. As it grew, the company saw how automation could transform workflows and bridge the gaps between production, creative, and technical teams. After analyzing how each department works, project after project, the RedToast team realized that in virtually every situation, these teams needed better communication.

The company’s new mission became fostering collaboration, whether across local teams or international crews, working together holistically instead of in silos. The team was driven by a passion for collaborative development principles that streamline production workflows.

When RedToast engaged with Amazon Web Services (AWS) to create plug-ins and proofs-of-concept for integrating with the newly launched AWS Deadline Cloud render farm service, it presented an opportunity to take the company’s capabilities to new heights.

Opening the door to software possibilities

RedToast made use of extensive customization features in Deadline Cloud to develop new submitters — plugins for creative applications including Blender, Autodesk 3DS Max, and Adobe After Effects. These are submitters anyone can use (available on the AWS Deadline Cloud GitHub repository) to submit jobs to Deadline Cloud and take advantage of its usage tracking and auto-scaling features.

RedToast was able to build tools easily and quickly, because Deadline Cloud includes comprehensive public APIs, a command line tool, and an open-source specification for submitting jobs to the service. The company used them to build the submitters, and believes that anyone can use them to tailor their own rendering pipeline, build new pipeline integrations, or create custom applications, all powered by cloud compute and management with Deadline Cloud.

The open-source job specification plays a key role in integrating pipeline tooling. Open Job Description (OpenJD) is a new, open standard for describing render workloads with a framework for building custom job types and integrating them into pipelines. Deadline Cloud directly supports jobs compliant with the OpenJD standard, which lets teams build all sorts of pipeline tools to support the software customers use for rendering.

RedToast used its pipeline expertise to dig in further and develop a proof-of-concept integration for one of the industry’s leading production management tools, Autodesk Flow Production Tracking, formerly ShotGrid.

Collaboration is key

The Deadline Cloud dashboard provides insights into render projects that complement a collaborative Flow Production Tracking pipeline, both to reduce workflow complexity and to unify production and render management in the same familiar interface. Insights include budget tracking, cost breakdowns, and resource usage relevant to a wide team.

For staff whose work takes place mostly inside the production tracking interface, or studios that have made Flow Production Tracking the core of their pipeline, it makes sense to expose Deadline Cloud budget monitoring directly inside the Flow Production Tracking interface to provide staff with real-time notifications and keep them informed of important budget events, all while keeping everyone working in the same environment.

At the same time, if specific teams can rapidly create Deadline Cloud render farms, queues, and fleets whenever they need to within the Flow Production Tracking interface, they can remain in their productive flow and simply make render farm creation a seamless part of their workflow.

Deadline Cloud usage explorer capabilities, and APIs for creating farms and queues were the tasks RedToast chose to experiment with in its quest to optimize the creative process. This was made easier because the default Deadline Cloud dashboards are powered by the same API to which the production teams have access.

New capability, unlocked

In the past, the existing ShotGrid notifications were the key to transforming the RedToast workflow. Incorporating Deadline Cloud budget limits gives the user firm control over rendering resources, ensuring that on-demand rendering never throws the budget into uncertainty.

Adding Deadline Cloud budget notifications and farm creation ability to Flow Production Tracking is straightforward; it’s just like adding any other web script.

Figure 1: Select “Add Action Menu Item” in the user interface of Autodesk Flow Production Tracking to add the integration for budget notifications and farm creation.

The budget notification system automatically generates Groups for your farm queues. Users can opt-in to receive notifications by being added to the respective queue’s group.

Figure 2: Users can opt-in to receive budget notifications as a member of a Group generated automatically for farm queues.

If a budgeted queue hits any predetermined limits, the notification system sends out notices to everyone in the group.

Figure 3: Every user in a Group will receive a notification when the Deadline Cloud budget limit set by a manager, producer or admin is reached.

The heart of the team’s workflow

Importantly, RedToast found that the fine-grained permissions in Deadline Cloud could work with its Flow Production Tracking instance to ensure that every team member has access tailored to their specific production needs. From render wranglers to production staff, everyone plays a vital role in keeping renders on track and within budget.

In particular, production staff need access to budgets and the usage explorer for setting budget limits, analyzing cost breakdowns, and preparing budget reports. This is new and valuable information available in the Deadline Cloud dashboard. RedToast wanted these front-and-center inside Flow Production Tracking, and with this new project, now it has it.

One integration, five capabilities

The Deadline Cloud integration RedToast created for its Flow Production Tracking instance has transformed how its teams handle rendering. This has been a game changer for RedToast, and we’re excited to share this product with other companies to help keep their projects on target with powerful new capabilities:

• Budget Monitoring/Notifications: Configurable budget limits can be set on rendering queues to control costs. A script inside Flow Production Tracking continuously monitors budget consumption across queues.
• Farm/Queue Creation: An Action Menu Item allows authorized Flow Production Tracking users to spin up new Deadline Cloud farms, queues, and fleets based on project data—automating a previously manual process.
• Granular Access Control: Multiple isolated farms support stringent security needs for sensitive projects while maintaining centralized render management.
• Real-Time Notifications: When budget thresholds are approached, the script automatically triggers notifications to relevant groups/users within the production tracking interface.
• Flexible Notification Groups: Flow Production Tracking Groups map to rendering queues, allowing teams to opt-in to receive only relevant budget notifications.

Try our integration

RedToast invites you to download the integration and try it for yourself. We’re using an open-source model to make integrating the notifications straightforward—simply download from our public repo, configure Flow Production Tracking credentials, and deploy:

https://github.com/redtoastproductions/deadline-cloud-for-shotgrid-farm-creator/

https://github.com/redtoastproductions/deadline-cloud-for-shotgrid-notifications/

(Please note: in the case of our workflow, the integration we built relies on a physical or virtual machine. If you’re running the integration on an Amazon EC2 compute instance, it will incur costs for scheduled push/pull action. This integration is the result of our project, and not included as part of the Deadline Cloud service.)

Also, if you choose to install it, managing Production Tracking integrations involves keeping an eye on Scripts and Action Menu Items, usually handled by your administrator. They’ll want to stay updated with any changes in your plugins and extensions, which you can find in the integration repositories. 

Continuous collaboration

Whether your team already has expertise in building functionality for Flow Production Tracking or you’re just getting started with in-house solutions, it’s easy to add support for Deadline Cloud to your studio’s Flow Production Tracking instance. RedToast was able to rely on Deadline Cloud with its well-documented developer tooling for the functionalities we developed for our POC.

As rendering technologies evolve, the integrations we’ve built can seamlessly incorporate new capabilities, keeping our workflows optimized. We believe that by sharing our project and our thought process with other companies, we can help them achieve similar timesaving strategies and enhance their overall production workflows. Collectively, with others in the industry, continuous collaboration enables us to push what’s possible.