CoMix Wave Films Leverages AWS Rendering Management Tools and Storage, Creates Hit Film Suzume with a 300-node Cloud Render Farm

CoMix Wave Films (hereinafter, "CWF") covers everything from writer management, production of animated films, theatrical distribution, package sales, and overseas selling. For the studio’s latest work, Suzume, the Systems Administration department reflected on the previous film, Weathering with You, aiming to support creativity by developing, operating, and maintaining systems while controlling the IT environment. Its two main challenges were rendering and file storage.

For Weathering with You, the studio used different rendering management tools for computer graphics (CG) and compositing. This complicated troubleshooting as errors needed to be checked for each tool. The multiple tools also swallowed more rendering resources, with the number of rendering servers increasing from 14 to around 40 over the project. “Adding hardware created more problems, such as kitting; sorting and grouping to specifications; power requirements and heat generation; and securing storage,” says Masahide Miyakogawa in Systems Administration at CWS. “We struggled to cope.”

CWF was also keen to avoid the storage problems it encountered during production of Weathering with You, where it started with a single on-premises shared storage device, but as production progressed, various problems arose and more storage devices were added each time.

“Storage sprawl made data management decentralized and fragmented,” says Miyakogawa. “We also had a lot of other issues, such as more user connections than expected and problems with transfer speeds.”

For Suzume, CWF solved its rendering issues by unifying rendering tools with AWS Thinkbox Deadline and building a cloud rendering farm. AWS Thinkbox Deadline leverages the cloud and various other resources for large-scale rendering processing while offering compatibility with popular content production software.

“We wanted to use AWS Thinkbox Deadline from the beginning, so there was no point in examining other clouds,” explains Miyakogawa. “The attentive support of the AWS media and entertainment team across storage, networks, and several other aspects was very reassuring when developing the cloud rendering farm.”

CWF built its cloud rendering farm with Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances for better scaling and cost optimizations. A major goal was to enable users to submit work to on-premises and cloud servers without a noticeable difference. After unifying rendering tools across the company with AWS Thinkbox Deadline, CWF started with an on-premises rendering server in September 2021 before spinning up a cloud server for the CG department a year later. AWS Direct Connect linked the on-premises and cloud servers, and users employed on-premises or cloud rendering servers via AWS Thinkbox Deadline.

CWF also unified its main storage. Despite the studio having solved numerous other challenges, a decline in speed due to IOPS (I/O per second) impeded production, spurring CWF to adopt Amazon FSx for NetApp ONTAP in July 2022.

“Our initial infrastructure capped out at 200 IOPS, which wasn’t enough for a production with innumerable small files,” says Miyakogawa. “Switching to Amazon FSx for NetApp ONTAP accelerated IOPS to 21,600, and we were able to clear the IOPS hurdle within budget.”

Although CWF’s local infrastructure communicated with AWS over a 10 Gbps connection with AWS Direct Connect, geographical distance caused a delay when sending and receiving a large number of small files. However, the company overcame the latency challenge by adding FlexCache caching to the local environment.

Because of plug-in restrictions for rendering, the filming department uses on-premises rendering servers for its immense graphics memory needs. Meanwhile, the CG department leverages on-premises and cloud rendering servers for its CPU-dependent rendering needs while avoiding plug-in limit caps.

According to Miyakogawa, “The CG department alone could render on up to 300 cloud nodes simultaneously. As a result, we were able to process 6,202 tasks, 2,221 jobs, and 1,205 hours of rendering in a week.”

All departments except the CG department employ Amazon FSx for NetApp ONTAP for storage. CWF was able to handle 1.6 times more data (55.6 TB) and 1.8 times more files (16.9 million) than with Weathering with You. The studio also implemented DR measures to alleviate concerns and achieve a reliable operating environment.

“Rapid recovery measures for virus infections had been an issue,” says Miyakogawa. “On-premises backups only cover the previous day, but Amazon FSx for NetApp ONTAP provides seven days of backups, so we can promptly restore and continue work if a virus strikes.”

“This was my first experience with AWS, and as a designer, I had limited knowledge of PCs,” Miyakogawa says. “It was easy to learn on our own as AWS has all the necessary documentation, and the AWS media and entertainment team provided authoritative support when asked. They also helped examine our challenges and introduced us to vendors matching our requirements and capabilities.

“As filmmakers, we’re sensitive to the speed and responsiveness of production environments, so we’ll further leverage the cloud to achieve an on-premises feel.”