Vela Games Cuts Game Build Times by 60% Using Infrastructure on AWS

Vela Games (Vela) is using the power of the cloud to develop its first game faster. By using scalable infrastructure on Amazon Web Services (AWS), Vela can do more with less, freeing up time and resources to focus on creating a fantastic gaming experience for players. Vela powers its build farms with simple, high-performance shared storage using Amazon FSx for OpenZFS—which provides fully managed shared file storage built on the OpenZFS file system—reducing build times by 60 percent.

Since 2017, Vela has been developing its first title, Evercore Heroes, a near-real-time, skill-based, competitive player-versus-environment experience. Powered by Unreal Engine from Epic Games Inc. (Epic Games), an AWS Partner, Evercore Heroes is built using some of the latest cloud technology. From the start, Vela has used AWS services, such as Amazon Elastic Compute Cloud (Amazon EC2), which provides secure and resizable compute capacity for virtually any workload.

“The key for a startup is speed of iteration,” says Adam Comerford, technical director of infrastructure and operations at Vela Games. Vela identified that its development velocity would depend on how quickly its developers could build and test new features. “Anything we can do to make it easier to build, deploy, and test the game helps us get out of the developers’ way,” says Comerford.

With Vela’s initial architecture, however, the continuous integration (CI) pipelines ran slowly, creating a bottleneck for developers testing the game. It sought to redesign its pipelines to run parts of the build jobs in parallel rather than sequentially.

“We wanted to improve build times to be able to iterate faster and not have developers waiting on a pipeline to run for an hour just to test the game,” says Esteban Garcia, senior infrastructure engineer for platform engineering at Vela Games.

Using Amazon FSx for OpenZFS to Speed Up Build Times

After watching Epic Games’ talk at AWS re:Invent 2021, Vela set out to improve its infrastructure pipelines. Vela changed its CI system to CircleCI and started using BuildGraph, an Unreal Engine tool that lets users split up the different job stages of compilation, asset cooking, and deployment to run those stages in parallel. This change has significantly decreased compile times, so Vela can iterate faster.

To use BuildGraph effectively, Vela needed a simple, high-performance shared storage solution. Based on the example of Epic Games—which uses Amazon FSx for Windows File Server, a fully managed file storage built on Windows Server—Vela chose to use another Amazon FSx solution: FSx for OpenZFS. FSx for OpenZFS provides the native Linux compatibility that Vela Games needed for its use case while offering the same simple, fully managed storage experience and data management capabilities of other Amazon FSx file systems. FSx for OpenZFS delivers over 1 million IOPS and 12.5 GBps throughput per file system with latencies of 200–300 microseconds. Each file system can scale up to 500 TiB.

Vela stores artifacts in FSx for OpenZFS. The CI pipelines can access these artifacts to run multiple jobs in parallel, such as fast intermediate file sharing and running the Derived Data Cache from Unreal Engine. “Amazon FSx is one of the main components that make this pipeline work,” says Garcia. When compiling code or cooking assets are complete, BuildGraph copies the resulting artifacts from that job to FSx for OpenZFS. Vela uses the long-lived shared storage to retain intermediate results across sequential jobs that would otherwise be deleted after the job is complete. Subsequent jobs can use these results to skip a lengthy initialization process and run much faster.

“The speed and simplicity of Amazon FSx unlock our ability to use BuildGraph. Using this solution, we can parallelize and reuse assets quickly, which cuts down build time,” says Comerford. By using FSx for OpenZFS and BuildGraph, Vela has reduced its total build time from an average of 56 minutes to 24 minutes.

Updating Architecture to Run Fast, Scalable Build Farms on AWS

Vela uses a multiaccount strategy on AWS. The infrastructure services account runs tooling that supports game development, including parallelized jobs from CircleCI. Vela created a script layer that interfaces between CircleCI and BuildGraph, and it runs build jobs, which use FSx for OpenZFS, on Amazon EC2. The pipeline now takes 25–30 minutes and uses AWS Auto Scaling, a service that monitors applications and automatically adjusts capacity to maintain steady, predictable performance. Vela developed an in-house solution that queries CircleCI for job requests and spins up new instances using AWS Auto Scaling groups.

Another account manages Vela’s containers using Amazon Elastic Kubernetes Services (Amazon EKS), a managed container service to run and scale Kubernetes applications. “Creating a Kubernetes cluster takes mere minutes using Amazon EKS instead of hours,” says Garcia.

Vela chose to use AWS because it already had good experiences using the environment and because of the outstanding support it receives from the AWS team. “The team at AWS has been very receptive and proactive,” says Comerford. “They put us in touch with other games companies and have given us real-world examples.”

Improving Speed to Market with a Lean Team Using AWS

Using AWS, Vela speeds up the development process by giving developers faster turnaround times to test changes to the game. The reduction in build times lets developers complete more tests and achieve better results for the game overall. Vela runs its pipeline 20 times per day on average; using its AWS architecture, it saves its developers an average of 640 minutes in manual labor per day. That time savings allows them to focus on more innovative tasks that generate value, such as building and testing new features.

Vela has begun running play tests in various regions. Using AWS, Vela can spin up servers in only 2 days to run testing in different regions, such as the United States and Southeast Asia. “The number of regions that AWS runs in gives us flexibility and better latency for players,” says Comerford. “We can quickly spin up a new Amazon EKS cluster to get our game servers out there.”

By using AWS, Vela’s infrastructure team can remain small while still accessing a wide range of powerful tools and managed services. This frees up most of Vela’s team to dedicate themselves to game development, which is the company’s main focus. “We’re a games company, not an infrastructure company,” says Comerford. “We want to use infrastructure in a way that gives us an advantage to get the game to market to as many players as possible and as economically as possible.”

By using AWS, Vela achieves quick, simple scalability. Its developers can run complex programs in the cloud directly from their laptops without worrying about server support or maintenance. With a remote and hybrid workforce, Vela has grown from 12 to 50 employees in just 2 years while keeping the infrastructure team small.

Vela Games is always looking for talented and ambitious individuals to help build the platform for the upcoming release of Evercore Heroes. “Using AWS makes it easier to be a startup, which is priceless,” says Comerford. “Working within our AWS environment is an excellent opportunity to work on a cutting-edge project and hone skills in a fun, collaborative environment.”