Taking Apocalypse Studio to the cloud via AWS

In this guest post, we hear from Apocalypse Studio founder Denis Dyack. Denis has a long history in game development, working on titles including Blood Omen: Legacy of Kain, Eternal Darkness: Sanity’s Requiem, Metal Gear: Solid Twin Snakes, and Too Human. Denis also has a background in Computer Science, earning a Masters. Sc. in Artificial Intelligence, Neural Networks and User Interfaces.

Based in Niagara, Ontario, Apocalypse is a game studio startup. By taking a cloud-first approach, our goal is to transform the way people play games. Creating online multiplayer experiences, we aim to connect and entertain players around the world.

With over 30 years in the games industry, I’ve worked on traditional console titles, including Eternal Darkness, Metal Gear Solid: Twin Snakes and Too Human. Over that time, the industry, and the technology we use to power it, has altered substantively. When we started Apocalypse Studio in 2018, we knew that we needed a fresh, cloud forward perspective.

Our studio and first title Deadhaus Sonata are quite literally built from the cloud up. When COVID-19 began, we transitioned from traditional office space with an in-house custom-built server, to using a studio in the cloud built on AWS. It was so successful that we permanently closed our office in mid-July of 2020. Now we can support our employees to work comfortably in remote locations indefinitely.

Understanding clouds

The cloud has had a significant economic impact on games. Over a decade ago, I forwarded that its social and economic value would be higher than the traditional premium business model because of commoditization. Now free-to-play (F2P) online games account for 86% of world-wide gaming revenue.

To become a cloud-first company, we needed to work from the ground up, starting with our engine. We wanted to create a game for the cloud, allowing for deep social media integration and the highest quality standards. We compared support, cost, features, and the long-term outlook of all of the engines. For us, Amazon Lumberyard outperformed them all when combining these categories. Choosing Amazon Lumberyard had several extra benefits that helped propel us through the successful transition to the cloud.

When we committed to Amazon Lumberyard, we immediately built several relationships with groups within AWS Game Tech. We enthusiastically discussed build processes, procedural generation, the future of video games, and much more with these teams. We also discussed what we thought was the far-out possibility of moving our entire production process to the cloud.

Propelling our journey into the cloud

Fast-forward to March 2020. We were preparing for the Game Developers Conference (GDC) in San Francisco. We had just completed “Memento Mori” for Deadhaus Sonata – a narrative-driven action RPG where you play as the undead fighting the living. Then GDC and other large gatherings were suddenly closed in response to the COVID-19 pandemic. Our offices were next.

We now had the challenge of supporting our team with remote connectivity and development. We had to work quickly and there was much to consider. The last mile of internet connection from the main optical hubs to residential housing is notoriously slow and troublesome. Game development is also extremely asset-heavy. Would people be able to work at home with non-commercial internet connections? What would productivity be working from home as opposed to an office setting?

To maintain continuity for our team, we turned to AWS for assistance. We immediately uploaded a backup of our entire Git repository that houses our codebase and game assets, to an Amazon Simple Storage Service (S3) bucket. This took a week to complete.

Simultaneously, we launched a new Git server using Amazon EC2. Within hours of completing the backup upload, our Git server was up and running again. These builds were previously only accessible through LAN at the office. Today, our source code and assets are globally and securely available, with no physical hardware to maintain.

Next we discussed how to get automatic daily builds running on AWS. Our previous solution was to run builds on a separate, dedicated server that had to be in the same room as our Git server. Discussions started in earnest to experiment with automatic builds for Lumberyard.

After investigating several paths, we adopted a custom-built solution created entirely on AWS. We use AWS Lambda to run periodic polls for pending build jobs on our GitLab instance. If there are any such jobs, it launches instances from an EC2 Launch Template. Build artifacts are uploaded to Amazon S3 and then later downloaded by a simple script. This allows our developers to stay up to date with latest changes without having to build Lumberyard locally. Previously, our daily builds were taking between 6–7 hours on a custom-built server in the office and were completed nightly. Now our builds on AWS are completed in 20 minutes, at the cost of only $2 per build.

To monitor the health of our entire architecture, we use Amazon CloudWatch. We receive email alerts if any part of this pipeline ever fails, but in nearly three months of operation, it hasn’t failed once. Our previous solution was error-prone and required constant maintenance. Using Amazon CloudWatch again helped us remove several pieces of physical hardware, which we no longer have to maintain.

Apocalypse Studio, powered by AWS

Since moving our game production pipeline to the cloud, we’ve seen some significant benefits. Using AWS, we are able to transfer large game assets between colleagues. And while data transfer does take marginally longer, it has no meaningful impact on the process. Our team works effectively from home, and we continue to explore how AWS can support our game. We are adding multiplayer support into Deadhaus Sonata, using AWS Lambda, Amazon API Gateway, Amazon Cognito, Amazon DynamoDB, and Amazon GameLift. We’re also migrating our storefront to AWS.

Before the transition we feared the last mile. We were prepared to upgrade home internet connections of our developers to mitigate issues they may face. To our surprise, the concern never became a reality. We thought the costs of using AWS would be prohibitive but are well below expectations, and performance is well above. While our team misses face-to-face interaction, and communication is more optimal this way, no one feels that interaction outweighs personal safety and the risks associated with COVID-19. We have permanently transitioned to remote work stations, never looking back. We would never have been able to make this transition so seamlessly without the help of AWS.