Redhill Games Operates Games at Scale Using Amazon GameLift and Amazon EKS

In 2019, multinational gaming company Redhill Games (Redhill) introduced Nine to Five, a new kind of first-person shooter game that prioritizes team collaboration over individual reflexes. Because Nine to Five involves rapid 3v3v3 battles that unfold over three phases, achieving horizontal scalability to accommodate many players at the same time was a top priority. To scale to large numbers of game servers at a fast rate, Redhill needed a reliable control plane and solutions that would enable efficient resource usage and cost optimization.

Since Redhill’s beginning, it has been building on Amazon Web Services (AWS), so it looked for additional AWS solutions to improve performance and deliver inexpensive, resilient game hosting for players around the world. Amazon Elastic Kubernetes Service (Amazon EKS), a managed service that enables scalability of containerized applications on AWS, enabled Redhill to scale horizontally, and Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances, which enable users to take advantage of unused compute capacity on AWS, helped Redhill optimize costs.

Redhill also used Amazon GameLift, a dedicated game server–hosting solution that deploys, operates, and scales cloud servers for multiplayer games. And by using Amazon GameLift FleetIQ, which optimizes the use of Spot Instances for cloud-based game hosting, Redhill reduced the risk of reclaimed instances. “There are so many tools on AWS to solve different problems,” says Mikko Uromo, chief technology officer and cofounder of Redhill. “Hooking them up to talk to each other is simple and has been a huge time-saver for us.”

Founded by a team with extensive experience developing AAA PC titles, Redhill pursued a new challenge: to create online gaming experiences for cooperative play and to heavily incorporate player testing and early feedback to drive player-centric development. The result, Nine to Five, requires three teams of three players to battle each other in rapid-fire rounds, and the game offers different ways to approach problems. “There are multiple pathways you can take and multiple ways to defend a position,” says Uromo. “You have the freedom to maneuver and experiment.”

Built on Unreal Engine for Windows, Nine to Five also uses a Linux server running on AWS. Because of the importance of horizontal scalability for Nine to Five and the complexity of the game, Redhill decided to use container services—specifically, Amazon EKS. And because of the short life span of game servers, it used Spot Instances as the underlying compute model for Amazon EKS cluster nodes. Redhill estimates that it saves 30 percent on costs for every battle server it runs using Spot Instances.

Amazon GameLift FleetIQ provides the Redhill team with direct control over hosting resources in Amazon EC2, which provides secure, resizable compute capacity in the cloud, and AWS Auto Scaling, which automatically adjusts capacity to maintain steady, predictable performance. And while Amazon GameLift FleetIQ applies specifically to Spot Instances, Redhill developed an innovative solution that enables Amazon GameLift FleetIQ to manage automatic scaling groups of Amazon EKS cluster nodes, too, so that the company can use containers for its game servers.

Altogether, the solution combines the control-plane resilience and rapid scaling from Amazon EKS and the benefits of Amazon GameLift FleetIQ, including the ability to decrease Spot Instance reclamation rates and service interruptions. “I feel very confident that with the current architecture, we can support all the players who want to play,” says Vinicius Suarez, lead backend programmer at Redhill. “We wouldn’t be able to say that without the support we get from AWS.”

In May 2020, Redhill invited approximately 50,000 early alpha testers to take Nine to Five for a trial run. During testing, it was important for the company to be able to efficiently run servers anywhere in the world so that players could experience the game without disruption. Redhill accomplished that goal at an early stage of development, enabling the team to immediately start collecting and incorporating player feedback. With the infrastructure in place, Redhill can apply server-side patches with zero downtime to players. The company also used AWS to test new features inexpensively and run real-time analytics—such as how many shots it takes players to bring down a target—to understand and improve the player experience. To achieve this, Redhill used Amazon Redshift, a cloud data warehouse for analytics, to query data in a data lake. This was key to gaining insights from game play, and the Redhill team used AWS Glue—a serverless extract, transform, load service—to transform incoming JSON data to database tables and save it to Amazon Simple Storage Service (Amazon S3), an object storage service. Finally, Amazon API Gateway, a fully managed service for managing APIs, enabled Redhill to access and monitor data. With real-time data, Redhill can find bugs early and fix them quickly. “For instance, we have a robust armor system for player characters. But when a player shot them with a shotgun, nothing would happen,” explains Uromo. “Since we were tracking each pellet from the shotgun, we could see from the analytics that actually, only one pellet was hitting.”

The next phase was to hold a limited “closed alpha” launch of Nine to Five in June 2020. During the launch, the game’s infrastructure was able to scale from zero to several thousand nodes in less than 2 hours without significant interruptions or reclamation rates. “We had zero issues with scalability,” says Uromo. “We were prepared to have someone always on call to make sure everything was running smoothly, but after the first day, we were like, ‘I guess we can go to sleep now.’” On AWS, the Redhill team doesn’t have to manage its own servers, and it uses AWS solutions for everything from code pipelines to player identity and matchmaking services. The team can focus its efforts on quickly iterating new fixes and features instead of worrying about server maintenance.

Redhill split its servers for North American and European regions during alpha testing, and in February 2021, it held a limited beta weekend on Steam for about 100,000 players from five regions: Europe, US East, US West, Asia, and Australia. Global APIs enabled gamers from around the world to play together smoothly. The team at Redhill also plans to take the custom solution it developed using Amazon GameLift FleetIQ and Amazon EKS and make it open source so that everyone in the gaming industry can access the technology.

Developing and testing Nine to Five on AWS enabled Redhill to focus its efforts on creating innovative solutions and experiences—and to respond quickly to player feedback—rather than on the nuts and bolts of backend architecture. “If we’d had to build all that from scratch, we never could have followed our timeline,” says Suarez. “We would have spent so much more energy and effort just on building. But now, we can focus on making the game fun.”