Containers

Game DevOps made easy with AWS Game-Server CD Pipeline

This is a guest post by Anita Buehrle of Weaveworks.

The biggest challenge faced by game publishers is the ability to deliver new features to players as quickly as possible. Not only do new features have to arrive quickly and reliably, but they also need to be delivered in a way that optimizes costs and maintains security. Continuous integration and continuous deployment (CI/CD) pipelines with GitOps is an effective way for game publishers to improve games and deliver new features throughout the development lifecycle. In this post we’ll discuss how CD pipelines when combined with GitOps best practices can be used to automate the distribution of game binaries and their assets to an EKS cluster on AWS in a scalable, secure, and cost effective way.

Overview of CI/CD pipeline architecture

The diagram below describes the CI/CD pipeline architecture and its data flow. We use GitOps to manage both infrastructure as well as the application configuration for the game in git as a single source of truth. With everything kept in git, developers create pull requests to operate and deploy updates to game-servers running on EKS without having to log directly into the cluster. With GitOps, the state of the production cluster is continuously compared with the declarative configuration kept under source control.

GitOps assumes that your CI system has tested and integrated the newly pushed code. The CI portion of the pipeline also builds and pushes a Docker image to the Elastic Container Registry (ECR). With the newly built images in the registry, Flux CD takes over from there. Flux is a GitOps operator that runs in the cluster and watches for new images in ECR. When Flux notices that a new image has been pushed to ECR, it checks out the corresponding manifest file updates it and then checks it back into git. After which the new image is automatically deployed to the cluster, updating the game server and maintaining its state against the source of truth kept in git.

 

Why Continuous Deployment (CD) for games development?

CD enabled by GitOps for managing game-server deployments benefits both game developers and game operations teams in the following ways:

Increased developer productivity – The game development process is exciting. Developers continually build immersive, ambitious experiences to keep players playing. This requires a mechanism so that developers can ship game updates and features faster. GitOps allows developers to use familiar tools like git to manage updates and new game features without having to know the internals of Kubernetes. Since deployments are managed through pull requests any platform nuances that host the game are abstracted. For example, developers do not need to know where the game is being deployed, whether it is EKS or ECS.

Streamlined game operations – As with any other type of software, game players expect stability and reliability. When you use Git workflows to manage your cluster, you automatically gain an audit trail of who did what, and when to your cluster ensuring stability of the game-server host. GitOps also defines consistent and standardized workflows that allow DevOps teams to use across all deployments regardless of the game-server. Finally, since access to game-server clusters and other infrastructure is limited security is vastly improved. Using the GitOps model, developers do not need kubectl or the AWS Management Console to push changes to the staging system. Deployment permissions can instead be managed through the structure of your git repository. If AWS CodeCommit is chosen as your git provider, permissions can be set with AWS IAM.

How GitOps helps games DevOps?

Consider a new DevOps team member who just deployed a new game-feature that has a bug and you are the on-call DevOps person for the day. Before the change is fully rolled out, kubediff fires an alert citing the change in git that caused the problem.

With that small ad hoc config modification, problems will not be allowed to compound. Instead, your dedicated game-server deployments are more consistent and stable, reducing heroism when your team is continuously delivering changes to production environments.

How does it work?

For Continuous Deployments (CD) with GitOps using Flux, the manifest files for the game-server and assets Docker images must be kept a separate git repository from the game binary raw code.

The CD system is therefore comprised of:

  • An EKS cluster that hosts the dedicated game-servers.
  • A template GitOps profile that includes Flux CD, as well game-server specifications and monitoring components. Flux is the GitOps operator that manages deployments to maintain cluster state. The monitoring components send logs to synchronize with CloudWatch or to ElasticSearch. Operational metrics are sent to Prometheus as well as native Kubernetes dashboards.
  • Accessible Docker images with the game binaries and assets in the ECR container registry.

To set up the cluster with Flux and a profile with eksctl:

Firtst, we deploy EKS cluster in a respective region using eksctl tool. For example:

eksctl create cluster -f cluster.yaml

We use cluster.yaml as an example cluster spec.

Next, enable the GitOps operator to setup the Flux and Helm on the cluster created.
This is an experimental feature. To enable it, set the environment variable EKSCTL_EXPERIMENTAL=true.

eksctl enable repo -r us-west-2
--git-url git@github.com:me/my-gs.git \
--git-branch my-gs-us-west-2 \
--git-email devops@example.com \
--cluster gs-us-west-2 \
--region us-west-2

The enable repo command clones the repo set by —git-url to temporary directory. The command takes a while to run and it’s a good idea to scan the output. You note a similar bit of information in the log like this one:

[ℹ] Flux will only operate properly once it has write-access to the Git repository [ℹ] please configure git@github.com:me/my-gs.git so that the following Flux SSH public key has write access to it ssh-rsa AAAAB3NzaC1yc2

Copy the lines starting with ssh-rsa and give it read/write access to your repository. For example, in GitHub, by adding it as a deploy key. There you can easily do this in the Settings > Deploy keys > Add deploy key. Just make sure you check Allow write access as well. The next time Flux syncs from Git, it will start updating the cluster and actively deploying. If you run git pull next, you see that Flux has committed them to your git config repository already.

Third, enable the profile on the Oregon EKS cluster:
This is an experimental feature. To enable it, set the environment variable EKSCTL_EXPERIMENTAL=true.
Replace the —get-url value, me/my-gs.git with the git repository you ought to use for controlling the EKS cluster.

eksctl enable profile -r us-west-2 \
--git-url git@github.com:me/my-gs.git \
--git-branch my-gs-us-west-2 
--cluster gs-us-west-2 \
git@github.com:aws-samples/amazon-eks-profile-for-gameserver

This command sets up your cluster with the amazon-eks-profile-for-gameserver profile. All of the config files you need for a production-ready cluster are in the git repo you have provided and those components are deployed to your cluster. When you make changes in the configuration, they are reflected on your cluster.

[ℹ] cloning repository "git@github.com:aws-samples/amazon-eks-profile-for-gameserver":master
[ℹ] processing template files in repository
[ℹ] writing new manifests to "/var/folders/n9/53xbvtmd7sjg1q1l55xmpg58n4mgng/T/my-gs/base"

Deploying the game-servers to a different cluster in another region

When you need to deploy a game-server to a different cluster or to an entirely new cluster, you can take advantage of separate git branches.

First, we first deploy the secondary EKS cluster in a respective region using eksctl tool. For example:

eksctl create cluster -f cluster.yaml

Next, the following eksctl command, uses the same git repository, but specifies a different git branch for the cluster:

eksctl enable repo -r eu-west-1 \
--git-url git@github.com:me/my-gs.git \
--git-branch my-gs-eu-west-1 \
--git-email devops@example.com \
--cluster gs-eu-west-1 \

This command produces similar results as the previous enable profile execution. I demonstrates the flexibility provided by git branches. Clusters are assigned per branch and can be used to deploy gradual cross-regional changes using standard git pull requests and merge requests with no special tooling required.

Third, enable the profile on the Dublin EKS cluster:

eksctl enable profile -r us-west-1 \
--git-url git@github.com:me/my-gs.git \
--git-branch my-gs-eu-west-1 \
--cluster gs-eu-west-1 \
git@github.com:aws-samples/amazon-eks-profile-for-gameserver

The outputs of these commands are similar to the outputs of the Oregon region setup.

Try it out for yourself

We published four examples to guide you through the process of building an Amazon EKS cluster and deploying a containerized game server with large binaries.