AWS for Games Blog
How ArenaNet moved Guild Wars to the cloud: An MMORPG migration story
ArenaNet is the wholly owned subsidiary of Korean online game publisher NCSOFT that developed the critically acclaimed series of games in the Guild Wars franchise. Guild Wars (released 2005) and the successor Guild Wars 2 (released 2012) both focus on player skill and horizontal player progression where players mix skills into novel builds in cooperation with their friends. Guild Wars 2 took the franchise into a fully rendered and explorable 3D world. After launch, Guild Wars 2 became the one of the fastest selling MMORPG’s of all time, and that success has continued for years, with the release of two expansions and regular ‘Living World’ episodes which are free to owners of the game.
Located in Bellevue, Washington, the engineering team at ArenaNet is responsible for building out the core gameplay systems as well as ensuring players can access the game anytime. They strive for 100% uptime which they achieved for three and a half years starting in August 2017 until March 2020 when they had a 14-minute outage due to a configuration error.
As a Massively Multiplayer Online Role-Playing Game (MMORPG) it is important to the engineering staff to keep the game available to players at all times. This was originally accomplished for both games in the franchise using internal datacenters (IDC) located in Dallas and Frankfurt. “Our hardware was purchased in 2012 and was starting to fail more often – especially the disk drives – and fixing this was simply not the best use of our time. And even without failures, we were somewhat stuck where we were – if we wanted to experiment with faster hardware, or more RAM, or faster drives, or better networking we had to order and provision it ourselves which is a huge impediment to experimentation,” said Dr. Stephen Clarke-Willson, VP Technology. The team had to consider the best way forward – a big capital expenditure for new hardware? Move to leased hardware? Or go all-in with a cloud system like AWS?
“We had some familiarity with AWS as we already used Amazon Route 53 and Amazon CloudFront services. The big question for us was whether AWS could host these multiplayer maps for extended periods of time,” Clarke-Willson explained.“Our biggest concern with moving to AWS was the length of time any one server instance would remain up. We felt AWS was originally designed for highly transactional web-style content: if one machine goes offline, the load balancer routes around it to a different instance. The same is true for games with shorter play sessions. But for MMORPG’s like Guild Wars and Guild Wars 2 we needed a higher level of reliability because map instances can stay running for weeks at a time. A crash in a map server is very disruptive to players,” Clarke-Willson shared.
Game launches battle-tested on AWS
To test the resilience of AWS servers for its persistent maps, in September 2017 ArenaNet began a hybrid deployment of the game. “And since we are crazy, we did it at the same time we happened to be releasing our second expansion, Path of Fire,” Clarke-Willson explained, “We were able to scale up quickly for the influx of players in-game (double our normal active users) and then stabilize those maps in AWS.”
After the successful live deployment, ArenaNet began migrating its remaining microservices to AWS. “The Guild Wars 2: Path of Fire rollout worked so well we started moving all other services from our IDC’s to AWS. We did it without any downtime, including the process of mirroring our IDC databases into the cloud. “Our operating system and database licensing costs went down when we moved to AWS. Bandwidth also reduced as we were able to pay by the gigabyte without concern for ‘95p’ ISP pricing.”
Today, Guild Wars 2 reliability has increased since migrating to AWS. The development teams have launched two expansions, five Living World story arcs, and updated the game every two weeks without any downtime or player interruption for many years. In addition to using Amazon Route 53 and Amazon CloudFront, ArenaNet is also utilizes AWS CloudFormation for deployment, Amazon CloudWatch for monitoring, and a diverse variety of Amazon EC2 instance types including T3, C5, M5 and R5, of all different sizes. “Over the first year we used the AWS monitoring and billing systems to aggressively analyse our deployment and ended up cutting our operational spend in half,” says Clarke-Willson. “Overall, we can provide better experiences to players on hardware that we no longer have to actively manage. In many cases, when an instance stops in AWS, it restarts on new hardware with the exact same configuration usually within 10 or 20 minutes, long before we would even have gotten a call about the problem in the IDC. So everyone gets better sleep as a result.”
Key to deciding to use AWS was Amazon’s commitment to customer support. “I think we were the first Windows-based ‘all-in’ MMO deployment in AWS, and moving the game into the cloud without downtime was made possible by the amazing customer service provided by AWS,” Clarke-Willson said. “I’m proud of the way the ArenaNet engineering team and the AWS support teams helped us achieve this successful transition.”