Skai Improves Throughput by 250% Using Amazon Keyspaces

As a mature business, Skai’s architecture is composed of several layers that evolved over time with a large monolith at its core supporting requests from over 150 microservices and dozens of serverless AWS Lambda functions. This monolithic cluster had been running on an on-premises architecture for over a decade, Maintenance of the on-premises cluster had become an operational burden, requiring many engineering hours monthly from the team and distracting Skai’s infrastructure engineering teams from working on other more critical parts of their business. Skai made a strategic decision to move to AWS to gain agility, performance, reliability, improved total cost of ownership, and the many benefits that are offered by AWS managed services.

Skai’s suite of products relied on an 80 node, self-managed Apache Cassandra cluster as its operational database for ingestion and processing of tens of millions of ad attribution events per day. This processing was a critical part of the system, and any downtime resulted in direct ad revenue loss to Skai’s customers and a material financial risk to Skai. Considering that client uptime expectation for the system was 99.99 percent, Skai needed to find a highly-reliable solution for that workload.

As the decision to migrate was being made, Skai was looking for a maintenance free, fully managed, and highly available Cassandra-compatible solution.

In late 2022, Skai started modernizing their Cassandra workload by gradually migrating to Amazon Keyspaces and completed the migration by the end of 2023. The migration was staged by data size and by customers.

Because the system is highly critical, Skai chose to safeguard the migration by creating a dual-write data validation while implementing a shadow logger pattern. A logger, based on an Amazon Elastic Kubernetes Service (Amazon EKS) microservice, was developed to duplicate the data across the on-premises cluster and Keyspaces, while a data validation process was developed running on a Lambda function to constantly sample and compare the data on both tables. Comparison metadata and results were stored in an Amazon DynamoDB table for later analysis.

As recommended for most migrations, Skai started with a short proof of concept and a few months of testing, during which the team made application adjustments to prepare Skai’s backend processes for working with a managed, serverless database. This process included adding retry mechanisms, adjusting parallelism settings, and stress testing in a production-like environment to make sure their application was able to cope with request throttling limits. Another element Skai needed to consider was scale and “hot spots” - areas of data that pass the allowed number of requests per partition. This required ensuring that Skai’s data is well distributed between partitions and involved a “warm-up” of selected Keyspaces tables before the migration to validate sufficient physical indexes had been created on Keyspaces to be able to serve the initial load post the migration.

By the time Skai successfully tested the Keyspaces setup, the shadow logger has been running for about 3 months. This generated a sufficient amount of history for the Skai team to be ready to switch over from on-premises as primary to Keyspaces as the primary source of truth for its monolithic service.

In addition, the Skai infrastructure engineering team took the opportunity to transform provisioning of Amazon Keyspaces to a GitOps based, Infrastructure as Code process, using Pulumi. Amazon Keyspaces tables, schemas, and scaling operations were now decoupled from the application and managed in a declarative manner from Skai’s source code management leading to significant operational benefits and improved reliability.

In 2023, Skai completely migrated to Amazon Keyspaces. With the adoption of Keyspaces for ad attribution and performance events, Skai experienced a 250 percent improvement in event processing throughput and increased general system reliability, going from 1-2 major infrastructure related outages per year to zero in the past year, due to the nature of a fully managed AWS service. Furthermore, the Skai infrastructure engineering team reduced its operational burden by 30 percent and was available to support other critical areas of the Skai suite of products.

Danny Zalkind, senior director of Infrastructure Engineering shared, “Skai was able to greatly reduce engineering overhead around management and costs of managing our Apache Cassandra infrastructure. We freed up 30 percent of our data infrastructure team capacity and eliminated the need to invest 3 weeks of developer time yearly to prepare for the holiday peaks around end of the year due to Amazon Keyspaces advanced auto-scaling capabilities. Migrating to Keyspaces has demonstrated how powerful AWS cloud is. The ability to scale out a workload of tens of millions of events in several minutes, compared to a year-long server procurement process on our on-premises datacenter was a game changer for Skai’s engineering team, allowing us to deal with any demand spikes and letting us sleep better at night, as we know our system’s uptime will be at least 99.99 percent per our Service Level Agreements."