Scaling Adobe Experience Platform on AWS with cell-based architecture

Adobe offers document, image, audio, and video solutions in addition to tools for enhancing consumer experiences. Supporting over 12,000 customers, AEP along with its cloud infrastructure has grown rapidly since 2019. Adobe wanted AEP to scale in line with future customer growth, so it needed a scaling strategy to meet system reliability goals and continue its expansion. “We wanted a repeatable, testable way of scaling as we added more customers or expanded to other geographic regions,” says Dave Beus, principal cloud architect at Adobe. Given the limited number of IP addresses on IPv4—which AEP uses for compatibility with technologies in the software stack—the design had to scale within the quotas of the available private IP address space.

After discussions and engineering deep dives between AEP and the AWS Solutions Architecture team in August 2023, the teams began architecting AEP on AWS to align with cell-based architecture principles. This design would help meet AEP’s networking, scaling, reliability, and testing needs. Over the next 6 months, the teams designed, built, staged, and validated AEP’s stack in a staging environment on AWS. Thus, they met the infrastructure needs and onboarded more than 50 engineering teams and several hundred AEP services on AWS. 

Development began in January 2024 on the initial cell and automation APIs to build the cell infrastructure, deploy AWS services, and onboard AEP services. Adobe completed a preproduction environment by May and deployed the first production environment on AWS by July. In 1 year, Adobe completed the project, making all services and backend data applications available on AWS.

Cell-based architecture is akin to the bulkhead of a ship, which is designed so that internal walls divide the ship interior into watertight compartments to limit damage in case of flooding. Working alongside the AWS Solutions Architecture team and AWS Professional Services—which helps companies realize their desired business outcomes when using AWS—the AEP team created self-contained infrastructure cells. Each cell was independent and consisted of the entire stack of AEP services.

To meet network capacity and IPv4 addressing challenges, AEP architected a cell-based network on AWS. Cell isolation helped reuse the same IPv4 address space in each cell, conserving private IP address space in Adobe’s enterprise-level backbone network and providing a uniform deployment environment for AEP services across each cell. 

The infrastructure within each cell consists of multiple Amazon Virtual Private Clouds (Amazon VPCs) to define and launch AWS resources in logically isolated virtual networks. To administer and manage the intracell network, AEP created a new cell networking service team. The team administers shared VPCs within each cell and shares these with other AEP service teams for deploying their infrastructure. This infrastructure includes Amazon Elastic Kubernetes Service (Amazon EKS), which businesses use to start, run, and scale Kubernetes without thinking about cluster management; Amazon Elastic Compute Cloud (Amazon EC2), which provides secure and resizable compute capacity for virtually any workload; Amazon Aurora databases, which provide high performance and availability at global scale; and Amazon ElastiCache clusters, which provide near real-time (NRT) performance for NRT applications. VPC peering provides high throughput and low-latency connectivity between the Kubernetes clusters in each cell and the shared VPCs. AWS Transit Gateway—which connects Amazon VPCs, AWS accounts, and on-premises networks to a single gateway—is deployed within each cell to route traffic between the shared VPCs and the Kubernetes clusters using a low- or medium–throughput connection. AEP also uses AWS PrivateLink, which lets companies establish connectivity between VPCs and AWS services, to help maintain high-throughput networking and security as the engineering teams use those services (see architecture diagram).

Cells provide repeatable patterns of capacity units, and horizontal scaling simply involves adding more cells to expand capacity in a manageable way. That reduces the cognitive burden of deployment and scaling.

This architecture helps combine scaling up with scaling out to avoid reaching capacity quotas. “We can reduce the scope of growing inside a cloud provider region to a fixed upper bound of infrastructure deployment, test it, and replicate it, as opposed to growing organically to the limit where our software fails or the cloud provider can’t meet our needs,” says Trent Davies, senior principal architect at Adobe. The architecture also helps automate new cell deployments for scaling and scope failures to each infrastructure cell and a subset of AEP customers. It provides well-tested and monitored cell-scaling boundaries. It supports staggered code deployments across cells, cell-scoped disaster recovery mechanisms, and cell-level observability and troubleshooting boundaries.

AEP is generally available on AWS in the United States, and the team is also working on enhancing observability for service quotas, resource usage, security, and scaling boundaries. AEP uses Amazon CloudWatch to observe and monitor resources and applications, ingesting the metrics to improve visibility.

Because cell-based architectures are replicable and simpler to manage and scale, Adobe is considering them for other Adobe products on AWS. “The architecture we’ve worked on alongside AWS helps us scale virtually infinitely within a cloud provider region,” says Davies. “We plan to replicate this design pattern across AEP deployments and other Adobe products deployed on AWS.”