Capillary Modernizes AWS Architecture to Scale 15x While Controlling Costs

Scaling a successful business is equally a reward and a challenge. It’s an ongoing exercise that often happens in phases as companies hit new milestones in terms of customers and scope of operations. For Capillary Technologies, its biggest scaling challenge came after 11 years, when the company acquired new Fortune 500 clients such as Walmart and Yum! Brands. Capillary Technologies delivers a loyalty program management and customer engagement software-as-a-service platform for retail marketing.

Capillary solutions power more than 100 loyalty programs across the globe, and the company continues to onboard increasingly larger enterprises every year. In 2019, Capillary onboarded a multinational corporation with daily transaction volumes that were 10–15 times higher than any of Capillary’s existing customers. This sparked conversations about maintaining performance and profit margins at scale, and improving elasticity to support such high-volume customers.

As a digital business born on the Amazon Web Services (AWS) Cloud, Capillary approached the AWS team to discuss how to boost scalability and elasticity while optimizing costs. This led to an intense workshop, during which Capillary and AWS teams spent 5 hours a day for more than a week doing a deep dive into each component of Capillary’s cloud architecture to identify areas for improvement. By late 2020, the company had notched 30 percent savings on its monthly infrastructure bill and modernized its architecture to improve scalability in an automated fashion.

One of the low-hanging fruits that Capillary immediately addressed was switching to a different storage volume type on Amazon Elastic Block Store (Amazon EBS). Many of the company’s workloads were attached to higher throughput storage volumes on Amazon EBS, but the teams found redundancies during the deep dive. Capillary was able to cut storage costs by about 40 percent by switching to more cost-effective volumes, while still maintaining the high-performance levels the business required.

To unlock further savings on the AWS Cloud, Capillary has taken up more managed services on AWS. “For several years, we had the notion that managed services are costlier and that we could do a better job, but the last year and a half has been eye-opening in that regard,” says Piyush Goel, vice president of Engineering at Capillary Technologies. “We are realizing the value of managed services on AWS, the scale they provide, and the amount of engineering effort that we save.”
 
Database administration, typically a major IT cost center for businesses, was the first target for conversion to a managed services model. The company stores and processes 200–250 TB of data, and database maintenance was a time-consuming task for its engineers. To address this, Capillary migrated from self-managed MySQL database clusters—which it has used since its founding—to Amazon Aurora. “With Amazon Aurora, we’re spending just one-third of the time we spent on our previous databases, and we no longer have to worry about recovery and failover, which are completely automated,” says Goel. Engineers now spend their time writing code and looking at new ways to automate performance monitoring as their infrastructure scales.

A fully containerized backend has also helped Capillary reduce both the cost and human intervention required in scaling. The company uses Amazon Elastic Kubernetes Service (Amazon EKS) as a managed service to provision and run “dockerized” applications. As a result, deployment time for delivering new features has decreased to 20 minutes from 50–60 minutes in its previous setup.

In addition, Capillary’s customers no longer experience brief periods of downtime associated with deployment. In most cases, Capillary is exceeding its customer service level agreement of 99.5 percent uptime, and it now achieves 99.9 percent availability thanks to containerization on AWS. “This improvement in customer satisfaction gives the sales team the confidence to target multinational clients,” Goel says.

On the backend, the implementation of containers has empowered immutable deployments. This ensures consistency and reliability in software delivery, resulting in higher-quality releases. In the past, Capillary’s engineers could only roll out 1–2 releases every week, but now they have the capability to deploy multiple releases every day. This accelerates time-to-market for customer-facing app enhancements.  

The company is on a journey to move completely to a continuous integration/continuous deployment (CI/CD) model, which Goel says has brought a lot more engineering discipline to his teams. “Because containers can scale up and be deployed in a rolling pattern, this shift is an organic first step to a fully developed CI/CD approach.”

As part of its analytics offering to customers, Capillary has built a 10-stage data processing pipeline using Amazon Elastic Inference. The company streams video and audio data from retail sites and analyzes the data via machine-learning (ML) models to yield insights on consumers. ML models on the TensorFlow platform have been optimized to run on Amazon Elastic Inference.

With this service, Capillary benefits from an always-on model for inferencing that automatically scales up and down according to demand. This reduces the cost of inferencing by avoiding overprovisioning of memory-heavy GPU compute resources. “We can host multiple ML models on the same Amazon Elastic Inference unit, and we optimized the cost even further with multiple units on the same host machine, which keeps CPU utilization low,” Goel says.

With the savings realized on AWS, Capillary is reinvesting in its business, mainly in research and development for new products. Examples include a real-time notification platform that streams customer data into Capillary clients’ data lakes and integration with social platforms such as LINE.

Looking back, Capillary recognizes the value of the collaborative approach to problem-solving it has developed with AWS. “AWS solutions architects, sales and support teams have worked closely with us to understand our app architecture in great detail,” says Goel. “They keep us updated on how we can leverage particular components of our existing architecture, and consistently provide personalized and relevant suggestions.”