Modernizing Infrastructure Using Karpenter and Amazon EKS with PicPay

Founded in 2012, PicPay offers financial service applications and transactions—such as payments, money transfer, a digital wallet, investments, and insurance—to consumers and sellers in Brazil. In 2022, PicPay began to modernize and optimize its architecture to support its rapidly increasing customer base.

Before, the company had a single Kubernetes cluster of 900 nodes where its deployment involved significant manual work to create pods and resources. To manage its Kubernetes clusters, PicPay chose to implement Amazon EKS and then apply Karpenter, which works on the application level, to auto-scale Amazon EKS. Using this solution, PicPay transformed its architecture to 121 clusters.

PicPay began its modernization journey in August 2022, implemented Karpenter within 6 months—in early 2023—and continued to take advantage of its new features as they became available. “Because our solution was born on AWS and we planned the cluster modernization around Karpenter, we had no problems implementing the solution to evolve our architecture,” says Daniel Vieira Carneiro, principal IT infrastructure engineer at PicPay. Because the company used Karpenter to auto-scale Amazon EKS, PicPay developers could focus on increasing productivity and reducing costs. PicPay also used features such as infrastructure as code to deploy applications and enhance its microservices.

To facilitate the increasing scale of these clusters, PicPay needed to modernize its infrastructure by removing idle resources. This meant rearchitecting its infrastructure to meet sudden scaling needs, such as on paydays, when more resources are needed due to significantly higher traffic.

After modernizing its architecture, PicPay achieved significant benefits. Kubernetes releases a new version every 3 or 4 months, so keeping up to date is an important process. The company reduced cluster upgrade time from 3 weeks to 2 hours using Karpenter, a significant improvement in operational efficiency. PicPay also reduced cluster support tickets by over 90 percent. In 2021, the company had 30 million users, which grew to over 60 million by 2024 with 35 million active users, growth the company was able to seamlessly scale up to using its modernized architecture. In addition, PicPay improved its availability for services and applications.

Along with this monumental customer growth, PicPay sought to optimize compute by using Karpenter to optimize Amazon EC2 Spot Instances (Spot Instances), which are built to run fault-tolerant workloads for up to 90 percent off. PicPay uses Spot Instances for 100 percent of its nonproduction workloads and 40 percent of its production workloads. The company uses spot-to-spot consolidation to further optimize and rightsize Spot Instances. Using Karpenter, it implements best practices for using Spot Instances. For example, when replacing and rightsizing a Spot Instance due to consolidation, PicPay can choose from a minimum of 15 instance types in its Karpenter environment, helping make sure that the operation does not affect the frequency of termination. “We're using the latest version of Karpenter with spot-to-spot consolidation,” says Carneiro. “It's a very important feature that helps us select the right instances and shorten any disruption time while keeping infrastructure costs low.”

Overall, PicPay has reduced its monthly costs by over 50 percent, well exceeding its initial estimates. To supplement these cost savings, the company has also begun to use AWS Graviton-based Amazon EC2 Instances (AWS Graviton-based Instances)—which provide the best price performance for cloud workloads running on Amazon EC2—for around 10 percent of its critical infrastructure nodes. PicPay is saving yet further costs by using Karpenter to run multiple CPU architectures in one configuration.

Modernization is an ongoing process, and while PicPay has achieved significant improvements already, the company is pushing for additional optimization. For example, it intends to increase its use of Spot Instances from 40 percent to 70 percent and increase its use of AWS Graviton-Based Instances from 10 percent to 50 percent by the end of 2024. PicPay will continue to optimize using AWS as its customer base scales up, continuing to modernize its infrastructure along the way to stay at the forefront of innovation in the financial services industry.

“In our next big projects using AWS, we will evolve in our use of Amazon EKS,” says Carneiro. “We will be using Karpenter for the infrastructure in all these projects.”