Katalon Delivers 70%+ Savings with Serverless Streaming Analytics Using AWS Lambda

In 2016, Katalon launched its AI-powered, comprehensive quality management platform to help customers alleviate bottlenecks in the software development lifecycle by seamlessly integrating automation into the testing process for web, API, desktop, and mobile applications. Katalon Platform incorporates AI to optimize test creation, planning, organization, runtime analysis, and maintenance.

Initially, Katalon built its applications using open-source Apache Kafka to capture near-real-time streaming data. Katalon engineers had to set up and manage workflows for a growing number of customers who generated workloads with large fluctuations in volume depending on the time of day. “At first, we didn’t have a messaging system, and using a database to store all the events caused a crunch in processing,” says Vu Bui, application architect at Katalon. “We figured out that this is not the way to scale.”

In 2019, Katalon turned to Amazon Managed Streaming for Apache Kafka (Amazon MSK), which companies can use to ingest and process streaming data in near real time with fully managed Apache Kafka. For example, companies that run automation test cases using Katalon Studio, the company’s desktop application, produce extensive results that upload automatically to the Katalon Platform for processing and analysis, displaying results to customers in a dashboard. Behind the scenes, engineers can create, update, and delete Amazon MSK clusters with a few clicks while managing a separate cluster of processing power using AWS Elastic Beanstalk, a service that companies can use to deploy web applications. “We were growing very fast,” says Tran. “We wanted to continue to optimize our infrastructure so that it could grow with us. We always have to prioritize the ability to scale in order to provide our solution in a reliable manner to our customers.”

Katalon embarked on a project to further modernize its infrastructure by using microservices to handle the event messaging workflow, offload management of compute clusters, and optimize the use of automation. To seamlessly accommodate customers’ fluctuating workloads, Katalon turned to AWS Lambda, a serverless, event-driven compute service that lets organizations run code for virtually any type of application or backend service without provisioning or managing servers. More than 30,000 software, DevOps, and quality teams test and launch their software faster using Katalon’s solutions. “The best benefit is the scalability,” says Bui. “Using AWS Lambda, we don’t need to manage any physical node to provide the processing power. It handles the scaling.” Because customers test more efficiently, they save significant costs. “Some of our customers have said that they cut 40–70 percent of their testing costs after switching to Katalon solutions,” says Tran.

Katalon has also improved the productivity of its own teams. “We develop new products and features much faster than we did before,” says Tran. “Using AWS, we can focus on understanding our customers and addressing their needs. And we don’t have to worry about the technical feasibility of that.”

The company continues to focus on optimizing its serverless infrastructure. It has been working to remove or break down messages that exceed the 15-minute processing limit of AWS Lambda functions, with the idea of phasing out the use of AWS Elastic Beanstalk. The company also plans to upgrade to Java 11 so that it can take advantage of AWS Lambda SnapStart for Java, which can improve startup performance for latency-sensitive applications by up to 10 times at no extra cost, typically with no changes to function code. And new services launch using Amazon Elastic Kubernetes Service (Amazon EKS), a managed Kubernetes service on AWS, as Katalon plans to migrate the existing compute infrastructure to Amazon EKS as its next modernization focus.