Keboola Saves 40% on Compute and Data Processing Costs after Partnering with pipetail to Migrate to Amazon EKS

Keboola provides a single data stack for customers to integrate multiple data sources onto one easy-to-use platform. It provides GDPR-compliant services for banks and auditors. Keboola relies on managed services from Amazon Web Services (AWS) to run its Kubernetes infrastructure so it can support customers around the world without having to think about maintaining or scaling systems. Keboola turned to AWS Partner pipetail to help it migrate on to a new scalable, automated system using Amazon Elastic Kubernetes Service (Amazon EKS).

Data operations company Keboola wanted to make it easy to create single-tenancy setups for customers with sensitive and security-conscious workloads. It also needed a cloud provider with 24-hour a day support so that it could continue its international expansion into the US. 

Keboola was looking for a platform that would both work across the major cloud providers and support customers that want to run its systems on their own hardware. Using a single system that runs on managed services makes managing and provisioning easier and aligns with Keboola’s philosophy to buy rather than build things that are not core to its business. 

Keboola chose AWS because it was the only cloud provider with the services that it needed at the time. In addition, AWS Regions meant that compliance with different data protection and privacy regulations was easier for its existing and future customers wherever they were based.

Keboola launched in 2011 and ran its systems on Amazon Elastic Container Service (Amazon ECS), which provides a fully managed container orchestration service that makes it easy to deploy, manage, and scale applications. In late 2020, Keboola decided to shift to Amazon Elastic Kubernetes Service (Amazon EKS) to better support both multi-cloud setups and customers that still wanted to use on-premises hardware.

To support this critical migration, Keboola turned to AWS Partner pipetail to help it get the most value from automation and improved scalability. “We wanted to work with pipetail because we needed expertise in shifting workloads from Amazon ECS to Amazon EKS without adding management overheads—we knew we had to have some help to automate as much as possible,” says Vojta Tůma, chief information officer at Keboola. “We just fell in love with pipetail from the very first meeting. There’s a great cultural fit and they get work done in good time and for a reasonable price.”

Using its Kubernetes expertise, pipetail helped to design the migration and then guided Keboola through the stages. The two companies worked in step, making iterative changes every week rather than attempting to migrate all at once. The teams held regular workshops and design meetings as the project evolved. 

The move to Amazon EKS has also resulted in faster customer integration and cost savings. Keboola can now get a straightforward customer deployment up and running in hours rather than days. “Everything is managed by Terraform, so deployment is much more automated. We can deploy the whole solution in 3 hours rather than 14 days—so we can scale massively as a business. It has changed how we can work as a business,” says Tůma. Terraform is an open-source infrastructure as a code software tool that lets users safely and predictably create, change, and improve infrastructure.

Previously, Keboola was using a stable pool of Amazon Elastic Compute Cloud (Amazon EC2) instances, which meant that systems were over-provisioned during quiet periods. Now it can autoscale up and down, providing a 40 percent saving on compute and data processing costs. Keboola and pipetail have also automated almost all the manual steps previously required to scale and maintain AWS workloads. The consolidated stack has reduced complexity and workloads for staff and speeded up deployments.

Keboola saw annual growth rates of 30 percent during the first 2 years of the COVID-19 pandemic, but it expects to return to 100 percent annual growth now that business is returning to normal. Using AWS, and automated deployment built by pipetail, Keboola can deal with the rapid increase in demand for new stacks.

Using AWS has also helped Keboola and its clients to weather the changes that the COVID-19 pandemic brought to its business. For example, AWS scalability meant that Keboola’s restaurant customers could scale down to zero overnight when compulsory closures were announced. But online groceries and courier companies needed to rapidly increase system capacity as demand for their services grew exponentially. This was all handled by automated tooling running on AWS to analyze, report, and optimize costs.

The company uses Snowflake Data Cloud on AWS—a highly-scalable data cloud that can securely integrate multiple silos of data for analysis and machine learning. This makes it easier for its customers to build and activate advanced modeling to get more value from their data while only paying for the storage and compute that they use.

Keboola is now focused on growing its business in the US and Canada. It is concentrating on enterprise customers that typically require single-tenancy solutions, which Keboola can now deploy far faster thanks to its work with pipetail.