Optimizing Costs and Time to Market for CI/CD Using Amazon EC2 Spot Instances with ClickHouse

Released as an open-source project in 2016, ClickHouse started as a database solution launched for analytical workloads. Its column-oriented system gives users the ability to aggregate and query massive volumes of data in milliseconds, delivering results 100–1,000 times faster than traditional databases. “Whenever you want to collect data—like events, metrics, logs, or user behavior—and analyze it in near real time, ClickHouse is an optimal solution,” says Alexey Milovidov, cofounder and chief technology officer at ClickHouse Inc.

Developers can use ClickHouse Inc.’s open-source solution for free, but they must build their own infrastructure to support near-real-time processing of analytics. As ClickHouse became more popular, the company recognized the demand for a fully managed alternative. “Time to market is very important for developers and new startups,” says Tanya Bragin, vice president of product at ClickHouse Inc. “They want to start consuming a service that offers them the speed and efficiency of ClickHouse without having to build a backend themselves.” This demand led the company to conceptualize ClickHouse Cloud.

The team set a 1-year deadline to develop this new product. “It was clear that our first place to go was AWS in terms of where we would host our offering,” says Bragin. “We didn’t want to build anything from scratch. We had to use off-the-shelf services from AWS to accomplish our goal within 1 year.” To identify which AWS services to use, ClickHouse Inc. engaged AWS Enterprise Support, which provides enterprises with concierge-like service for achieving their outcomes and finding success in the cloud.

ClickHouse Inc. collaborated alongside AWS Enterprise Support to deliver a high-performing, fully managed product to help its customers accelerate their time to market. ClickHouse Inc. decided to run all the infrastructure for its SaaS offering using Amazon Elastic Kubernetes Service (Amazon EKS), a managed Kubernetes service to run Kubernetes in the AWS Cloud and on-premises data centers. To maintain high availability on its SaaS offering, the company’s developers configured 20 clusters that run across seven AWS Regions using Amazon EKS. The team also containerized the workloads on its continuous integration and continuous delivery (CI/CD) pipeline, freeing its developers from infrastructure maintenance. “By using Amazon EKS, we were able to shave the timeline for this project by a quarter,” says Roopa Tangirala, senior director at ClickHouse Inc. “Otherwise, we would’ve had to hire more engineers.”

One of the company’s goals while developing ClickHouse Cloud was to offer a free private preview to 50 users. To do so, ClickHouse Inc. needed to optimize and track its cloud costs. “We spent a lot of time thinking about how we would charge our customers for this new offering,” says Bragin. “Having deeper conversations with the AWS team about how other companies operate within our space gave us the comfort that we were getting the best advice that we could find.”

To offer ClickHouse Cloud at a competitive price point, the company reduced the cost of running its SaaS offering on Amazon EKS by signing up for Savings Plans, a flexible pricing model that can save up to 72 percent compared to On-Demand prices. “We were expecting pretty significant growth, especially in terms of compute and storage usage,” says Bragin. “Forecasting our capacity on the compute side was critical in managing our spend.”

The company’s team of developers agreed on monthly deadlines for delivering product features. ClickHouse Inc. knew that its CI/CD pipeline needed to scale accordingly while staying within budget. So the AWS Enterprise Support team recommended that ClickHouse Inc. run its pipeline using Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances, giving its team the ability to take advantage of unused Amazon EC2 capacity to run fault-tolerant workloads at up to 90 percent off compared to On-Demand prices.

ClickHouse Inc. also chose Spot Instances because they are designed for various stateless, fault-tolerant, and flexible applications, such as big data, containerized workloads, web servers, CI/CD applications, high-performance computing, and test and development workloads. ClickHouse Inc. combined its use of Spot Instances with AWS Graviton processors, which are designed for optimal price performance for cloud workloads running in Amazon EC2. Every day, ClickHouse Inc. performs up to 12 million functional tests on the company’s CI/CD pipeline, which helps its teams review and merge more than 800 contributions to its code base each month.