Liquid Analytics Delivers Near Real-Time Analytics Powered by Amazon EKS and DuckDB

Liquid Decisions helps people make sales and finance decisions based on their own data. “We want to develop a system that empowers our business users to build their own data-aware solutions,” says Vish Canaran, chief AI officer for Liquid Analytics. “It’s their data, so it should be their decisions and their model.”

The company uses AWS to modernize and transform HPC workloads for its customers. The core of its solution is built using generative AI, Kubernetes, and DuckDB—an open source relational database management system—to run analytics on large datasets at a fast pace and low cost. Liquid Analytics designed containerized, Kubernetes-based architecture around DuckDB on AWS.

To harness the capabilities of DuckDB quickly and cost effectively, Liquid Analytics modernized its architecture using Amazon EKS and Amazon FSx for Lustre, which provides fast storage performance for GPU instances in the cloud. The company chose AWS services after thorough testing, which included taking part in Experience-Based Acceleration (EBA)—a program in which companies use hands-on, agile, and immersive engagements to speed up digital transformation and cloud value realization. “AWS offers us a large number of tools, which means we can get our product to market rapidly with more customers in the sales and finance space,” says Canaran.

Since implementing Amazon EKS and FSx for Lustre, Liquid Analytics has been able to quickly scale, spinning up 1,000 pods in 1.2 seconds, a five times performance gain. Scaling horizontally at that level previously would have taken 2 hours using another provider with a traditional transactional SQL solution. Liquid Decisions is a highly distributed system with data lineage, so every time a job runs on a DuckDB dataset, the product creates a new version of the dataset in a new Kubernetes pod. Data stays native in DuckDB, and using FSx for Lustre—which provides subsecond latency—the company can bring jobs directly into the Kubernetes pod for compute without creating copies. This approach removes the overhead of copying DuckDB files, enhances performance, and reduces latency.

Because Kubernetes is a critical component of its infrastructure, Liquid Analytics also uses Karpenter—an open source automatic scaler for Kubernetes—to provision and scale clusters and achieve granular control over Amazon EKS. This facilitates near real-time scaling and cost management, which are crucial for rapid decision-making.

Liquid Analytics chose Amazon EKS in part because it gives the company control over the instance types it uses. By selecting the instance types that fit the job, the company reduced CPU waste by 90 percent. Liquid Analytics also saves 63 percent of costs by using Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances, which run fault-tolerant workloads for up to 90 percent off.

To handle transactional data and metadata, Liquid Analytics uses Amazon Aurora, which provides high performance and availability at global scale for PostgreSQL. “Aurora for PostgreSQL just works, and the AWS team helped us optimize how we use it,” says Canaran. Liquid Decisions also uses additional AWS services for Extract, Transform, and Load (ETL) processes and data streaming.

Liquid Decisions provides a low-code environment—using tools such as spreadsheets, chatbots, documents, wizards, and forms—that customers can use to customize their own analytics applications and make decisions. The solution helped customers across sales and finance domains process more than 1 million goals within 2-hour service-level agreements (SLAs).

Customers use the solution to avoid multimillion-dollar development projects by helping business users directly set up their configurations with less IT effort. For example, a distribution customer with 1,000 agents was looking to make 90,000 sales decisions a month. By adopting Liquid Decisions, the customer was up and running in 3 months instead of investing in another solution over 3 years, saving 3 million dollars per capability and reducing deployment time by 2 years.

Customers also benefit from the generative AI features of the solution, which are powered by Amazon Bedrock, a fully managed service that offers a choice of high-performing foundation models from leading AI companies. Liquid Decisions helps customers in the home-furnishing industry, for instance, aggregate product information that they historically stored in multiple documents and read manually. Now, sales representatives can use the consolidated pricing, availability, and product attribute information to create customized quotes in 15 minutes instead of over 1 day. As a result, they can reduce sales costs by 50 percent, increase revenue by 4 times, and accelerate response time by 10 times.

As it evolves, Liquid Analytics is designing additional generative AI–powered features on AWS, which provide comprehensive capabilities that companies can implement to innovate. “We are developing generative AI building blocks that our customers can use to create AI agents without the need to write SQL queries and Python code,” says Canaran. “I see Amazon Bedrock permeating our entire infrastructure so that people can talk to their data and make goal-based decisions.”