Bigabid Slashes Latency and Boosts Query Performance 400x Using AWS and Firebolt

Israel-based Bigabid is a digital advertising technology company that uses big data and machine learning (ML) to help application developers increase the take-up and usage of their apps while optimizing advertising spend. Its infrastructure was built using massive SQL databases running on multiple Amazon Web Services (AWS), but they weren’t able to query data fast enough to produce the required results. In 2022, Bigabid started working with AWS Partner Firebolt to improve performance and consolidate its internal business intelligence and analytics platforms. Bigabid has now improved its search performance 400 times. It now only takes a few seconds to deliver analytics results that would previously have taken days or weeks to calculate.

Digital advertising technology company Bigabid uses big data and machine learning (ML) to drive app growth for developers. Founded in Israel in 2016, Bigabid’s platform processes vast amounts of data and connects with multiple ad suppliers and ad exchanges in near real-time to provide clients with insights into how their apps are performing. This allows Bigabid’s clients to target highly specific audiences, increasing app usage, and optimizing their advertising spend.

Bigabid’s business intelligence (BI) platform analyzes how a client’s app is being used, measuring everything, including impressions, ad clicks, app installs, and in-app purchases. It also has a separate internal data analysis platform that is used by its developers and campaign managers to continuously optimize performance. Bigabid’s AdTech platform was built using AWS. It uses Amazon Simple Storage Solution (Amazon S3) object storage, built to retrieve any amount of data from anywhere, for its data lakes, and Amazon Elastic Compute Cloud (Amazon EC2), which provides secure and resizable compute capacity for virtually any workload.

Yaron Cohen-Leo, BI team lead at Bigabid, states that they chose AWS because of its reputation for reliability and its diverse range of managed services, explaining “We use many AWS tools for various cases. Having such a robust feature set helps simplify our data efforts across the company.”

Bigabid’s analytical databases were originally based on MySQL and were falling short of the performance the company required. It was taking days to generate data insights and the company struggled to view data older than three months due to process-heavy data aggregations, that made it impossible to compare results seasonally, or year to year. The company wanted to do more. It wanted to be able to analyze data for a million ad auctions every second and manipulate data lakes containing hundreds of terabytes of data in near real-time while accessing tables with billions of rows to create hundreds of live dashboards.

To do this, Bigabid undertook the ambitious project of building a high-performance big data infrastructure. This would require it to find a high-performance database and merge its internal BI and data analysis platforms into a central data platform. Bigabid evaluated several high-performance database options and was impressed with AWS Partner Firebolt. “Firebolt is a massive improvement to our BI efforts,” says Cohen-Leo. “Using the same test dataset of 100 million records, other databases took minutes, Firebolt analyzed in seconds.”

In August 2022, Bigabid chose to adopt Firebolt’s analytics using its existing Amazon S3 data lake and merged its BI and analytics systems. The results were so impressive that by the start of 2023 the company had completed its migration project and was optimizing its systems and building new dashboards. “We now rely on Firebolt and AWS," says Cohen-Leo. “We don’t have to manage anything—everything is in one place. We can query a database containing 30 billion records and receive results in a second.”

A year on from starting the project, Bigabid has significantly optimized its data warehouse resources using AWS and Firebolt. Using Firebolt, the query response times for searching a 31 TB table have improved 400 times over the previous system. Firebolt uses its own compression system, which has reduced the amount of storage needed for the same table from 31 TB to 7 TB. “The Firebolt compression means we need less storage, which also reduces our costs,” says Cohen-Leo.

The migration has also solved Bigabid’s challenge of analyzing older data. Now, the data is delivered to dashboards in near real-time, and there is no longer a limit on how far back the data can be queried. “Now we can easily analyze data for seasonal and year-on-year changes, which produces more valuable business insights,” says Cohen-Leo. “We have a single BI dashboard that reveals the real-time state of our business at a glance—and it can be customized in less than a second.”

Now, a single table serves both the analytics and BI roles and is updated in near-real time. From this master dashboard, the company has created many dashboards to provide more granular information to support different clients and provide insights into other parts of the business. Looking to the future, Bigabid expects to be able to squeeze even more power out of the Firebolt and AWS infrastructure. “As we grow rapidly, our requirements increase, so we need to continue optimizing the platform and gain a better understanding of the increasing volume of data,” says Cohen-Leo.