Bigabid Improves Its Modeling Accuracy on AWS by a Scale of 200x over One Year

Consistently growing the user base for a mobile application is a substantial business opportunity for app developers. The challenge, however, is reaching and acquiring quality users reliably and cost-effectively. “For most mobile app developers, social media platforms and user search engines seem to offer the only reliable route for quality user acquisition,” says Amit Attias, co-founder, chief technology officer, and vice president of research and development at Bigabid.

Bigabid’s mission is to bring performance-based advertisements to app developers. With performance-based ads, clients only pay when new users come to the application through the ad. What differentiates the Bigabid solution is how it takes advantage of machine learning (ML) for predictive decision-making and goes beyond social media and search engine data sourcing to create an in-depth customer user profile.

“We believe app developers need access to more relevant audiences outside of social media and search engines, in order to scale their businesses,” says Attias. “This is why at the heart of our technology is an innovative approach to user segmentation. Our approach requires us to rebuild all of the segmentation pipeline, from processing new unstructured data sources to building rich user profiles, which are used by our ML models to match the best ad for each user.” 

When it first launched, Bigabid batch-processed user data on Amazon Web Services (AWS) two to three times daily to update and refine its ML models and predictive engine for clients. As Bigabid continued to scale and develop ad campaigns, the company sought to move from batch processing to real-time stream processing on AWS.

“Our ML models are based on user behavior on the internet, understanding current session use and how one uses the current applications, how one reacts to advertisements, and so forth,” says Attias. “Our first models were based on batch updates completed a few times a day. We wanted the data to be much fresher and more updated in real-time. So, we began to look at building a real-time stream processing solution on AWS.”

Attias first evaluated using in-house resources to build a real-time stream processing system to run on AWS. The system would need to manage hundreds of thousands of recommendations per second and 100s of terabytes of data. Finding that building such a system would take six months to a year of engineering work with four engineers devoted to the task, the Bigabid team began to evaluate third-party solutions. Following an introduction at AWS re:Invent, Bigabid chose to move forward with Upsolver.  

Upsolver, an APN Advanced Technology Partner and Data & Analytics Competency Partner, focuses on helping companies that generate event streams quickly analyze data using Amazon Simple Storage Service (Amazon S3) through its easy-to-use, highly scalable platform and visual tool. Upsolver enables companies to drive value from both real-time and historical data by securely combining the data through its platform. It took only a few hours of discussion and experimentation for Bigabid to realize the value it could drive by using Upsolver on AWS.

“We met with Upsolver at re:Invent and we were very impressed with the team’s perspective on data and how it aligned with our views and needs. After a solution demonstration, we streamed data into the system to get a sense of what we could gain from the platform. Using our data, we quickly understood the potential benefits of Upsolver,” says Attias.

Using Upsolver’s stream processing platform, Bigabid built a working proof of concept for its real-time pipeline in hours. Today, Bigabid uses Upsolver extensively and continues to scale its use of the platform each quarter. Upsolver integrates with Bigabid’s Amazon S3, Amazon Athena, Amazon Elastic MapReduce (Amazon EMR), and Amazon Kinesis Firehose footprint.

Using Kinesis Firehose, the company ingests multiple data streams into its Amazon S3 data lake. It then uses Upsolver for data ETL (extract, transform, load) to build complete user profiles in real-time. The company also uses Upsolver and Athena for business intelligence (BI) reporting that is used by its data science team to improve machine learning models.  

Diagram: How Bigabid Uses Upsolver on AWS 

By using Upsolver’s platform for real-time stream processing, Bigabid saved months of development time and enabled its engineering team to focus on driving value from its data. “A startup like us should be focused on its business goals and not on building infrastructure,” says Attias. “Instead of building this solution ourselves, using Upsolver and AWS, we could focus immediately on how we use the solution and driving value from our data. Upsolver helped us grow very quickly and gave us a year jump start to begin using real-time stream processing at the scale we required. I personally don't think any company should build a solution like this themselves when there are professionals like Upsolver who have built the tooling and built it very well.”

Bigabid drives better insights for its clients by providing them with a dashboard to help them visualize the benefits of using the Bigabid solution based on their specific Key Performance Indicators (KPIs). Since using Upsolver on AWS, Bigabid has improved its modeling accuracy and its performance by a scale of 200x compared to last year. “Clients are happy with the results we have been able to drive and I credit our success to our ability to focus time on our algorithm and models,” says Attias. “We are very thankful to Upsolver for making our lives easier.”