How The Meet Group Extracts Real-Time Insights from Streaming Data Using Upsolver and Amazon Athena

The Meet Group is a leading provider of interactive online dating solutions. After several acquisitions, The Meet Group sought a solution to integrate its data pipelines and central data collection to drive better real-time aggregate insights and analysis. With help from Upsolver, an AWS Partner Network (APN) Partner, The Meet Group developed a powerful and scalable data lake that enables the company to save time and innovate faster to meet business demands.

Nothing can replace the powerful feelings we get when we genuinely connect with another human being. Sometimes, the challenge lies in how we get there.

The Meet Group's mission is to help safely foster human connections using the power of technology. The Meet Group's mobile apps are comprised of five brands and offer different experiences and options for connecting individuals. Its applications are used by millions across the globe daily and are available in multiple languages. Given the scope of The Meet Group’s reach, the company generates hundreds of millions of events from its software daily. The team then seeks to use the collected data to understand user behavior at an aggregate level, to drive better short- and long-term user experiences and application development.

“We have many daily high-velocity events we categorize into thousands of streams and collect into multiple data silos for analytics and microservices coordination," says Bob Lunney, Lead Data Architect for The Meet Group. “We use this data at an aggregate level to analyze user behavior and actions, to understand how well our services are running, and to measure how new features are performing.”

The Meet Group grew quickly through acquisitions. With each acquisition, Lunney and the team were tasked with managing different data pipelines initially created to support a separate brand now managed under The Meet Group umbrella. "We have five different applications we run that all run off individual backends," says Lunney.

The data team recognized that to drive insights for the company at an aggregate level and provide a more global perspective, they needed to consolidate information coming from each brand. Given the streaming nature of the data, The Meet Group sought to find a solution to integrate its data pipelines and central data collection to drive better real-time aggregate insights and analysis. Having already decided to focus all new development on Amazon Web Services (AWS), Lunney and the team turned to their AWS contacts for guidance.

Upsolver, an APN Advanced Technology Partner, strives to make traditionally difficult data ingestion and ETL processing easier for users by removing the need to write code. The company's mission is to empower users to focus on driving new business value with data, rather than managing and operating different data pipelines.

After becoming familiar with Upsolver’s solution, The Meet Group found it easy to get up-and-running with Upsolver quickly. “Upsolver makes big data much easier than it would be if we had to research all of the technology it covers, learn how to apply it, implement the same type of solution, and then deploy it. We felt we would probably end up with a less generalized solution requiring constant fiddling,” says Lunney. “Furthermore, Upsolver has been very responsive to our requests for help and enhancements. Their support is phenomenal.”

Diagram: The Meet Group’s Data Pipeline Architecture on AWS

The Meet Group leverages Upsolver as a real-time collection and transformation engine that connects data producers such as Apache Kafka, Amazon Kinesis, and operational databases to analysis tools such as Amazon Athena, Amazon Redshift, and Amazon Elasticsearch Service. By choosing a data lake on AWS, The Meet Group can leverage the best tool for each analytics and data science use case, using Upsolver for scaling and optimizing the data pipelines automatically to meet their business demands.

All of The Meet Group’s data is collected to a centralized Kafka datahub. Upsolver ingests this data to Amazon Simple Storage Service (Amazon S3) and extracts a strongly typed schema-on-read which is updated in real time and presented in Upsolver’s visual user interface. In turn, The Meet Group users create data transformation jobs by mapping the schema-on-read to tables in various analytics tools. The syntax for these jobs is SQL-based and each job can mesh multiple data sources into a single table. Once the jobs are run by the user, Upsolver continuously updates the target tables and data becomes queryable in Athena. The data Upsolver creates for Athena is optimized for faster queries by using columnar formats such as Apache Parquet, compression, partitioning, and compaction of small files. The metadata for Athena tables is maintained by Upsolver in the AWS Glue Catalog.

Using Upsolver and AWS has helped Lunney and the team focus their efforts on data analysis, rather than pipeline development and operations. “Frankly, to do what Upsolver does for us would have taken our team months and months of coding—and we would not have come up with a tool as optimized as Upsolver,” explains Lunney. “Replacing our data pipelines with Upsolver dramatically reduced the operational and developmental burden on our team.”

By using outputs in Upsolver, The Meet Group has been able to send data to other affiliates and across suppliers very quickly and easily. “Trying to set that up on our own, particularly for streaming data, would have taken us quite a long time,” says Lunney. The ease with which Upsolver enables The Meet Group to set up outputs in its platform also helps the company take better advantage of AWS services for data management and analysis.

The intuitive nature of Upsolver’s tool has been invaluable for The Meet Group. “Data processing is a very difficult space to be involved in. With the constant evolution of data requirements and the speed required to deliver usable implementations to stakeholders is critical in today’s business,” explains Lunney of The Meet Group. “Upsolver helps time-to-delivery for ETL like solutions in the streaming data realm. A great UI helps developers bring non-SQL literate personnel into a world of data design, aggregation, and modeling that they would otherwise be unproductive in. With the introduction of Upsolver SQL, both ends of the data engineering spectrum can be met.”

“Upsolver is far more than a data warehouse ETL tool,” says Lunney. “It’s a core part of our data strategy and architecture.”