Meta Networks Analyzes Network Traffic with an AWS Data Lake and Accelerated Big Data Engineering

Modern technology allows companies to go global quickly. With more employees working remotely, businesses are no longer tethered to a physical location to drive productivity. However, companies with remote employees face security and accessibility challenges.

Meta Networks’ mission is to provide companies with a holistic solution to network security through a software-defined perimeter (SDP), an alternative to virtual private network (VPN) technology for secure remote access to any application. Meta Networks is a cloud-based solution and uses Amazon Web Services (AWS) in the management plane to manage and govern AWS and non-AWS resources.  

“Organizations no longer have a well-defined perimeter for network security,” says Alon Horowitz, co-founder and vice president of research and development at Meta Networks. “Our goal is to provide a network platform that deploys a zero-trust security model—in which users have a unique, fixed identity and one-to-one connections are created dynamically between a user and the resources they need to access—to embolden organizations to manage network security in the cloud.” 

Given the nature of its solution and the company’s rapid growth, the amount of inbound and outbound traffic and streaming data the Meta NaaS platform collects in real-time is substantial. The company knew that by taking a data-driven approach and building a big data solution, it could develop new opportunities for both internal teams and customers.

“We built a big data solution to optimize data ingestion and streaming of data into an Amazon Simple Storage Service (Amazon S3), along with analysis and monitoring capabilities designed to provide visibility at multiple points,” says Horowitz. “Our logs contain meaningful insights and information for customers as well as useful data for internal research, offline analysis, and reporting.”

The Meta Networks team decided to find a partner to engage in building and tailoring a big data solution for their needs. "Most of our developers don't come from a big data background and we didn't want to develop this expertise in-house," says Horowitz. "My goal was to find an easy-to-use solution with self-service capabilities that I could assign one developer to manage." 

Meta Networks soon became familiar with Upsolver, an AWS Partner Network (APN) Advanced Technology Partner. Upsolver’s service focuses on helping companies that generate event streams to quickly analyze data using Amazon Simple Storage Service (Amazon S3) as a storage layer. Upsolver’s visual interface is used to ingest data from Amazon Kinesis to S3 and then ETL the data from S3 to Amazon Athena, Amazon Elasticsearch, and Upsolver’s own real-time lookup tables. Upsolver’s data lake ETL include a join between Kinesis and S3 so Meta Networks could easily combine real-time and historical data in their analytics.

“Given Upsolver’s ease-of-use, we got up-to-speed and into production very quickly,” says Horowitz.  

How Meta Networks uses Upsolver to Ingest Data into its Data Lake 

Within three weeks, Meta Networks deployed an analytics solution on Amazon Athena to production and could slice and dice its network data from multiple customer deployments using SQL. The company later used Upsolver’s lookup tables and Elasticsearch to create customer facing dashboards that present interactive real-time data at the end point level.

“Upsolver enabled us to switch to a data lake architecture 6-months earlier than we planned,” says Horowitz. “Since it was so easy to use, we got up to speed and into production very quickly.”  

By choosing Upsolver, Meta Networks saved months of development time and hundreds of thousands of dollars in developer hours that would have been required to develop an alternative solution in-house using Apache Spark. “A key advantage for us is that by using Upsolver, we have just one developer maintaining our AWS data lake,” says Horowitz. “It’s not something we need to invest additional money or resources in and frees up our developers to focus on using data to develop new solutions for our customers.”

Today, Meta Networks processes terabytes of streaming data daily and experiences a one second average query response time for data. Both Meta Networks' internal teams and customers benefit from its rapid query times. "Our customers get important insights and transparency and can use the information we provide with other security solutions to drive additional insights and actions," says Horowitz.

Recently acquired by Proofpoint, a cloud-based cyber-security solutions provider, Meta Networks plans to advance its use of Upsolver and continue to focus on developing its data solutions to take advantage of new insights for customers through predictive analytics. “Upsolver’s simple integrations are very valuable for us,” says Horowitz. “When we have new ideas to explore, Upsolver’s team is very responsive. We are very pleased with the solution and the service.”