Near Streamlines Big Data Processing with Data Lake on AWS

With information on 1.6 billion people in 44 countries, Near has one the largest datasets of people’s behavior on the planet. Near is a data intelligence company whose AI-powered platform helps media companies and publishing houses deliver more effective campaigns. Founded in Singapore in 2012, the company expanded to Australia in 2014 and Europe in 2015 and is now growing its business in the US. Near’s combination of products and analytics yield real-time results for customer queries that support marketing campaigns.

The company migrated to the Amazon Web Services (AWS) Cloud in 2019 because it was struggling with latency and long lead times for provisioning, often taking 8 hours or more to scale up its bare-metal cloud infrastructure from another provider. “Our goals for migration were high elasticity, low latency, and increased efficiency in our data science team,” explains Mahesh Vandi Chalil, senior vice president and head of engineering at Near. Retrieving and processing raw data from various third-party sources was often a lengthy and painstaking process, and engineers were limited in terms of big data analysis.

“From a data science perspective, the motivators for moving to AWS included scaling our data platform in new countries and fully automating our lifecycle management for machine learning and artificial intelligence models,” adds Ravi Kaushik, vice president and head of data science at Near.

Security was also a major factor for Near. Amit Kumar, head of DevOps at Near, says, “By leveraging solutions such as AWS Identity and Access Management (IAM), we can now manage access, create roles, and set permissions to services and resources securely and that’s a great plus.”

Engineers performed tests before implementing AWS to ensure latency was consistently under 100 milliseconds, the target SLA required to serve global customers such as mass media company News Corp. Many of Near’s engineers were new to the AWS Cloud, so Mahesh subscribed to AWS Enterprise Support to get priority technical assistance, especially during migration. “The knowledge of AWS support teams is remarkable. This really stood out from our previous vendor,” Mahesh comments.

Near’s 30 engineers have regular meetings with their AWS technical account manager and solutions architect to learn about the nuances of AWS products currently in use and how to optimize their architecture. Even during the peak of migration, when support tickets were running into double digits, the company received the help it needed to avoid system errors and downtime. This assistance often entailed finding the right Amazon Elastic Compute Cloud (Amazon EC2) instance sizes or types for each workload. “With AWS, we get the right tools for the right job. An example is the Amazon EC2 R5 instances that are memory optimized and suited to run our Apache Spark workloads,” Ravi says.

Near had been using Apache Hadoop and its corresponding Hadoop Distributed File System (HDFS) to store and process data. However, the team switched to Amazon Simple Storage Service (Amazon S3) to improve storage scalability and start building a data lake. “Accessing data in Amazon S3 is much easier than our HDFS solution,” Mahesh says. “Amazon S3 keeps things organized with logical partitions and provides a single storage for all data types. It’s extremely stable with high availability, and the redundancy aspect is taken care of by automatically storing data across multiple devices spanning a minimum of three AWS Availability Zones.”

Engineering teams are now incorporating Amazon Redshift and Amazon Athena to set up a data warehouse and run serverless queries on top of Amazon S3. “We are seeing firsthand how powerful these services are for running big data queries for some of our critical use cases,” Mahesh says.

For data processing, the company switched from Cloudera to Amazon EMR. “On-demand processing for our data science and engineering teams has become much easier with Amazon EMR,” Mahesh says. Capacity planning used to be a headache for Mahesh, but Amazon EMR has helped address the uncertainties that come with expansion. Furthermore, with the support of AWS Professional Services, Near experienced a smooth migration to Amazon EMR. “The AWS Professional Services team validated our architecture and adoption of Amazon EMR and was always available to provide highly skilled support. We worked as one team,” Mahesh adds.

About 40 percent of Near’s Amazon EMR clusters are running on Amazon EC2 Spot Instances, which has reduced both the cost and time required to run big data queries. Since migrating to AWS, the number of data requests Near handles daily has gone from five billion to eight billion. Although its total spend has remained relatively stable, the volume of data has spiked, the number of data scientists has doubled, and more customers have come on board. Data scientists can also perform more complex queries for analysis.

This is expected to yield long-term benefits in terms of winning bigger projects and attracting new clients because global marketers often require their marketing technology vendors to have a minimum amount of data before awarding new contracts. Of particular interest to global marketers is Near’s data enrichment service. This service provides consumer intelligence that enhances marketers’ first-party data, ultimately leading to more detailed insights about their audiences. The more data Near is able to ingest and process, the greater its potential for attracting clients seeking data enrichment.

Mahesh has peace of mind knowing that Near’s architecture is robust enough to handle expansion at any scale. “Whenever we decide to further expand in terms of new countries or markets, I’m not worried, because I can scale to any level with our global AWS infrastructure,” he says.

Furthermore, the company is also confident that it can manage costs seamlessly while focusing on expansion. Sooraj Balakrishnan, principle engineer at Near, says, “By using tools such as Amazon CloudWatch, we’re able to gain a high level of visibility across our applications to better manage operational performance and resources, resulting in improved cost predictability and projections.”

Going forward, DevOps engineers are excited about trying new AWS tools such as Amazon CodeGuru to automate code hygiene. Mahesh says, “Developers are happy, and our data science teams have benefited from using AWS to drive innovation.”