Malayala Manorama Builds Revolutionary Analytics Engine on AWS

Manorama online

A vernacular news leader in India based out of Kerala

If you’re a Malayali in the Indian state of Kerala, chances are you’ve visited Manorama Online to read about current affairs or even to check the latest movie listings. The company—and its corresponding web assets—represent the digital branch of Malayala Manorama, a large Indian media conglomerate that specializes in local content for Malayalis. About 30 percent of is news; the remaining 70 percent includes content such as entertainment, classifieds, lifestyle services, and digital subscriptions. The website attracts about two million daily users.

In 2016, Manorama embarked on a project to develop an in-house analytics pipeline that could unify enormous amounts of raw data from multiple web domains and convert it into meaningful insights. The company currently has 10 major domains, including its matrimonial and real estate sites, and it has plans to further expand its digital footprint.

High-Volume, High-Velocity Data Pipeline

Manorama Online was an early cloud adopter, beginning its journey with Amazon Web Services (AWS) in 2011. Like many media companies in India, the business was also using the free version of Google Analytics. This tool served basic analytics needs, but Manorama became frustrated with its limitations—it analyzes only data samples for a particular period and was incapable of processing large datasets.

HiFX, an AWS Partner Network (APN) Select Consulting Partner, has been Malayala Manorama’s technology partner for more than 20 years and was approached to design its new analytics pipeline, which it completed in late 2018. The Lens analytics pipeline currently handles about 1 TB of data per day and can process up to one million events in less than a minute, with an average of 300 million events processed per day. The horizontally scalable architecture of Lens allows for more events to be processed.

“We’re very impressed with AWS products and solutions and the pace at which new features and services are added to the platform.”

Renjit Paul, Deputy General Manager of IT, Malayala Manorama

  • About Manorama Online
  • Manorama Online is the digital arm of parent company Malayala Manorama, offering in-depth local news coverage of the southern Indian state of Kerala, plus other content such as entertainment, classifieds, lifestyle services, and digital subscriptions. The brand has a strong following from Malayalis across the globe, and it has two million daily visitors to its website.

  • Benefits
    • Processes up to 1 million events in the data pipeline in less than 1 minute
    • Displays data on concurrent users in a real-time dashboard in less than 5 seconds
    • Helps marketing team track page and quality views and distinguish between unique and repeat visitors
    • Increased conversion rates for a sales campaign by 13%
    • Eliminates data security concern
  • AWS Services Used

More Conversions for Sales Campaigns

Prior to Lens, Manorama struggled with real-time analytics, a vital activity for news organizations. Now, the API turnaround latency for most workloads is just 12.6 milliseconds. Video editors can now see in real time how users are engaging with content on the website and make immediate adjustments to an article’s title or publish videos in different sections to attract more traffic. They can also track the number of page views per visit, the average time spent on each page, and the number of quality views, as well as differentiate between unique and repeat visitors.

For its classified services—especially on its matrimonial site for individuals seeking a partner—Manorama’s telemarketing team is able to better segment and target certain communities or religions. “Using the Lens dashboards and recommendation engine, we are able to get a 360-degree view of our customers and their interests and this helps us to improve our operational efficiency, conversion rates, and revenues,” says Renjit Paul, deputy general manager of IT at Malayala Manorama. The company uses Lens to measure the effectiveness of a telemarketing campaign and to get more site visitors to sign up and subscribe for its matrimonial services. They found that conversion rates increased 13 percent with Lens compared to similar previous campaigns.

As a long-time user of AWS Business Support, Manorama upgraded to AWS Enterprise Support at the start of the project. “We have been receiving tremendous help from the Enterprise Support team. We are able to consult with them on our business plans and technical challenges along the way and get real-time feedback,” says Paul. “Before the business embarks on any project, we always work with the AWS Enterprise Support team and take them through our technical design and implementation plan. They know our existing stack and our requirements, which helps them guide us in the right direction, at times using new services or apprising us of service roadmaps.” Manorama also follows the AWS Well-Architected Framework to ensure that all applications are designed properly for optimal resource utilization and performance.

Real-Time Dashboard

“Lens is quite a revolution in the media space,” says Mohan Thomas, president of technology at HiFX. The system handles both data streaming and batch analytics, and it includes a centralized dashboard accessible across all Manorama divisions. “Coming up with a real-time dashboard was not easy, but we found a solution using Amazon Kinesis, Spark Structured Streaming on Amazon EMR, and Amazon Elasticsearch Service,” he says. “We were then able to show data on concurrent users active on the site—which is a variable process—in less than 5 seconds.”

The company uses the Spark engine on Amazon EMR for batch data processing, after which data is written to an Amazon Elasticsearch Service and Amazon Redshift component for dashboard purposes. Before this takes place, data enters through an Amazon Kinesis stream and is cross-checked for errors. The data is then written to a data lake primarily formed with an Amazon Simple Storage Service (Amazon S3) bucket. Only then does processing and analysis begin. “AWS recommended that we use Amazon S3 for the data lake, and that saved us quite a lot of money and effort,” Thomas adds.

Digital Marketing Validation

Marketing teams have also used Lens to boost application traffic and continually assess their efforts. Less than two months after launching the pipeline, Manorama saw the daily usage rate of its mobile application for matrimonial classifieds increase by 52 percent. Editors can continually tweak the mobile app and the main Manorama application to bring the features that people are using most to the forefront or landing page. On its websites, the company has seen 16 percent more concurrent users since introducing Lens and is targeting a 50 percent increase in 2019. “We consider this an effective measurement of our digital marketing efforts,” says Paul. “Furthermore, we can validate our marketing data to make sure we are not losing money or resources on unnecessary campaigns.”

Data security was a nagging concern with the previous free analytics tool, because Manorama stores a lot of personal data on its users. With Lens, security concerns have disappeared. In addition, Manorama can integrate unique and confidential data into its dashboards, as well as report functions for complete visibility into individual usage patterns.

Increased Engagement with Machine Learning

Looking ahead, the company hopes to boost subscriptions to its websites with the introduction of a content recommendation and personalization engine, which is currently in development using Amazon SageMaker. Plans are also underway to use Lens for analyzing usage patterns in different countries to better cater to those markets. Increased user engagement could also lead to more advertising revenue and further increase sales conversions.

Overall, says Paul, “We are very impressed with AWS products and solutions and the pace at which new features and services are added to the platform. We get complete visibility into upcoming services and the future roadmap as part of the AWS Enterprise Support, and that’s something we haven’t seen with any competitor. The support is top-notch, and we see how passionate AWS is to make you successful.”