Pinterest Scales Daily Log Search and Analytics from 500 GB to 1.7 TB and Reduces Costs by 30% Using Amazon OpenSearch Service
In 2016, Pinterest—one of the largest visual-bookmarking tools and social networks in the world, now with 400 million monthly active users and growing—was creating 300 GB of logs daily, and that volume was increasing rapidly. But the self-managed open-source Elasticsearch tool the company used to search and analyze the data couldn’t handle the scale. It required constant administrative work from the company’s engineering staff, resulting in ineffective data analysis and an unsustainable operational overhead. Pinterest moved to a third-party proprietary Elasticsearch solution but ultimately found the cost to be unsustainable and the solution to be unable to scale with demand.
Faced with the enormous demand for faster, more efficient log analytics at a lower cost, Pinterest moved to managed analytics using Amazon OpenSearch Service on Amazon Web Services (AWS). There, Pinterest was able not only to scale its log analysis capabilities but also to reduce operational burdens on its software engineers, improve the
security of its proprietary information and private user data, and save costs by as much as 30 percent.
Using [Amazon OpenSearch Service], we can see that the scale has grown tremendously. And since we only have one person managing the pipeline, that shows immense resource efficiency.”
Staff engineer, Pinterest
Enabling Continuous Software Integration and Deployment
The Pinterest observability team relies on Elasticsearch to monitor and issue alerts for new software deployments on the main Pinterest site. With Elasticsearch, the team can identify problems during software deployment, then quickly analyze logs to troubleshoot root causes. “We use metrics to detect all incidents when they happen, but log search is the main tool we use to find out what’s causing them,” says Wei Zhu, a staff engineer for Pinterest’s observability team.
Pinterest’s search for a cost-effective, scalable solution for log analysis began in 2016. Initially, Zhu’s team was self-managing open-source Elasticsearch software to monitor log data and troubleshoot issues with software deployment. But the costly operational overhead and the ever-increasing volume of daily data necessitated a change. Pinterest signed a 3-year contract with a third-party vendor, but that arrangement presented new challenges: the vendor’s high licensing cost drove up the overall cost of the solution, and its unique query language produced a steep learning curve for Pinterest’s software engineers.
In the second half of 2019, with the third-party vendor’s contract expiration approaching, Zhu’s team began migrating its data to Amazon OpenSearch Service. The move made sense: Pinterest was already using a variety of AWS services to scale its processing, storage, and data analysis workloads, and Zhu’s team determined that the move would reduce costs. Also, Amazon OpenSearch Service’s support for Kibana—a free and open user interface that lets users visualize Elasticsearch data—made it more readily accessible for Pinterest engineers. “The cost is much less because we don’t spend resources to manage the infrastructure,” says Zhu. “And Kibana is open source, which makes using the tool much, much easier—if we have questions about using it, we can find the answers online ourselves.” Overall, Amazon OpenSearch Service enables Pinterest to quickly find and resolve issues as part of its continuous integration and deployment of software, helping Pinterest ship new features to Pinterest users faster.
Scaling Data Ingestion and Reducing Costs
In just 1 year after the migration to Amazon OpenSearch Service, Pinterest’s observability team went from ingesting 500 GB of data per day to 1.7 TB per day. By the end of 2020, the team expects to be able to ingest over 3 TB of data per day. Currently, Pinterest is able to monitor and issue alerts for 20 new software deployments a day. “Using [Amazon OpenSearch Service], we can see that the scale has grown tremendously,” says Zhu. “And since we only have one person managing the pipeline, that shows immense resource efficiency.” With only one engineer overseeing its Elasticsearch deployment, the team can turn away from low-value work to focus on innovation and other business-critical tasks, such as exploring new use cases and adding more resources to the log search.
Security was also an important factor for Pinterest’s observability team as it was considering the migration to Amazon OpenSearch Service. “We’re really careful with our proprietary information and private user data, so we set up a whole pipeline with that in mind,” says Zhu. In addition, the team plans to adopt Amazon OpenSearch Service’s fine-grained access control, which enables different roles to access data at the index, document, and field levels. “We are looking to move more fine-grained access control for the search capability,” says Zhu. “Better control of personally identifiable information data would be a big win for us.”
Most recently, Pinterest started using UltraWarm for Amazon OpenSearch Service to save costs for Elasticsearch clusters. This new low-cost storage tier, which provides fast and interactive analytics on up to 3 PB of log data at a fraction of the cost of the current Amazon OpenSearch Service storage tier, has helped Pinterest reduce costs by 30 percent—savings Zhu expects to rise to 40–50 percent. UltraWarm is not only saving costs but also creating opportunities to pursue new use cases. “We’re onboarding a new use case that ingests about 2.2 TB of data per day,” says Zhu. “Without UltraWarm saving costs, I don’t think the use case would be possible. It’s eye opening to see how AWS puts customers first by innovating services to reduce costs for us.”
Investing More to Discover More
Zhu anticipates his team will adopt Amazon OpenSearch Service for analyzing all its Hadoop cluster logs by the end of 2020 in order to save costs on the more than 2.2 TB of data being ingested each day. “In the second half of 2020, we’re going to focus on investing more into log search because we think there’s a lot more value to discover here with Elasticsearch,” says Zhu.
Amazon OpenSearch Service helped Pinterest quickly scale its data-ingestion capabilities and reduce the burden of low-value work on engineers. As a result, team productivity increased. Now, Zhu and his team can quickly and efficiently monitor and issue alerts for new software deployments, helping Pinterest deliver quality features to hundreds of millions of daily users.
Founded in 2010, Pinterest is one of the largest visual-bookmarking tools and social networks in the world. Its 400 million monthly active users explore and experience Pinterest’s 200 billion saved ideas, discovering inspiration to apply to everyday life.
Benefits of AWS
- Scaled data-ingestion capabilities from 500 GB per day to 1.7 TB per day in one year
- Scaled monitoring and alerting capabilities for software development
- Reduced operational costs by 30% with expected cost reduction of 40-50%
- Increased productivity by freeing software engineers from low value work.
- Improved security of private user data and proprietary information in backend infrastructure
AWS Services Used
Amazon OpenSearch Service
Amazon OpenSearch Service makes it easy for you to perform interactive log analytics, real-time application monitoring, website search, and more. OpenSearch is an open source, distributed search and analytics suite derived from Elasticsearch. Amazon OpenSearch Service offers the latest versions of OpenSearch, support for 19 versions of Elasticsearch (1.5 to 7.10 versions), and visualization capabilities powered by OpenSearch Dashboards and Kibana (1.5 to 7.10 versions).
Companies of all sizes across all industries are transforming their businesses every day using AWS. Contact our experts and start your own AWS Cloud journey today.