Yotpo Reduced Operational Overhead for Mission-Critical Offering Using Amazon OpenSearch Service

Yotpo’s main offering is its retention marketing platform, which connects solutions for loyalty, SMS, email, reviews, and other marketing tools. The solution is built on a customer data platform (CDP) that centralizes customer data so that brands can run analytics, send out targeted marketing campaigns and personalized SMSs or emails to customers, and even use predictive analytics powered by artificial intelligence. The main technology component of the CDP is an Elasticsearch cluster.

In 2023, Yotpo realized that it was spending a lot of time on maintenance for its Elasticsearch cluster. Additionally, the company experienced multiple incidents due to human errors arising from manually configured changes to the cluster. “We could remove these issues if we had automated processes or a managed solution, and we preferred developing solutions for our customers rather than maintaining our cluster or developing an automation solution ourselves,” says Dan Gleyzer, senior backend developer at Yotpo.

The company looked for a solution from AWS. “AWS is our go-to solution provider in the cloud,” says Leon Lantsman, group manager of Open Data Platform at Yotpo. “AWS provides high-quality, professional service, which was an important influence in our decision.” Yotpo wanted to continue using Elasticsearch or similar technology for its CDP, and it wanted to find a managed service to run its CDP cluster. “We prefer managed solutions because we want to focus on the business logic implementation and providing new capabilities to our customers instead of working on maintenance,” says Liat Iusim, tech lead at Yotpo. “Even scripts for automating the process would require maintenance and are error prone, so we chose to use Amazon OpenSearch Service as a fully managed solution.”

Yotpo began the Amazon OpenSearch Service implementation with a four-phase proof of concept. It first tested that the CDP would function properly using the service, then tested that it could easily change configurations for its cluster in the service. During the third phase, Yotpo verified that the performance using Amazon OpenSearch Service was equal to its self-managed cluster, including during peak traffic conditions such as Black Friday. And finally, it estimated the costs to verify that making the change would stay in the same budgetary range. Migrating a mission-critical search cluster required specialized experience, so Yotpo enlisted BigData Boutique, an AWS Partner, to help plan the migration. With many successful migrations under their belt, working with a partner like BigData Boutique helped ensure Yotpo’s migration would have a higher likelihood of success.

Yotpo began the project in December 2023, completed the proof-of-concept tests in January 2024, and migrated the 50 TB of its CDP to Amazon OpenSearch Service in February 2024. The tests were thorough, making it possible for the migration to be completed in 3 days with zero downtime while keeping customer data fresh throughout the process. In the first 3 months after the migration to Amazon OpenSearch Service, Yotpo had smooth operations with no incidents. Additionally, the team has been able to easily change the configuration of its cluster—adjusting throughput, storage, disk size, and input/output operations per second—using infrastructure-as-code files and blue/green deployments, all with zero downtime and no incidents of human error.

The migration sped up Yotpo’s team capabilities because the development team now owns the cluster directly. When the cluster was self-managed, every change took several days to implement in coordination with the DevOps team, adding up to around 3 weeks per year. Now, it takes a few minutes for the developers to make the changes themselves. “This independence gives us the ability to move faster with any changes that we need to make,” says Gleyzer. “The configuration in Amazon OpenSearch Service verifies that we implement things correctly, which will help avoid incidents and reduce potential downtime.”

Yotpo was also able to reduce the costs of its CDP cluster by 11 percent, using instances powered by AWS Graviton processor, a family of processors designed to deliver the best price performance for cloud workloads. It can also deploy its cluster in multiple Availability Zones for added reliability without additional costs.

Yotpo has a very high ingestion rate of over 3,000 events per second and a search rate of around 1,000 requests per second. During the Black Friday and Cyber Monday season, those rates triple. “In previous years when we didn’t have Amazon OpenSearch Service, the scale testing for the Black Friday season was very time consuming for us,” says Lantsman. “Now, when we’re looking at Black Friday and Cyber Monday, we are all very confident knowing that it will be a smooth process to scale on AWS.”

Yotpo will be upgrading to the latest version of Amazon OpenSearch Service to benefit from multiple performance improvements as well as new features such as segment concurrent search and flat object data type. The company expects these new features to reduce costs and increase ingestion and search performance.

Overall, the migration to Amazon OpenSearch Service was a success. “This project has been on the table for a long time, and once we decided to tackle this using AWS, it was very doable,” says Lantsman. “The results are amazing: we got what we aimed for, and the transition itself was so smooth.”