Trustpilot Processes 20 Million Daily Events Using Amazon EventBridge Pipes

Trustpilot helps consumers share and read reviews based on the experiences of other users, while its business-to-business offering helps companies manage their online reputations and gather insights. As Trustpilot’s user base grew, so did the complexity of managing data across multiple systems, especially when it came to tracking the status of consumer review invitations and sending this information to its newly launched Salesforce package. With that increased complexity, managing the flow of information between systems required extensive custom code.

Trustpilot stores large amounts of data in Amazon DynamoDB, a serverless, NoSQL database service that lets companies develop modern applications at scale. “At Trustpilot, our largest Amazon DynamoDB table has over 3.5 billion items and close to 10 billion events taking place yearly and constantly growing,” says Tudor Cojocaru, principal software engineer at Trustpilot. “We were processing an average of 35 million events per day from a single table, and our engineers had to focus on writing integration code while focusing on scalability and resiliency instead of business logic and creating value for our customers.”

Trustpilot has run its infrastructure on AWS since 2013 and was already familiar with AWS serverless technologies. When Trustpilot’s engineering team learned about Amazon EventBridge Pipes, they recognized an opportunity to streamline their event processing pipeline at scale. The service could handle data processing—with out-of-the box filtering and basic transformation—at high volumes and without requiring custom integration code, which aligned with Trustpilot’s need to manage consumer and customer data at scale.

Beginning in August 2023, in just under 1 month, Trustpilot’s engineering team went from conception to production release of the Amazon EventBridge Pipes integration. The new solution is centered around its largest Amazon DynamoDB table, which stores over 3.5 billion items. When an operation takes place in the table, it activates an event through DynamoDB Streams, a feature that captures a time-ordered sequence of item-level modifications in any Amazon DynamoDB table and stores the information in a log for up to 24 hours.

After retrieving events from the Amazon DynamoDB streams, Amazon EventBridge Pipes filters them, focusing specifically on INSERT and MODIFY eventName instances while disregarding any REMOVE instances. The filtered data is then transformed, enriched, and streamed to Amazon Simple Notification Service (Amazon SNS), a fully managed Pub/Sub service, which distributes the events to multiple data consumers across Trustpilot’s systems.

“Amazon EventBridge Pipes helped us consume the updates in Amazon DynamoDB using Amazon SNS in a resilient way,” says Cojocaru. “That empowered our engineers to focus on customer value creation while facilitating faster iteration of ideas and prototypes with resilience and scalability.”

To manage the integration with its Salesforce package, Trustpilot uses functions in AWS Lambda, a compute service that runs a firm’s code in response to events and automatically manages the compute resources. Trustpilot’s AWS Lambda functions subscribe to Amazon SNS topics and perform additional filtering based on requirements that need to be synchronized with customer Salesforce instances.

The solution proved remarkably resilient in handling Trustpilot’s demanding workload. When traffic spiked by five times the usual volume within 1 minute, Amazon EventBridge Pipes automatically scaled to manage the increased load without intervention from the engineering team. Since the implementation, the system has seamlessly handled tens of millions of events daily with virtually no errors. “We never had to touch the Amazon EventBridge Pipes implementation,” says Cojocaru. “We achieved virtually 100 percent uptime and never experienced any event drops despite unpredictable instances of exponential growth in traffic.”

The implementation of Amazon EventBridge Pipes reduced Trustpilot’s event processing expenses by 30 percent compared with traditional approaches. That cost-efficiency stems from the service’s unique filtering capabilities—Trustpilot only pays for the events that are passed through to the target.

More significantly, the solution transformed how Trustpilot’s engineering teams do their work. By removing the need to write and maintain custom integration code, engineers have been able to focus on developing new features and products for customers. The fully managed service removed the burden of infrastructure maintenance, empowering the team to work on innovation rather than troubleshooting. That shift accelerated Trustpilot’s ability to bring new products to market and enhance existing services for its customers.

Looking ahead, Trustpilot plans to expand Amazon EventBridge Pipes to other use cases, such as streamlining its use of Amazon Managed Streaming for Apache Kafka (Amazon MSK), a streaming data service. As it moves forward, Trustpilot remains committed to delivering innovative products while maintaining operational efficiency—and it will continue to use AWS as it continues this pivotal journey.