Deluxe Modernizes Matching Platform on AWS with Impetus, Cutting Costs by 99% and Tripling Processing Speed

As a payments and data leader, Deluxe supports a variety of matching workloads, ranging from routine reconciliations to large-scale, high-frequency data matching operations. These workloads often require comparing incoming datasets against large proprietary data assets, using both exact and fuzzy matching methods. Deluxe relies on its data matching operations to power a variety of customer-facing services, including sending payments and marketing to the right audience. Aiming to modernize its matching needs, Deluxe experimented with OpenSearch for storing vector embeddings and vector search. While this approach provided a starting point, it also delivered inconsistent results. For example, the OpenSearch algorithm occasionally missed high-scoring matches.

“Accuracy and performance aren’t just technical metrics; they are fundamental to the reliability of our matching operations,” said Ashish Agrawal, director of machine learning at Deluxe. “Inaccurate matches, whether false positives or false negatives, can lead to operational inefficiencies, missed connections, or delays in delivering results. Matching is a foundational process that supports a wide range of data reconciliation and validation needs across our business.”

In addition, as data volumes increased and client demands grew, the matching platform encountered limitations. The solution struggled with more than 30 concurrent workloads, resulting in throttling, job failures, and persistent “too many requests” errors even after aggressive scaling. This ultimately affected timely execution. Even the most accurate match is only valuable if it is delivered within the required timeframes. What began as a technical limitation quickly escalated into a broader operational issue. SLAs were at risk, engineering teams were frequently diverted to address failures, and confidence in results diminished. At the same time, running a 24/7 OpenSearch cluster was costly and did not deliver the speed, reliability, or flexibility required. “We needed a system that could scale without compromising accuracy and do it in a way that was sustainable from a cost and operations standpoint,” Agrawal said. “That meant rethinking the architecture to handle growing data volumes, maintain consistency in results, and streamline operations.”

Working with AWS Premier Tier Partner Impetus, Deluxe redesigned its matching platform from the ground up to address scalability, performance, and cost-efficiency challenges. Deluxe brought deep knowledge of its internal systems and business rules, while Impetus led the technical execution, drawing on its experience modernizing large-scale data platforms on Amazon Web Services (AWS). Deluxe had an existing partnership with Impetus for data and AI development, and that prior knowledge was key for collaboration and delivery. “That familiarity helped accelerate our work as we built something fundamentally new,” Agrawal said. “Impetus already understood our systems, which made them the ideal partner to lead the redesign.” The new architecture moves away from a monolithic, always-on system to a flexible, distributed, cloud-native service capable of adapting to varying workload sizes and processing demands.

The system is orchestrated across several AWS services. AWS Lambda and Amazon DynamoDB manage configuration settings and concurrency limits, queuing and validating job requests. AWS Glue performs preprocessing steps such as filtering incomplete records, applying hash-based matching, and partitioning data based on operational requirements. Partitioned datasets are processed in Amazon SageMaker using FAISS-based in-memory vector search to narrow potential matches for each record. A final AWS Glue job applies proprietary matching rules and writes the results to Amazon Simple Storage Service (Amazon S3) for downstream use.

To maximize efficiency, the 24/7 infrastructure was replaced with on-demand compute that automatically adjusts resources based on job size and complexity. Python UDFs were converted to Spark-native equivalents, improving execution speed by up to 7–10 times, and unused intermediate data operations were eliminated to reduce overhead. To better align with deterministic matching needs, vector embeddings were replaced with string similarity algorithms such as Soundex and Jaro-Winkler. The team also upgraded to AWS Glue 5.0 to take advantage of Apache Arrow support and other performance enhancements. The result is a modular, event-driven matching service composed of independently scalable jobs—each optimized for its workload. By separating storage from compute and applying intelligent orchestration, the platform now supports virtually unlimited matching runs (bounded only by AWS service quotas) while aligning with cost, scalability, and reliability objectives.

With the new matching platform in place, Deluxe transitioned from slow, failure-prone batch jobs to predictable, high-speed execution powered by on-demand processing. Matching workloads that previously took hours—or in some cases weeks—now complete in a fraction of the time. For example, a dataset exceeding 100 million records now processes in about two hours instead of weeks. These performance improvements were matched by significant cost reductions. By adopting a just-in-time, partitioned execution model, monthly vector search costs dropped from approximately $75,000 to around $700, bringing the cost per match run to about $1 per million records. Operational overhead was also reduced, with compute resources dynamically allocated based on workload requirements and no manual scaling needed.

The new architecture also eliminated prior concurrency constraints. Where the earlier solution could not reliably handle more than 30 parallel jobs, the redesigned platform supports hundreds of concurrent jobs in production, with the flexibility to increase it further by increasing AWS service quotas. This elasticity allows Deluxe to process more workloads in parallel, support larger and more complex datasets, and respond quickly to changing operational demands. “With Impetus and AWS, we’re no longer constrained by processing time or concurrency limits,” said Satish Balasubramanian, divisional CTO of data and enterprise architecture at Deluxe. “That’s been a game-changer for scaling our operations.” By modernizing the matching service on AWS, Deluxe removed key bottlenecks and established a scalable, cost-effective foundation capable of meeting current needs while adapting to future growth.