OpenSecrets uses AWS to transform political transparency through enhanced data matching

OpenSecrets is a nonpartisan, independent nonprofit organization whose mission is to serve as the trusted authority on money in American politics. It pursues this mission by providing comprehensive and reliable data, analysis, and tools for policymakers, storytellers, and citizens. Its vision is for Americans to use data on money in politics to create a more vibrant, representative, and responsive democracy.

Through the AWS Imagine Grant—a public grant program that provides both cash and Amazon Web Services (AWS) credit funding to registered nonprofit organizations that are using cloud technology to accelerate their missions—OpenSecrets embarked on an ambitious project to revolutionize its political contribution database. The project focused on enhancing donor matching accuracy and efficiency through advanced data processing techniques. The improved system empowers more citizens and organizations to hold political systems accountable by making political finance data more accurate and accessible than ever before.

Wrestling with inconsistent political data

Political contribution data arrives from multiple sources with varying formats, naming conventions, and data quality standards. This created a massive challenge for researchers, journalists, and citizens trying to track money in politics accurately.

The OpenSecrets team was spending disproportionate amounts of time cleaning and reconciling data rather than analyzing it for meaningful insights. This manual process was not only time-intensive, but also prone to human error—potentially compromising the accuracy of the organization’s data. The challenge was particularly urgent because political finance data grows exponentially during election cycles, making manual processing increasingly unsustainable.

Without an automated solution, OpenSecrets risked falling behind in its mission to provide timely, accurate information about campaign funding and lobbying activities. The organization needed a system that could handle hundreds of millions of records while maintaining the high precision standards required for political transparency work.

Building a scalable data matching solution

OpenSecrets initially proposed using machine learning for entity resolution, but as the project progressed, the team shifted to a more deterministic approach that better served their specific needs. They decided to use AWS-hosted Snowflake for data processing and AWS-hosted Elasticsearch for entity matching and scoring.

Running both services on AWS provided OpenSecrets with the scalability, speed, and centralized infrastructure necessary to handle its massive datasets. The AWS infrastructure allowed researchers and the tech team to process hundreds of millions of records efficiently while maintaining the flexibility to adapt their approach as they learned more about their data challenges.

The approach they ultimately chose offered several advantages over their original machine learning proposal. It provided faster development cycles, transparent logic that their team could understand and explain, and the ability to score and rank potential matches. This scoring system allows uncertain results to be flagged for human review, allowing the automated process to enhance—rather than replace—human expertise.

Transforming political finance research

The new system matches hundreds of millions of records with greater accuracy, automating entity resolution while flagging records with insufficient confidence for human review. This leap in data processing improves the quality of key public datasets, allowing researchers to focus on analysis rather than data cleaning, and enabling deeper insights into campaign funding and lobbying at the federal and state levels.

The enhanced data quality enables journalists to write more accurate stories, researchers to conduct reliable studies, and citizens to make informed decisions about political candidates. OpenSecrets’ transformation supports democracy through increased transparency, while the system’s scalability meets growing demands for political transparency tools as it continues to expand for new data sources.

Lessons for nonprofit technology implementation

OpenSecrets’ experience offers valuable guidance for other nonprofits embarking on technology transformation projects. The leadership team’s first piece of advice is to be flexible, as the original plan might not be the best path once you’re deep in the work.

The OpenSecrets team also emphasizes the importance of building with users in mind. The team needed clear, explainable match logic, not a black box solution. This user-centered approach led to the development of a final system that could be trusted and effectively utilized by OpenSecrets staff and external partners.

Perhaps most importantly, OpenSecrets learned not to wait for perfection. The team recommends launching, learning, and refining iteratively. Running everything on AWS made it easier to pivot quickly without re-architecting their entire system, enabling them to adapt their approach based on real-world testing and feedback.

How you can support OpenSecrets

Democracy requires political accountability, which can only be achieved through transparency. OpenSecrets supports these core tenets of our political system by providing comprehensive and reliable data, analysis and tools for policymakers, storytellers, and citizens. Learn more about how you can support OpenSecrets’ vision for an America in which data on money in politics is used to create a more vibrant, representative, and responsive democracy on the organization’s website.