T-Rex Supports First-Ever Online U.S. Census with Help from AWS Customer Enablement Services

T-Rex relied on AWS to ensure smooth operations for the 2020 U.S. Census—the first time the census had ever been conducted online. T-Rex, an AWS Advanced Consulting Partner, specializes in building large, mission-critical systems for government clients. The company took advantage of AWS Customer Enablement services—AWS Professional Services and AWS Enterprise Support—to access security assessments and AWS Infrastructure Event Management.

Occurring once every 10 years and conducted by the United States Census Bureau, the U.S. Census aims to count every resident in the United States. It is the largest operation conducted by the Federal Government, counting hundreds of millions of people. This critical moment in the nation’s public life determines the number of seats each state has in the U.S. House of Representatives and apportions billions of dollars in federal funds to local communities.

In 2010, the census was still conducted primarily using paper forms. For the 2020 count, the U.S. Census Bureau was determined to modernize the census by utilizing the internet to increase self-response and streamline follow-ups for those who didn’t initially respond.

In 2016, T-Rex Solutions, LLC, an Amazon Web Services (AWS) Advanced Services Partner, won the $1.4 billion technical integrator contract to manage these digital advancements. The job included identifying which of the 52 systems associated with the census would move to AWS; integrating administrative records into the new systems; and deploying, securing, testing, and operating the associated cloud and mobile infrastructure.

T-Rex chose AWS and its AWS GovCloud (US) regions to host census systems. The GovCloud (US) regions offer FedRAMP compliance and enabled the agency to handle sensitive data with high impact risk. “AWS was the right choice for the U.S. Census project because it enabled us to rapidly scale up and scale back down again to support a large, but temporary, effort,” says Jason Keplinger, chief technology and innovation officer at T-Rex.

Data collection for the census was built with two complementary components: Internet Self Response (ISR) and Non-Response Follow-Up (NRFU). ISR allowed the US population to complete their own response to the census through my2020census.gov, a site built to support 1.2 million concurrent users. NRFU was then used by census workers to gather information from individuals who were unable to complete either physical or online census forms. NRFU supported 248 field sites and approximately 400,000 mobile devices using a web-based mobile app. In total, the solution used over 4,000 Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Relational Database Service (Amazon RDS) instances.

With so much at stake, T-Rex worked closely with AWS to ensure the systems were secure and well-architected using AWS best practices. AWS Professional Services provided the Census Bureau with the technical expertise to implement best practices and make changes necessary in securing the census data. AWS Enterprise Support ensured uninterrupted operation during the data collection phase by providing AWS Infrastructure Event Management (IEM), an Enterprise Support offering, to facilitate both the ISR and NRFU components of the effort.

For each of the operations associated with the census, AWS Enterprise Support Technical Account Managers held a deep-dive architectural review. These comprehensive reviews covered incident management, identity and access, data policies, logging, alerts, encryption, and other key areas. These results guided enhancements the AWS Professional Services team used to improve resiliency and security of the census systems. “AWS Professional Services personnel worked side-by-side with us to implement recommendations, especially related to security and scripting, and to perform stress testing on the systems,” says Utpal Amin, deputy chief engineer at T-Rex.

Numerous improvements resulted from these reviews. For example, AWS Professional Services helped T-Rex increase the automation of failover scenarios to ensure continuous operation; even in edge cases. The T-Rex and AWS teams also worked together to implement a patching strategy that supported comprehensive testing before production deployment.

T-Rex reduced cost and added robustness to the Census Bureau’s use of Amazon DynamoDB by implementing recommendations provided by AWS Enterprise Support. “AWS Enterprise Support recommended that we use native Amazon DynamoDB global tables to replicate session information between AWS GovCloud (US) Regions,” says Amin. “This made sure that user sessions were saved and accessible, so they didn’t have to start over with their response.”

T-Rex also worked to build more resiliency into the already existing Apache Kafka infrastructure so data would be preserved even if the data ingestion system went down. They implemented the ability to write data to Amazon DynamoDB tables and then use AWS Lambda serverless functions to transfer data to Kafka in the event of an outage.

The participation of AWS Professional Services and AWS Enterprise Support during the period of the census ensured continuous operation. “During March and early April, we had AWS Professional Services and AWS Enterprise Support experts at the ready to help us address any issues that arose,” says Amin. “We are proud of the results we achieved. For the ISR, we ran a 212-day operation with zero downtime, collected over 59 percent of the census data, and supported NRFU at scale to complete the census.”

The massive, critical effort relying on many new and complex pieces of technology went remarkably smoothly thanks to the close collaboration between T-Rex and AWS. Together they were able to provide the American people with a complete census count more easily and efficiently than ever before.