AdaptX Uses AWS Serverless to Modernize Clinical Data Ingestion

By Gokhul Srinivasan, Sr. Partner Solutions Architect, Startup – AWS
By Andy Schuetz, Principal Startup Solutions Architect – AWS
By Chad McAvoy, VP DevOps – AdaptX
By Cris Gallardo, Cloud Architect – IronCloud

AdaptX

AdaptX clinical management software uses electronic medical records (EMRs) to derive insights and enable providers and clinical leaders to quickly and easily leverage their real-world data and transform patient care.

Accurate, relevant, and up-to-date data is the backbone of AdaptX’s Adaptive Clinical Management solution. AdaptX is an AWS Partner that provides healthcare providers with a clinical performance analytics platform to monitor, improve, and manage patient care.

This post explains how AdaptX built a modern data architecture that leverages Amazon Athena, AWS Lambda, and AWS Step Functions to automate the extraction, transformation, and loading (ETL) of clinical data from provider EMR systems. This new architecture resulted in improved accuracy, scalability, and a better end-customer experience.

Working with Amazon Web Services (AWS) and IronCloud as a design and implementation partner, AdaptX modernized its data ingestion process and realized efficiency gains and data quality improvements.

How AdaptX Modernized its Clinical Data Ingestion

AdaptX leveraged several AWS services to architect a serverless, event-driven solution to automate the ETL of clinical data, as shown in Figure 1. Data flows through the architecture, from left to right, following the numbered steps in the image.

Figure 1 – AdaptX serverless data ingestion architecture.

Hospitals transfer clinical data, as CSV files, using secure file transfer protocol (SFTP) to an Amazon Simple Storage Service (Amazon S3) bucket. A given dataset can have multiple files associated with it, and AdaptX process all files before ingesting them.
When a new object is added to the bucket under the /IN prefix, Amazon S3 sends an event to Amazon EventBridge, which triggers the single file step functions state machine.
The validation Lambda function evaluates the incoming file with the following steps:
- It retrieves a validation schema for the file from an AdaptX internal service.
- Then, it compares the file contents with the validation schema.
- If errors are encountered, it writes them to an error file in Amazon S3 under the /Errors prefix.
If errors occur, the email notifications Lambda function invokes Amazon Simple Email Service (Amazon SES) to email the data team with a link to the error files stored in Amazon S3.
If no errors occur, the Lambda function copies the file to the /Validated prefix in Amazon S3.
The SQL builder Lambda function prepares an SQL query and invokes Amazon Athena to load the data from Amazon S3.
The update file tracker Lambda function adds the file record to the serverless data flow tracker in Amazon DynamoDB.
Amazon EventBridge invokes the query file tracker Lambda function at regular intervals to query DynamoDB for any dataset jobs that have all of the required files processed and recorded in the database.
The AdaptX platform requires a fixed number of files, defined collaboratively with the healthcare providers. The query file tracker Lambda monitors the completeness of a new data payload received from a provider. This Lambda function starts the client daily rollup step functions workflow when all required files are present and recorded in DynamoDB.
The Athena helper Lambda function fetches the required transformations for the data, stored as saved queries in Athena.
The Step Functions Map state executes each query on the Athena tables and outputs the results to Amazon S3 under the /Processed key.
The final file aggregator Lambda function updates the Athena output CSV files under the /Processed key with the required table names, zips up the files, and copies the zip archive to the /ProdReady key to be picked up by AdaptX’s ingestion service.
The final update processing status Lambda function updates the data set in DynamoDB as complete.

Analyzing Errors/Fall Out

Previously, AdaptX had to manually debug data issues and engage with the hospital’s data team to resolve them. This often required multiple costly and time-consuming back-and-forth communications.

With the modern architecture, the system automatically notifies the AdaptX team if it finds any errors in the uploaded files, meaning only complete and accurate data is advanced to the customer-facing solution. The validation process automatically identifies any missing columns, fields, and incorrect data types across all the files and sends any errors to an Amazon S3 bucket.

The solution uses state machines within AWS Step Functions for workflow and Lambda functions for independent tasks. The state machine orchestrates tasks to identify, filter, and communicate problematic data elements to the providers at the hospital. It then sends a comprehensive email summarizing the issue(s) to the hospital’s point-of-contact, helping hospital data analysts fix issues and resubmit the files.

Customer Benefits

This modern data architecture helps AdaptX onboard healthcare providers faster, lowering their operational cost, while improving the experience of clinical analysts. A key benefit is the flexibility to onboard with one or more topics available as part of AdaptX platform. This coalesces relevant data to empower clinicians to manage care, implement institutional changes, and measure their effectiveness in real-time.

AdaptX offers a list of filters and measures as a jumping-off point for a specific service line. Providers can further supplement with other metrics that they want to manage. With the new architecture, AdaptX provides a seamless clinician experience, requiring minimal engagement from the hospital’s IT staff. Using the serverless architecture, AdaptX collates and communicates all potential data mapping issues, thereby avoiding fragmented communications and remediation efforts.

This automation provides a near real-time data submission experience. The hospital team now enjoys a fully managed service offering from AdaptX, requiring minimal spend of time and resources.

Benefits to AdaptX

The data automation workflow operates on a “fail fast” principle. This enables AdaptX to provide healthcare providers with timely and reliable clinical data ingestion process. The automation makes it easier for hospitals to incorporate additional data as they grow their use of the solution.

The automated workflow allows AdaptX data analysts to onboard new healthcare providers faster. At the same time, the AWS workflow eliminates the need for analysts to schedule tasks and run queries.

Through this modernization, AdaptX saw improvements in their internal metrics, including:

Time savings for analyst: Time savings, especially on initial files from healthcare providers, is drastically reduced through automation and validation of data files, reducing churn, and significantly expediting time to production.
Improved clinician experience: Providing a tighter feedback loop to healthcare providers and their analysts through automated notification of errors resulted in an improved clinician experience.
Improved performance: AdaptX improved platform performance by selecting specific AWS resources with native integration and orchestration between them.
Additional insights: Integrating with Amazon CloudWatch Application Insights to give the data team a real-time and historical view of all data flow through the platform.

Conclusion

Using Amazon EventBridge, Amazon Athena, AWS Lambda, AWS Step Functions, and Amazon DynamoDB, AdaptX built a serverless solution that helps clinical leaders monitor, evaluate, and adapt care across patients, treatments, teams, and workflows to improve clinical outcomes.

AdaptX reduces the tedious and time-consuming effort spent on data validation, freeing up clinical analysts to work on challenging problems.

To learn more about how AdaptX can help uncover clinical patterns and identify measures to improve clinical care, visit their website.

.
.

AdaptX – AWS Partner Spotlight

AdaptX is an AWS Partner that provides healthcare providers with a clinical performance analytics platform that enables them to monitor, improve, and manage patient care.

Contact AdaptX | Partner Overview | AWS Marketplace

AWS Partner Network (APN) Blog