HST Pathways Builds a Near Real-Time Data Warehouse for Rapid Analytics in Healthcare Management

Achieving near real-time latency with disparate data sources: In one year, HST Pathways merged with four companies with different tech stacks and products. As a result, data for their healthcare provider clients was spread across hundreds of databases, both on-premises and across three different cloud providers, making it difficult and time-consuming to get a holistic view of their data and leverage it for decision-making.

HST Pathways wanted a solution that could simplify data aggregation and storage while giving them the flexibility and performance needed to run analytics on large, multi-tenant data sets. As the healthcare industry is further empowered by new technology, HST Pathways foresees the growing demands from sophisticated customers for near real-time data analytics. Therefore, one performance requirement is to update critical data every 15 minutes in their data warehouse.

Building a data warehouse for large-scale analytics: HST Pathways collaborated with the AWS Data Lab to develop a centralized data lake from which they could build data warehouses custom-fit to the needs of their customers: individual data warehouses for their large enterprise customers and a combined data warehouse to run analytics for their customers with smaller data sets. This solution includes a low-latency pipeline that's capable of ingesting data from 100+ on-premises SQL Server tables into Amazon Redshift, a fully managed, scalable cloud data warehouse on AWS.

Using AWS Database Migration Service (DMS), HST Pathways built a near real-time solution that streams data into an Amazon Simple Storage Service (S3) data lake. Lastly, the team created an Amazon Redshift Spectrum pipeline that applies incremental updates to their Redshift aggregate tables every 15 mins to support near real-time data analytics on clinic-critical data. See architecture diagram below.

Speed, scale, and a workable data foundation: HST Pathways deployed their data warehouse solution to production within only six weeks of working with the Data Lab. The Build Lab created an environment where HST Pathways could design and build a prototype of a near real-time data warehouse, a centralized data lake, and a scalable data streaming service in only four days.

The data lake built during the lab will serve as HST Pathways’ central hub for long-term data consumption and analytics for a broad spectrum of healthcare customers.

HST Pathways also had time during their Build Lab to test out several prototype alternatives with their dedicated Data Lab Architect, helping their engineering team upskill on best practices for near-real time streaming and search that will prepare them for tackling future analytics use cases for the organization.