Amazon EventBridge integration for AWS HealthLake

Introduction

This blog post describes a feature in AWS HealthLake which allows direct integration with Amazon EventBridge. Additionally, this blog post will show how AWS HealthLake’s event-driven capabilities can be leveraged to track events at the FHIR resource levels, allowing you to address complex notifications and subscription scenarios with native AWS services. AWS HealthLake, a HIPAA eligible service, streamlines the storage, transformation, querying, and analysis of health data. Customers are using HealthLake to ingest and manage large quantities of healthcare data, but in the past had to create their own polling and external mechanisms for monitoring the progress of HealthLake events, often by using the HealthLake CLI as illustrated in this example. While this approach is straightforward, it required customers to manage the process of importing and monitoring their HealthLake data stores. This blog post will show you how to use the new HealthLake and EventBridge integration to automatically capture events in HealthLake in order to build more event-driven solutions with HealthLake.

Overview of the AWS HealthLake and Amazon EventBridge integration

AWS HealthLake generates events and makes them available via Amazon EventBridge so that customers can monitor these events to build event driven architectures. Leveraging various EventBridge features, such as event buses, rules, pipes and API destinations, customers can seamlessly integrate these events in their loosely coupled architecture. We will demonstrate how to use EventBridge with HealthLake in the following section.

Step-by-step guide for using events with EventBridge

1. Go to the EventBridge service page and navigate to Rules. Configure rules on an event bus, ensuring that all future events are directed to this event bus. If there is no rule in place to process these events, EventBridge intelligently drops those events.

2. Use a default event bus and then Create rule.

Figure 1: Create rule in EventBridge console

3. Provide a name, and optionally a description. Ensure that Rule with an event pattern is selected and proceed to the next step.

Figure 2: Provide rule details for EventBridge rule

4. In the Creation method, choose Use pattern form. Under Event pattern, select Event source as AWS services. In the AWS service dropdown choose HealthLake as the event source. You should see various event types supported by HealthLake in the event type dropdown.

Figure 3: Event types supported by HealthLake in EventBridge

5. From the Event type dropdown, select Import Job Submitted. The event pattern should automatically populate to match the events. Click Next. NOTE: This example uses a detail-type filter for monitoring Import Jobs. You can leave this off if you would like to monitor all.

Figure 4: Selecting the HealthLake event pattern in EventBridge

6. On this page, choose the target(s) to route your events to. Options include EventBridge event bus, API destination or various AWS services. For this demonstration, select CloudWatch log group, and type a Log Group name. You can add up to 5 targets per rule.

Figure 5: Selecting the EventBridge target for the rule

7. Click Next to optionally add tags, and then proceed to the Review and create screen by clicking Next again.

8. After reviewing the information, click on Create rule.

9. Once the rule is created test it by submitting an import job in HealthLake by following the service documentation.

10. After submitting the job, go to CloudWatch, Log groups and select /aws/events/healthlake-eventbridge log group used in the EventBridge rule configuration. You should observe a new entry logged by the EventBridge rule showing the event triggered by HealthLake. The entry looks like this

{
  "version": "0",
  "id": "a1b2c3d4-5678-90ab-cdef-EXAMPLE11111",
  "detail-type": "Import Job Submitted",
  "source": "aws.healthlake",
  "account": "111122223333",
  "time": "2024-01-05T19:30:17Z",
  "region": "AWS (Example 1)",
  "resources": [
    "arn:aws:healthlake:us-east-1: 111122223333:datastore/fhir/abcdef01234567890"
  ],
  "detail": {
    "jobId": "f6d73be6fa8388548ae5d0870914a0ad",
    "submitTime": "2024-01-05T19:30:17.283Z",
    "datastoreId": "abcdef01234567890",
    "inputDataConfig": {
      "s3Uri": "s3://111122223333-demo-healthlake-batch-1/Input"
    }
  }
}

Event types supported by HealthLake with EventBridge

HealthLake supports the following events with EventBridge. When configuring rules in EventBridge you can either filter on a specific event or select all events.

Data Store Active
Data Store Creating
Date Store Deleted
Data Store Deleting
Export Job Completed
Export Job Completed With Errors
Export Job Failed
Export Job In Progress
Export Job Submitted
Import Job Completed
Import Job Completed With Errors
Import Job Failed
Import Job In Progress
Import Job Submitted
AWS API Call via CloudTrail

Now that we have discussed how HealthLake makes events available in EventBridge, let’s delve into how this empowers customers to create compelling use cases.

Use Cases

Event-Based Ingestion

This new capability from AWS HealthLake provides us with several new use cases on how we can use AWS HealthLake. Prior to this feature, customers would need to poll the AWS HealthLake DescribeFHIRImportJob API in order to check the status of an import job, then when the status was completed, they would trigger a new job. While this is still a viable method for bringing data into AWS HealthLake, it is not optimized since an import job may be completed yet the polling method has not kicked off. With an event-based ingestion pipeline, it’s now possible to have EventBridge call an AWS Lambda function that immediately triggers another StartFHIRImportjob import API call. This capability allows for ingestion at scale.

Optimized Data Ingestion at Scale

AWS HealthLake has supported importing data from Amazon S3 since the service first launched. As of this writing, the AWS HealthLake service quota page outlines that AWS HealthLake supports one FHIR import job per account, where that one job can import 500GB of data from an Amazon S3 bucket (in .ndjson files), where each .ndjson file can have a max size of 5GB. In the past, we’ve seen customers use the bulk Amazon S3 import job as a way to ingest historical clinical data, and then once their historical loads are complete, they’ll switch over to using AWS HealthLake FHIR APIs in real-time (thus creating a 1:1 replica of real-time data in AWS HealthLake). When ingesting years of data, this meant several hundred jobs would need to be coordinated and managed through polling. By leveraging an event-based method through AWS EventBridge, customers can now optimize their data ingestion and reduce the time to getting value with AWS HealthLake.

Event-Based Integrations

Another way this feature can be used is when you need to trigger additional logic in your application based on the availability (or lack) of data in AWS HealthLake. For example, customers have expressed a need to trigger a series of logic or reports immediately based on the operations occurring on data within HealthLake. This new feature now provides that capability. To do so, you can simply do the following…

1. Configure your EventBridge event pattern to be the following to capture all HealthLake events.

{
  "source": ["aws.healthlake"]
}

2. In this example, simply POST and new Patient to your HealthLake endpoint.

3. Navigate to the CloudWatch log group and you will see an event with a
variety of details for the operation…

“eventSource”: “healthlake.amazonaws.com”,
“eventName”: “CreateResource”,
“awsRegion”: ” AWS (Example 1)”,
“requestParameters”: {
“resourceType”: “Patient”,
…
“additionalEventData”: { “resourceId”: ” a1b2c3d4-5678-90ab-cdef-EXAMPLE22222″ },

4. With this, you can then choose to take an action by using an AWS Lambda trigger depending on your requirements. HealthLake tells us what action, FHIR resource type, and FHIR ID were impacted in the operation.

We have also encountered scenarios where customers want to be alerted when data imports fail, often due to malformed FHIR resources. This new feature can now enable you to perform immediate data quality checks in such cases. Additionally, when meeting compliance is important for making certain data available within one business day, such as under CMS 9115F for Patient Access API, keeping logs on when data was received and ingestion into HealthLake for audit purposes can be critical. This feature allows you to choose what actions to take based on your requirements.

Cleaning up

To avoid ongoing charges, delete the resources you created with this solution:

Delete the EventBridge custom rule.
Delete the CloudWatch log group.
If you created a HealthLake datastore for this blog, delete the HealthLake datastore.

Conclusion

In this post you learned about how recent enhancements in the AWS HealthLake service help you seamlessly support event driven cases such as data ingestion at scale. AWS HealthLake enables interoperability, innovation, and ultimately, better patient outcomes. To get started with Amazon HealthLake and learn more about how you can securely store, transform, transact and analyze health data in minutes, please check following links to previous blogs. If you have any questions or concerns, leave them in the comments section.