Track healthcare data lineage in real time with Amazon S3 annotations

Organizations that store sensitive data need to answer a simple question: where did this data come from, and what happened to it? In healthcare, this isn’t optional. Regulations like HIPAA, GDPR, and SOX require organizations to produce this information on demand. When an auditor asks, for example, “Show me every transformation that touched patient record X since January,” the answer typically requires weeks of manual investigation across disconnected logs, catalogs, and spreadsheets.

The core problem is that most lineage solutions track metadata separately from the data itself. When files move, get copied, or replicate across systems, the lineage record doesn’t follow. Over time, what the catalog says diverges from what actually happened.

Many healthcare organizations already store clinical data at petabyte scale in Amazon S3, an object storage service built for durability and scale. S3 annotations extend this by letting you attach up to 1,000 structured metadata payloads (each up to 1 MiB) directly to any object. Because annotations travel with objects during copy and replication, lineage metadata stays co-located with data, avoiding drift. Combined with S3 Metadata annotation tables and Amazon Athena, a serverless query service, annotations become queryable across your entire data lake with standard SQL.

In this post, we show you how to build automated, real-time data lineage tracking for healthcare data using S3 annotations, AWS Lambda, and AWS Glue. The solution writes lineage and consent metadata at every ETL stage, making HIPAA compliance audits and GDPR right-to-erasure requests a single query instead of weeks of manual investigation.

Solution architecture

The solution tracks electronic health record (EHR) data lineage automatically as healthcare data flows through three processing stages: raw ingestion (HL7 messages), standardization (FHIR R4 JSON), and analytics aggregation (Parquet). The following diagram illustrates the architecture.

Figure 1: Architecture showing EHR data flowing through Bronze, Silver, and Gold S3 buckets with Lambda writing lineage annotations on ingestion, AWS Glue ETL writing annotations on transformation, and S3 Metadata annotation tables enabling Athena SQL queries for compliance auditors.

Each object gets two annotations:

• lineage – FHIR Provenance-aligned schema tracking origin, transformations, and operators
• consent – Tracks which patients’ data is in the object, enabling GDPR right-to-erasure queries

Prerequisites

To deploy this solution, you need:

• An AWS account with permissions to create AWS Identity and Access Management (IAM) roles, Lambda functions, AWS Glue jobs, and S3 buckets
• The AWS Command Line Interface (AWS CLI) v2 configured with credentials
• Python 3.13 or later
• Bash shell (macOS or Linux)

The estimated cost is under $5 for the demo with sample data (10 HL7 files, two AWS Glue job runs).

Deploy infrastructure

Clone the repository and run the deployment script:

git clone https://github.com/aws-samples/sample-s3-annotations-healthcare-data-lineage.git
cd sample-s3-annotations-healthcare-data-lineage/infrastructure
./deploy.sh

This creates an AWS CloudFormation stack with the following resources:

• S3 buckets for Bronze, Silver, and Gold tiers (encrypted, versioned, public access blocked, access logging enabled)
• S3 Metadata with annotation tables enabled on each data bucket, plus a federated AWS Glue catalog for Athena
• A Lambda function triggered by Amazon EventBridge on Bronze bucket uploads
• Two AWS Glue ETL jobs (Bronze-to-Silver and Silver-to-Gold)
• An Athena workgroup for querying annotation tables
• AWS CloudTrail with S3 data events for audit logging
• TLS enforcement using DenyInsecureTransport bucket policies on all buckets

Upload data and verify lineage

The deployment script generates synthetic HL7 patient data and uploads it to the Bronze bucket. You can also upload your own test data:

aws s3 cp test_file.hl7 s3://data-lineage-demo-bronze-<ACCOUNT_ID>/ehr/test/ \
  --metadata "patient_ids=MRN1001|MRN1002|MRN1003,consent_basis=explicit_consent"

The --metadata flag passes patient identifiers as S3 user-defined metadata, which Lambda reads to construct the initial consent annotation. The annotation itself is the persistent, mutable record. For production workloads handling protected health information (PHI), consider passing patient identifiers through an Amazon Simple Queue Service (Amazon SQS) message or EventBridge event payload rather than HTTP headers.

Within seconds, the Lambda function writes both annotations. Verify the lineage annotation:

aws s3api get-object-annotation \
  --bucket data-lineage-demo-bronze-<ACCOUNT_ID> \
  --key ehr/test/test_file.hl7 \
  --annotation-name lineage \
  /dev/stdout

The response includes FHIR Provenance-aligned fields:

{
  "lineage_type": "ingestion",
  "provenance": {
    "target": "s3://data-lineage-demo-bronze-.../ehr/test/test_file.hl7",
    "recorded": "2026-07-01T05:13:09Z",
    "activity": "ingestion",
    "agent": {
      "who": "arn:aws:sts::123456789012:assumed-role/lineage-tracker-role/...",
      "role": "assembler"
    },
    "entity": {
      "role": "source",
      "what": "EHR_Epic"
    }
  },
  "data_classification": "PHI",
  "compliance_tags": ["HIPAA", "PII"]
}

The response in this post is truncated for brevity. The full annotation also includes source_system, timestamp, operator, record_count, file_format, file_size_bytes, and s3_uri fields.

Run ETL pipeline

The deployment script automatically runs both AWS Glue jobs. Bronze-to-Silver transforms raw HL7 messages into FHIR R4 format with data quality scoring. Silver-to-Gold aggregates into Patient 360 analytics views. Each job writes lineage (with provenance) and consent annotations to its output objects, including source object references for full chain traversal.

Query lineage with Athena

The CloudFormation stack enables S3 Metadata annotation tables and creates a federated s3tablescatalog in the AWS Glue Data Catalog automatically. After the annotation tables finish backfilling (a few minutes, depends on object count), you can query lineage across your entire data lake with SQL:

SELECT
  object_key,
  JSON_EXTRACT_SCALAR(text_value, '$.lineage_type') AS operation,
  JSON_EXTRACT_SCALAR(text_value, '$.provenance.activity') AS activity,
  JSON_EXTRACT_SCALAR(text_value, '$.provenance.agent.who') AS operator,
  JSON_EXTRACT_SCALAR(text_value, '$.timestamp') AS when_processed
FROM "s3tablescatalog/aws-s3"."b_data-lineage-demo-gold-<ACCOUNT_ID>"."annotation"
WHERE name = 'lineage'
ORDER BY when_processed;

GDPR right-to-erasure query

The consent annotation enables the most critical compliance query: finding all objects containing a specific patient’s data across all tiers:

SELECT
  JSON_EXTRACT_SCALAR(text_value, '$.s3_uri') AS object_location,
  JSON_EXTRACT_SCALAR(text_value, '$.data_classification') AS classification,
  JSON_EXTRACT_SCALAR(text_value, '$.consent_basis') AS legal_basis
FROM (
  SELECT text_value FROM "s3tablescatalog/aws-s3"."b_data-lineage-demo-bronze-<ACCOUNT_ID>"."annotation"
    WHERE name = 'consent'
  UNION ALL
  SELECT text_value FROM "s3tablescatalog/aws-s3"."b_data-lineage-demo-silver-<ACCOUNT_ID>"."annotation"
    WHERE name = 'consent'
  UNION ALL
  SELECT text_value FROM "s3tablescatalog/aws-s3"."b_data-lineage-demo-gold-<ACCOUNT_ID>"."annotation"
    WHERE name = 'consent'
)
WHERE CONTAINS(
  CAST(JSON_EXTRACT(text_value, '$.patient_ids') AS ARRAY(VARCHAR)),
  'MRN1234567'
)

This query returns every object in your data lake that contains data for patient MRN1234567, across all tiers, in seconds rather than weeks.

Why annotations over alternatives

The following table compares alternative approaches to using S3 annotations.

Cleaning up

To avoid ongoing charges, run the teardown script:

cd infrastructure
./teardown.sh

This empties the S3 buckets and deletes the CloudFormation stack.

Next steps

Consider the following next steps to expand the solution:

• Automated erasure workflow – The GDPR query presented in this post identifies objects, but doesn’t delete them. Use AWS Step Functions to orchestrate a workflow that receives an erasure request, runs the cross-tier query, deletes or anonymizes each object, and writes an "erasure_completed" annotation as an audit receipt. GDPR Article 17 requires completion without undue delay and within 1 month.
• Consent withdrawal automation – When a patient revokes consent through your portal or FHIR Consent resource, use Amazon Simple Notification Service (Amazon SNS) to publish a consent-change event. A subscribed Lambda function can query annotation tables to find all affected objects, update their consent annotations to reflect the withdrawal, and trigger the erasure workflow. This turns consent management from a manual process into a real-time, auditable response.
• Natural language compliance queries – Non-technical compliance officers and legal staff need answers from lineage data without writing SQL. Use Amazon Quick Sight or Amazon Bedrock Agents to translate questions like “Show me all transformations that touched patient X’s data” into Athena queries against your annotation tables.
• Lineage visualization – Auditors present data flow summaries during reviews, not SQL results. Connect Amazon Quick Sight to your Athena annotation tables to build dashboards showing transformation history, data quality scores, and processing timelines across your data lake.

Conclusion

In this post, we showed how to use S3 annotations to build automated, real-time data lineage tracking for healthcare compliance. By co-locating lineage metadata with your data, you avoid metadata drift and make compliance audits queryable with standard SQL.

The FHIR Provenance-aligned annotation schema provides standards-compliant lineage, and the consent annotation enables GDPR right-to-erasure workflows that previously required weeks of manual investigation.

To get started, deploy the sample solution and adapt the annotation schemas to your own pipeline.

For more information, see:

• Annotating your objects
• FHIR Provenance resource specification
• Amazon S3 Metadata
• GitHub repository

We’d love to hear how you’re using S3 annotations for governance and compliance. Share your feedback in the comments.