How CSIRO built scalable, cost-optimized genomic variant querying on AWS

This is a guest post by Denis Bauer, Yatish Jain, Anuradha Wickramarachchi, Brendan Hosking, and Nick Edwards of CSIRO, in collaboration with the ASP Prototyping and Scaling Team at AWS.

In this post, we describe how researchers at CSIRO, Australia’s national science agency, built Serverless Beacon (sBeacon), a scalable serverless solution for securely querying genomic variant data on AWS, underpinning production-scale clinical and research applications.

The Beacon protocol is the widely adopted standard for exchanging genomic and phenotypic data developed by the Global Alliance for Genomics and Health (GA4GH). It uses an API to define how data is shared, with the goal of enabling efficient and secure data discovery across international research and clinical networks.

sBeacon is a production-ready implementation of this standard, built using AWS services: Amazon Simple Storage Service (Amazon S3), AWS Lambda, Amazon DynamoDB, and Amazon Athena. By using these foundational AWS serverless services, sBeacon is able to provide the following benefits to researchers and clinicians needing to perform genomic variant querying:

• Highly scalable for large cohorts: sBeacon can scale to support hundreds of millions of individuals (and billions of genomic locations), which makes it suitable even for mega-biobank-scale datasets.
• Low cost to run: Because it uses a serverless, cloud-native architecture, sBeacon can operate for approximately USD 0.40 per month for a 1000 Genomes-scale dataset. The following case study breaks down ingestion, query, and storage costs in detail.
• High performance and fast query response: Real-world queries return in seconds (about 5 seconds) because of the serverless compute and efficient architecture, for near real-time data lookups.
• No heavy data ingestion or transformation needed: sBeacon can directly consume standard VCF files (a common format for genomic variant data), which reduces the need to load data into databases or transform it to different data structures.
• Rapid onboarding of new data: Genomic data generation is accelerating because it underpins clinical diagnosis and treatment, and because its complexity demands ever-larger cohorts to study complex traits. As a result, both clinical services and research cohorts must continuously onboard new data, and Beacon supports real-time generation-to-use life cycles (about 18 seconds).
• Improved privacy, data ownership, and decentralization: Because sBeacon doesn’t require central databases and supports federated networks, data stays under the control of original holders, which can help data custodians address privacy and ethical considerations in sensitive genomic and medical data sharing.
• Lower barrier to entry for broader participation: Its affordability, simplicity, and small operational footprint can help make it more accessible for smaller or resource-limited institutions and countries, which can increase participation from underrepresented populations and improve data diversity.
• Zero trust model: sBeacon enforces explicit authentication, least-privilege data access, ephemeral compute isolation, and strict cloud-native boundary controls that help confirm no component, user, or request is implicitly trusted.

Prerequisites

sBeacon is deployed as a container that sets up the necessary development environment, with Terraform defining the resources for the deployment. To get started, clone the terraform-aws-serverless-beacon repository on the GitHub website.

git clone https://github.com/aehrc/terraform-aws-serverless-beacon.git

Make sure that your development environment contains Docker and has the necessary permissions for you to use it without super user access. Press Ctrl+Shift+P (Cmd+Shift+P on macOS) to open the command palette in VS Code, and then choose Reopen in Container. This opens the workspace in the container environment that we have defined.

Now, run the following command to initialize the necessary libraries and Lambda layers.

bash init.sh

Next, run the following command to initialize the Terraform environment.

terraform init

Optionally, you can define a backend by following the instructions in the repository. After the preceding command runs successfully, you can run the deployment command.

terraform apply

Enter yes when prompted to proceed with the deployment. After the deployment is complete, you receive information such as the API URL and the command to sign in as the admin or guest user. To shut down the entire service, run terraform destroy. Any created datasets are lost (but not the VCFs on which they are based).

Solution walkthrough

CSIRO developed sBeacon for sharing and querying genomic and medical data. sBeacon uses AWS serverless technology for the elastic scaling of compute resources.

The architecture of sBeacon performs two broad processes:

1. Data onboarding: the ingestion and indexing of genomic metadata into sBeacon.
2. Data querying: the querying of the genomic metadata by end users.

Data onboarding

During the onboarding process, you define where the genomic data and the metadata (such as disease status, age, and location) is located. Note that genomic data is not copied out of its original location but rather is referenced when needed. In contrast, metadata is loaded to sBeacon’s storage mechanisms because it is necessary to perform indexing that allows efficient querying. The user will need to ensure no sensitive or privacy-revealing data is disclosed. The example details the approach using CSIRO’s Ontoserver. However, sBeacon supports the API schema of the Ensembl OLS V4 specification.

Figure 1. Data onboarding.

The data onboarding process is summarized by the following steps:

1. The onboarding starts with the user submitting the location of the genomic data as request payloads to an API Gateway endpoint.
2. The request payloads are forwarded to an AWS Lambda function that handles the data indexing.
3. The metadata is written to an Amazon S3 bucket in the ORC format, to allow future querying and processing by Athena.
4. An AWS Lambda function is called to orchestrate the indexing process.
5. The CSIRO Ontoserver is called to build the ontology index for advanced metadata queries.
6. The resulting index files are written to Amazon S3.
7. CREATE TABLE AS SELECT (CTAS) queries are run on Amazon Athena to build the metadata tables.
8. Athena loads the metadata from Amazon S3 into the metadata tables.
9. The metadata tables are written back to Amazon S3 in ORC format.

Data querying

Querying in sBeacon is flexible, catering to a wide range of applications from human genetic disease to pathogen queries. We achieved this by designing the query architecture modularly. This approach let us separate the querying logic into several Lambda functions based on their querying scope, while maintaining a similar architecture.

The following architecture diagram describes the workflow for metadata querying, which uses the Variant Querying Module described later in this section.

Figure 2. Data querying.

1. The user submits their query to the API Gateway endpoint.
2. API Gateway calls the Microservice Lambda function.
3. The Microservice Lambda function looks up the relevant query ontology terms in an Amazon DynamoDB table.
4. The matching ontology descendent terms (and their codes) are returned to the Microservice Lambda function. The descendent terms are those that match a hierarchical descendent of each term, or each term itself, from the query.
5. Using the ontology codes from step 3, the metadata tables on Athena are queried.
6. The metadata associated with the query is returned from Athena.
7. If required by the query, the Microservice Lambda function queries the Variant Querying Module.
8. The variant data associated with the genomic conditions in the query is returned to the Microservice Lambda function.
9. The result is formatted according to the Beacon protocol and is returned to the user through Amazon API Gateway.
10. The response is received by the user.

Figure 3. Variant Querying Module.

Genomic variant queries are performed using the Variant Querying Module. The workflow of this module is as follows:

1. The Microservice Lambda function calls an Initiator Lambda function.
2. The Initiator Lambda function fans out the splitQuery Lambda function across the VCF files.
3. The performQuery Lambda function is then fanned out across the VCF regions in each of the files involved in the query.
4. The performQuery Lambda function fetches the VCF files from Amazon S3.
5. The query results are synchronously returned to the parent Initiator Lambda function.
6. If requested by the user, metadata can optionally be queried, where the Initiator Lambda function queries the metadata from Athena.
7. Athena queries the metadata from Amazon S3 (through an external table).
8. The metadata results are returned to Athena.
9. The Initiator Lambda function receives the metadata from Athena.
10. All the query results, including any optional metadata, are returned to the calling Microservice Lambda function.

Case study: 1000 Genomes dataset

We demonstrate sBeacon on chromosome 1 of the 1000 Genomes Project to report how it handles large-scale variant queries. We measure ingestion efficiency, query scalability, and cost for typical population-scale analyses, such as identifying SNP variants across defined genomic regions. The case study uses chromosome 1 (chr1, 8% of the genome) from the 1000 Genomes Project, which contains 2504 samples. This multi-sample VCF is approximately 1.1 GB compressed, with data stored in Amazon S3. Note that sBeacon can also process cohorts of single-sample VCF files. All costs in this section are for the Asia Pacific (Sydney) Region (ap-southeast-2), exclude applicable taxes, and reflect pricing at the time of writing.

sBeacon can ingest chromosome 1 from the 2504 individuals in 18 seconds, for less than 1 cent (USD 0.00052). This is because sBeacon does not copy the large genomic information but instead creates index files that enable random access. Cost is therefore driven predominantly by storing the copied metadata. After ingestion, sBeacon can be maintained for USD 0.000025 per month (1 MB of compressed metadata stored for 2504 samples in ORC format, plus genomic index files). If you store the genomic data as well, this would be USD 0.032 for chr1 (at USD 0.025 per GB in ap-southeast-2) or about USD 0.425 for the whole genome.

Query time is similarly near real time. For example, querying across a region of 10,000 base pairs to determine the genotypes in this region takes 1.52 seconds across the 2504 individuals. This would serve a query such as “Fetch all individuals with a specific BRCA1 mutation who have stage 3 cancer.” The cost for such a query is USD 0.00013. Note how the query time stays constant even with an increasing number of variants returned (for example, from 4 to 400).

Table 1. Query example costing and times (whole chromosome 1).

Table 2. Cost for ingestion, querying, and idling (whole chromosome 1 for 2504 genomes with less than 10 MB of metadata).

Security features

Security and compliance is a shared responsibility between AWS and the customer. AWS is responsible for protecting the infrastructure that runs the AWS services described in this post, and you are responsible for your use of those services, including how you configure them, which identities you grant access to, and which data you choose to onboard. Consider the services you choose carefully, because your responsibilities vary depending on the services used, how you integrate those services into your IT environment, and applicable laws and regulations. For more information, see the AWS Shared Responsibility Model.

Zero trust model

• Explicit authentication and authorization – Every API request must carry a valid JWT issued by the Amazon Cognito user pool (aws_api_gateway_authorizer.BeaconUserPool-authorizer, type COGNITO_USER_POOLS). The authorizer runs at API Gateway before any Lambda function is invoked, so requests do not reach a handler without Cognito validation. Token validation includes signature, expiry, and audience (Cognito app client ID). You can disable authentication during the first deployment with BEACON_ENABLE_AUTH = false for intentionally public or open beacons. This is an explicit operator decision, not a default.

Authorization (what a valid user can do) is enforced inside the Lambda layer, not in Amazon API Gateway:

• Group membership (sbeacon-record-access-user-group, and so on) controls the maximum granularity returned.
• Admin-only operations (dataset submission, deletion) check for sbeacon-admin-group membership before proceeding.
• Least-privilege data access – sBeacon implements role-based access control (RBAC) through Cognito groups that map directly to disclosure tiers. You assign each user one or more of the following:

The JWT carries the user’s group memberships as claims. The query Lambda function reads these claims to determine requested_granularity and include_details, then passes both flags to performQuery. performQuery computes only what was requested. A boolean-tier user’s request does not cause sample-level data to be computed or returned, even if it exists in the VCF.

• Ephemeral compute isolation – Lambda execution environments are stateless by design. Each cold start is a fresh container, /tmp (1,024 MB for performQuery) is cleared between cold starts, and concurrent invocations run in separate sandboxes with no shared memory. The bcftools subprocess inside performQuery runs and exits within the Lambda function lifetime (10 second timeout). No state persists after invocation.
• Cloud-native boundary controls – API Gateway is the public entry point in this architecture. Amazon S3 buckets, DynamoDB tables, Athena, and Amazon SNS topics have no public resource policies. Amazon S3 buckets are created with private ACLs and BucketOwnerPreferred ownership controls. Lambda functions run on AWS-managed VPCs with no inbound network access. Amazon SNS topics are account-private (no external principal grants).

Privacy and data ownership

Each institution deploys the entire Terraform stack into its own AWS account, so there is no shared infrastructure, no central data lake, and no cross-account trust. VCF files live in the deploying institution’s Amazon S3 bucket and do not leave it. performQuery passes the Amazon S3 URL directly to bcftools as a subprocess argument, which uses htslib HTTP byte-range requests to read only the tabix-indexed region of interest (about 1 KB per query). The raw genomic sequence bytes do not pass through Lambda memory as returnable data. What the query returns upstream (exists as a boolean, call_count as an integer, and variant representations) is aggregate result data, not source sequence.

Decentralization in sBeacon is achieved at the storage layer, not the compute layer. The _vcfLocations registered for a dataset are Amazon S3 URIs, and these can point to buckets owned by entirely different organizations. When a query runs, performQuery passes each URI directly to bcftools, and htslib issues HTTP byte-range requests (Range: bytes=X-Y) against the Amazon S3 REST API of whichever organization owns that bucket. The raw VCF bytes do not leave the source organization’s Amazon S3 bucket. Only the query result (exists, count, or variant record) is returned.

Data onboarding privacy

The submitDataset endpoint sits behind the same API Gateway Cognito authorizer as all other endpoints. An unauthenticated request receives a 401 response before reaching any Lambda function. Beyond authentication, the handler also checks that the caller is a member of sbeacon-admin-group. A valid token from a user in only record-access or count-access is rejected. This means the beacon operator explicitly controls the set of people who can introduce data into the system, so onboarding is not a self-service capability.

Further considerations

We chose AWS Lambda over AWS Step Functions in this architecture because it can process much larger payloads. Given the size and complexity of genomic data and the fan-in and fan-out architecture for parallel handling, AWS Lambda emerged as the lower-cost and more flexible approach for this workload.

As demonstrated in the sBeacon publication, the architecture can cater to population-scale datasets. However, if you accidentally attempt to run a range query of the entire genome, the architecture times out at the Amazon API Gateway level. Applying functional operations over the whole genome requires further architectural considerations.

Because a single fan-out query spawns many parallel Lambda invocations, you need to monitor concurrency consumption to confirm that burst queries do not exhaust the account’s concurrency pool and starve other functions. Tracking the ConcurrentExecutions metric at both the account and function level provides early visibility into capacity pressure.

Similarly, because synchronous Lambda invoke does not automatically retry on throttle, a 429 response from a performQuery invocation means the result is silently lost unless the application handles it explicitly. Setting Amazon CloudWatch alarms on the Throttles metric for performQuery allows you to take corrective action, such as requesting a concurrency limit increase, before throttles affect query accuracy. Alternatively, we have produced a separate architecture that sends alert email with diagnostic information when Lambda functions fail, available in the error-catcher repository on the GitHub website. You can implement this in the repository or set it up as a standalone service to catch Lambda errors thrown by sBeacon.

After idle periods, simultaneous performQuery invocations might encounter cold starts that add latency to query responses. Enabling provisioned concurrency on the query-path Lambda functions helps reduce this cold-start latency during burst fan-out scenarios at the price of increasing the idle cost.

Conclusion

In this post, we described how CSIRO built sBeacon, a fast, scalable, and low-cost way to run genomics workloads on AWS. sBeacon implements the GA4GH Beacon standard with a fully serverless and modular architecture. This publicly available solution supports near real-time querying of standard VCF data, scales to mega-biobank cohorts, minimizes ingestion effort, and supports privacy and zero-trust security. If you are considering genomics on AWS, you can deploy sBeacon on existing Amazon S3-hosted VCF data, integrate it with clinical or research workflows through the Beacon API, and progressively federate with other Beacons for secure, cross-institutional genomic data discovery. Set up sBeacon to query your genomic data and explore the possibilities of securely sharing insights with your collaborators. You can read more about sBeacon in our publication: Scalable genomic data exchange and analytics with sBeacon. The source code for sBeacon can be downloaded from our GitHub repository.

About the authors