How Regeneron built a multi-executor Omics platform with intelligent resource management on AWS

Regeneron Pharmaceuticals processes thousands of genomic samples each week, using Nextflow as a core part of its scientific computing infrastructure. When its genomics workloads like ATAC-seq (Assay for Transposase-Accessible Chromatin using sequencing) outgrew the on-premises HPC cluster, the team needed a cloud solution that preserved familiar scheduling, container workflows, and shared storage while scaling on demand.

AWS ParallelCluster let the team replicate their HPC environment in the cloud without re-engineering pipelines or retraining scientists. From there they built a platform with three capabilities: a multi-executor architecture that routes each pipeline process to the best-fit compute backend, an intelligent resource manager called Haizi Bot that uses machine learning to right-size every sample, and a high-performance storage layer that ties it all together.

This post explores each capability in turn – how they work independently, and how they combine into a production system that balances cost, speed, and scientific reproducibility.

Prerequisites

This post is an architecture deep-dive rather than a step-by-step tutorial. The prerequisites below describe the environment Regeneron assembled, so you can understand the building blocks and plan an equivalent setup.

• An AWS account with permissions to create compute, storage, and networking resources in your target Region.
• AWS Identity and Access Management (IAM) roles for each execution backend – the SLURM/head-node environment, AWS Batch, and Amazon Elastic Kubernetes Service (Amazon EKS) – following least-privilege principles.
• An Amazon Virtual Private Cloud (Amazon VPC) with appropriate subnets, security groups, and network ACLs for each executor type.
• AWS ParallelCluster (v3.x or later) installed and configured, with a SLURM scheduler and a head node sized to run the Nextflow orchestrator.
• Nextflow installed on the head node, with executor configurations (slurm, awsbatch, k8s) defined via withLabel selectors.
• Singularity (or Apptainer) available on compute nodes for reproducible container execution. (Confirm the exact runtime and version your environment uses.)
• AWS Batch compute environments and job queues provisioned for containerized, embarrassingly parallel tasks.
• An Amazon EKS cluster configured for Kubernetes-native execution, including any node groups required for GPU or orchestration workloads.
• Amazon FSx for Lustre provisioned as the shared high-performance storage layer, accessible from the execution backends. Container images and reference genomes should be cached here; ensure sufficient capacity for your file-striping strategy.
• Local NVMe instance storage on compute nodes for intermediate files.
• The nf-core/atacseq pipeline and its dependencies, plus the GRCh38 reference genome and associated indexes (BWA/Bowtie2/STAR/Chromap).
• (If applicable) Access to a resource-management service and a trained model, or an equivalent mechanism for generating per-run Nextflow configurations. Regeneron uses ‘HaiziBot’, an internally-built tool for this.
• Familiarity with genomics workflow concepts (FASTQ inputs, ATAC-seq, alignment, peak calling) and basic SLURM, container, and Nextflow operation.

Solution overview

The solution architecture consists of several key components working together. AWS ParallelCluster provides the managed HPC cluster with SLURM scheduling. Nextflow orchestrates genomics pipelines and routes processes to different executors. Singularity provides reproducible container execution in the HPC environment. Internally built, Haizi Bot uses machine learning to dynamically allocate resources. And AWS FSx for Lustre delivers high-performance shared storage across all execution backends.

Before diving into each capability, let’s review the overall architecture (Figure 1).

Figure 1 – Architecture diagram showing the multi-executor genomics platform

As in the figure 1, (1) scientists submit jobs to an AWS ParallelCluster head node running Nextflow, which (2) dispatches processes to SLURM, AWS Batch, or AWS EKS based on resource labels. The three executors (3) share FSx for Lustre volumes for high-performance storage.

Now that we understand the architecture, let’s dive into the three capabilities of the platform:

• Multi-executor architecture
• Intelligent resource management with Haizi Bot
• nf-core/atacseq pipeline optimization

Multi-executor architecture

Traditional HPC migrations move an entire workload to a single compute scheduler. Regeneron took a different approach – their platform supports three execution backends from a head node which houses the pipeline framework, and individual pipeline processes can be directed to whichever backend best suits their computational profile.

The three executors are:

1. SLURM through AWS ParallelCluster: Tightly coupled, non-containerized HPC jobs with fine-grained resource control. Scales dynamically from 1 to 1,000 nodes based on SLURM queue depth.
2. AWS Batch: Embarrassingly parallel containerized tasks. Automatically handles infrastructure provisioning and scaling, with zero infrastructure to manage.
3. AWS EKS: Kubernetes-native execution for advanced container orchestration, GPU workloads, and microservices architectures.

Nextflow’s withLabel selectors make multi-executor routing declarative within a single run. A single pipeline configuration routes standard alignment processes to SLURM, memory-intensive peak calling to AWS Batch, and complex orchestration tasks to AWS EKS – without modifying pipeline code:

process {
    executor = 'slurm'
    queue = 'compute'

    withLabel: 'bigmem' {
        executor = 'awsbatch'
        queue = 'high-memory-queue'
        containerOptions = '--memory 128000'
    }

    withLabel: 'orchestration' {
        executor = 'k8s'
        namespace = 'genomics-workflows'
        pod = [nodeSelector: 'workload-type=orchestration']
    }
}

Executor selection is declarative, so a single configuration routes each step to the right backend without changing pipeline code. This lets teams adopt cloud-native services incrementally: scientists keep submitting jobs as before, while new workloads use managed services, and the platform selects the appropriate container runtime automatically.

Intelligent resource management with Haizi Bot

Genomics workflows configured with static CPU and memory limits either over-provision (wasting cost) or under-provision (causing failures). Regeneron’s Haizi Bot initiative uses machine learning to predict and allocate resources dynamically for every sample submitted, transforming how scientists interact with computational resources. Figure 2 shows the intelligence architecture.

Figure 2 – Haizi Bot intelligence architecture

When a scientist submits a sample, Haizi Bot performs the following steps before the pipeline begins executing:

1. Inspects FASTQ quality metrics, read depth, and complexity indicators to build a sample complexity score
2. Maps the complexity score to expected CPU, memory, storage, and network bandwidth requirements using a model trained on thousands of prior executions
3. Selects an execution strategy based on the user’s priority (fast turnaround or cost efficiency) and current cluster state
4. Generates a Nextflow configuration for each run, that dynamically sets memory, CPU allocation, storage provisioning, queue assignments, and scaling parameters

The system analyzes incoming samples through an ML engine and decision engine, then generates dynamic configurations choosing between cost-optimized, hybrid, and time-optimized execution strategies.

Execution strategies

Haizi Bot offers three primary modes:

Cost-optimized mode prioritizes resource efficiency. It favors Spot Instances – spare Amazon EC2 capacity on AWS offered at a discount, and implements intelligent queuing to maximize utilization, and optimizes for economical execution – delivering up to 60% cost reduction (based on Regeneron’s internal benchmarks vs. static allocation).

Time-optimized mode prioritizes speed. It provisions On-Demand resources (Amazon EC2 capacity billed per second with no commitment and no risk of interruption) for critical-path operations, parallelizes aggressively, and uses guaranteed-availability instances – achieving 30% faster completion than static allocation (based on Regeneron’s internal benchmarks).

Hybrid mode that balances across cost and time optimisations using by balancing resources using a cost-to-performance analysis.

Continuous learning

After each run, actual resource utilization feeds back into the model. Haizi Bot continuously improves through sample complexity scoring, resource usage pattern learning, cost-performance modelling, and failure pattern recognition. The system performs predictive failure detection – identifying resource exhaustion patterns before they cause job failures – and achieves 95% cost-prediction accuracy before execution begins.

Production pipeline

Regeneron’s day-to-day production workload – the nf-core/atacseq pipeline – runs on AWS ParallelCluster with SLURM scheduling and Singularity containers.

Cluster configuration

The fleet scales dynamically based on SLURM queue depth. When no jobs are pending, nodes terminate – Regeneron pays only for active computation.

High-performance storage

The implementation evolved from Amazon Elastic Block Store (Amazon EBS) and Amazon Elastic File System (Amazon EFS) to Amazon FSx for Lustre, addressing both performance bottlenecks and storage quota limitations. FSx for Lustre provides high throughput scaling linearly with filesystem size (hundreds of GB/s), sub-millisecond latency by using in-memory and SSD caching, parallel access for hundreds of simultaneous compute nodes, and optimized metadata performance (millions of ops/second) – critical for workflows generating many small intermediate files.

Regeneron uses progressive file striping: single-stripe for files under 100 MB, 8-stripe for 100 MB–10 GB, and maximum striping for large reference genomes and BAM files. Container images and reference genomes are cached persistently on FSx, while intermediate files use local NVMe storage.

Container management

Containers are pre-pulled and cached on shared storage, so the orchestrator checks the cache before pulling from a registry. This cuts pipeline startup time substantially, keeps container versions consistent, and adds resilience against registry connectivity issues.

Pipeline performance

The nf-core/atacseq workflow processes samples through seven stages: quality control (FastQC), preprocessing (Trim Galore!), alignment (BWA/Bowtie2/STAR/Chromap against GRCh38), post-processing (duplicate marking, MAPQ filtering), peak calling (MACS2 broad/narrow), differential accessibility analysis (PCA, clustering), and visualization (IGV sessions, interactive QC reports).

With Haizi Bot managing resource allocation, the pipeline demonstrates linear scaling with sample count. A typical run provisions 40–45 compute nodes with efficient recycling, delivering 30% faster completion than static allocation.

Security and compliance

The multi-executor architecture requires a unified security posture. Regeneron’s implementation includes AWS Virtual Private Cloud configuration with security groups and NACLs for each executor type, fine-grained IAM roles for each environment, end-to-end encryption for storage and data transfers, comprehensive audit logging with centralized monitoring, and automated data lineage tracking with retention policy enforcement.

Conclusion

Regeneron’s platform shows that a multi-executor architecture – running each process on the backend best suited to it – can deliver better performance, lower cost, and greater flexibility than a single-scheduler approach. Layering intelligent resource management on top turns infrastructure decisions into a “submit and forget” experience for scientists, while a shared high-performance storage layer unifies every backend.

Regeneron scientists hand off the operational overhead of high-performance computing to AWS so they can stay focused on research and discovery. Haizi Bot, an automated sizing agent, provisions every step of each sample, with no scheduler syntax to learn and no pipeline rewrites. Time-optimized runs finish 30% faster (Regeneron benchmarking), cost-optimized runs cost up to 60% less (Regeneron benchmarking), and cost predictions are 95% accurate (Regeneron benchmarking) before a run starts. Cached containers keep results reproducible and audit logging keeps governance intact.

The architecture scales automatically from 1 to 1,000 nodes and from single-sample analysis to population-scale studies, so as the team extends it, Regeneron moves faster from genetic insight to medicine.

Ready to run genomics at scale? Consult with your AWS Solutions Architects or visit the AWS HPC resources page to learn more about AWS ParallelCluster and intelligent workload management.