AWS HPC Blog

Running accurate, comprehensive, and efficient genomics workflows on AWS using Illumina DRAGEN v4.0

This post was contributed by Shyamal Mehtalia, Principal Software Engineer 2 at Illumina, Duke Tran, Staff Bioinformatics Scientist at Illumina, Heejoon Jo, Senior Software Engineer at Illumina, Olivia Choudhury, PhD, Senior Partner SA at AWS, Jason McCloskey, Genomics Product SA at AWS, and Sujaya Srinivasan, Genomics SA at AWS.

Introduction

The reduced cost of DNA sequencing technology has led to an exponential growth of raw sequencing data. To keep pace with this development, secondary analysis tools that can provide fast and accurate results in a cost-effective manner are needed to extract actionable genomic insights. Illumina’s DRAGENTM (Dynamic Read Analysis for GENomics) addresses key pain points associated with next-generation sequencing (NGS) data analysis. These include a significant increase in sequencing data, limited on-premises compute options, inadequate CPU-based bioinformatics tools for rapid turnaround time, and high cost of secondary data analysis. DRAGEN empowers laboratories and organizations of all sizes to break through the bioinformatics bottleneck faster with accurate, comprehensive, and efficient secondary analysis pipelines. DRAGEN enables laboratories and researchers to do more with their genomic data by providing a robust suite of secondary analysis capabilities for a variety of experiment types, including whole genome, whole exome, RNA-Seq, methylome, and COVID detection pipelines.

DRAGEN’s DNA pipeline supports BCL to FASTQ conversion, alignment, sorting, duplicate marking, and variant calling. RNA and Methylation pipelines include RNA-Seq aligners, RNA-specific analysis, and downstream methylation calling. DRAGEN cloud offerings, powered by AWS, also include DRAGEN Apps on BaseSpace Sequence HubTM and DRAGEN pipelines on Illumina Connected AnalyticsTM.

In this blog, we provide a walkthrough of running DRAGEN on AWS and highlight our results from the DRAGEN v4.0 DNA sequencing pipeline showing that, in addition to accuracy and efficiency, it supports the most comprehensive suite of tools for DNA sequencing, including copy number analysis, structural variants, SMN callers, repeat expansion detection, and pharmacogenomics insights for complex genes. Results from these tools are available through a single call to DRAGEN with the appropriate command line options, streamlining results for customers of all disciplines. We also evaluate the performance of the suite with respect to key metrics: runtime, cost, and concordance.

Ways to Run DRAGEN

DRAGEN’s suite of secondary analysis pipelines are offered on a variety of platforms, enabling laboratories to select a solution that best fits their needs.

DRAGEN secondary analysis pipelines are available on Illumina Connected Analytics (ICA), a secure genomics data platform built to operationalize informatics and drive scientific insights. DRAGEN on ICA empowers customers to manage, analyze, and explore large volumes of data in a secure, scalable, and collaborative environment that couples the exceptional speed and accuracy of DRAGEN.

DRAGEN Bring Your Own License (BYOL) and DRAGEN on AWS Marketplace are available for laboratories looking to run the secondary analysis platform in their existing cloud environments. DRAGEN on AWS Marketplace is available for users looking to try out DRAGEN in a flexible environment with a premium pay-as-you-go model.

Multiple options for accessing DRAGEN allow labs to select the solution that best suits the type and scale of their projects. Regular and timely updates are made to DRAGEN across the various access points to provide performance improvements, enhancements, and additional functionality. This ensures that DRAGEN’s highly accurate, comprehensive, and efficient analysis pipelines are available and up-to-date for all users.

Pre-requisites

The pre-requisites for running Illumina DRAGEN on AWS include:

If using DRAGEN AMI in AWS Marketplace, additional pre-requisites include access to an AWS account with:

Benchmark Analysis

The DRAGEN DNA pipeline includes optimized algorithms for mapping, sorting, duplicate marking, and haplotype variant calling. It also provides lossless compression and optimized conversion of BCL files to FASTQ format. In addition to haplotype variant calling, the pipeline supports calling of copy number and structural variants, as well as detection of repeat expansions. Figure 1 presents the different processes that are available in the DRAGEN DNA pipeline.

Figure 1: DRAGEN DNA pipeline, showing FASTQ or BAM/CRAM input files being processed through a series of steps top obtain mapped reads and variant call files.

Figure 1: DRAGEN DNA pipeline, showing FASTQ or BAM/CRAM input files being processed through a series of steps top obtain mapped reads and variant call files.

For benchmark analysis, we used the Genome in a Bottle (GIAB) samples HG002, HG003, and HG004 that were obtained from the Google Brain dataset available at the AWS Registry of Open Data (RODA) at s3://genomics-benchmark-datasets/google-brain/. The samples are 30x whole genomes aligned to the hg38 alt-masked graph v2 genome hash table from Illumina.

We ran the DNA pipeline of DRAGEN v4.0 software using FASTQ files as input to perform mapping/alignment and calling/genotyping of all variants that the pipeline is capable of, including small variant, structural variant, CYP2B6, CYP2D6, GBA, SMN, PGx Star Allele, and repeat expansion detection with an expanded catalog of Short Tandem Repeat (STR) loci available beginning with v4.0. Mapping/alignment and comprehensive variant calling were performed using a single command line that runs all the pipelines in an end-to-end workflow, thereby eliminating the need for manual step-wise processing by the user. For comparative analysis, we ran the experiment on the three samples across f1.4xlarge and f1.16xlarge EC2 instances.

Launch F1 instance with DRAGEN AMI

DRAGEN is available on AWS Marketplace as an Amazon Machine Image (AMI). An AMI provides the information and pre-requisites to launch an EC2 instance. After subscribing to the DRAGEN AMI, we launched two EC2 F1 instances: f1.4xlarge and f1.16xlarge with 500 GB root volume for storage and default VPC and subnet.

As an alternative to launching the F1 instance using the DRAGEN AMI directly via SSH, customers can also use the Illumina DRAGEN QuickStart from AWS, which deploys the necessary backend infrastructure in the customer’s account, with minimal effort from the customer. The DRAGEN QuickStart, which leverages AWS Batch, allows customers to simplify the processing of 100’s or 1000’s of samples by submitting them to a job queue. AWS Batch manages the execution of the jobs, so that the customer does not need to manage any compute clusters.

Run DRAGEN DNA Pipeline

We executed the following commands to run DRAGEN’s end-to-end alignment and variant calling pipeline. This takes the paired-end compressed FASTQ files and reference hash table as input and generates a map/aligned, sorted, and duplicate-marked CRAM file and variant calls in VCF and gVCF file formats as output. We enabled flagging of duplicate output alignment records, saving of the output from the map/align stage, small variant calling, structural variant (SV) calling, copy number variant (CNV) calling in self normalization mode, CYP2D6 diplotyping, filtering of variant calls, and repeat genotyping (or expansion detection) using the corresponding flags: enable-duplicate-marking, enable-map-align-output, enable-variant-caller, enable-sv, enable-cnv, cnv-enable-self-normalization, enable-cyp2d6, vc-hard-filter, repeat-genotype-enable, respectively. Although we tested DRAGEN with BYOL option, customers can also use DRAGEN from AWS Marketplace to run these experiments.

The following is the DRAGEN command line used for the HG002 GIAB sample:

dragen --ref-dir /<REFERENCE-DIR> -1 HG002/HG002.novaseq.pcr-free.30x.R1.fastq.gz -2 HG002/HG002.novaseq.pcr-free.30x.R2.fastq.gz --enable-map-align true --enable-map-align-output true --output-format CRAM --enable-duplicate-marking true --enable-variant-caller true --vc-emit-ref-confidence GVCF --vc-enable-vcf-output true --enable-cnv true --cnv-segmentation-mode slm --enable-sv true --repeat-genotype-enable true --repeat-genotype-use-catalog expanded --enable-smn true --enable-cyp2d6 true --enable-cyp2b6 true --enable-gba true --enable-star-allele true --cnv-enable-self-normalization true --force --RGID Illumina_RGID --RGSM HG002 --output-file-prefix HG002 --output-directory <OUTPUT-DIR>/HG002/ --enable-cram-indexing true --enable-vcf-compression true --lic-server <LICENSE-CREDENTIAL>

The other GIAB samples, HG003 and HG004, were processed in a similar manner.

Results

For benchmark analysis, we ran the DRAGEN v4.0 DNA pipeline on the three GIAB datasets (HG002, HG003, and HG004), as discussed above. To compare the evaluation metrics, i.e., runtime, concordance, and cost, we conducted the experiments on two types of EC2 instances: f1.4xlarge and f1.16xlarge. The f1.4xlarge instance consists of 2 FPGAs, 8 cores, 16 vCPUs, 244 GB instance memory, and 940 GB SSD storage, whereas the f1.16xlarge instance consists of 8 FPGAs, 32 cores, 64 vCPUs, 976 GB instance memory, and 4 x 940 GB SSD storage.

Figure 2 presents the total runtime incurred by running the DRAGEN v4.0 DNA pipeline for the given dataset and EC2 instance types in the us-east-1 region. The runtime is further broken down into individual steps involved in the pipeline. As seen in the figure, the total runtime for the pipeline on f1.4xlarge is ~95 mins, and ~48 mins on f1.16xlarge. This is significantly faster than the GATK best practices pipeline [1], and includes a more comprehensive set of analysis, including all callers, copy number variants, structural variants, repeat genotyping, targeted callers CYP2B6, CYP2D6, GBA, SMN, and PGx Star Allele callers. Total runtime also included generation of quality control metrics using a hg38 graph-based reference genome. As of December 2022, the compute cost incurred in running this workload by the f1.4xlarge and f1.16xlarge on-demand instances in us-east-1 region are $6.60 and $13.20, respectively. This indicates that a majority of such workloads can be run on f1.4xlarge instances in a cost-efficient manner.

Figure 2: Runtime (mins) incurred by DRAGEN v4.0 DNA pipeline tested on GIAB samples across f1.4xlarge and f1.16xlarge EC2 instances. Note that runtimes for individual steps may be concurrent and are not necessarily additive to the total runtime as a whole.

Figure 2: Runtime (mins) incurred by DRAGEN v4.0 DNA pipeline tested on GIAB samples across f1.4xlarge and f1.16xlarge EC2 instances. Note that runtimes for individual steps may be concurrent and are not necessarily additive to the total runtime as a whole.

We also looked at the concordance of the small variants (SNPs and Indels) against the truth set from the GIAB Consortium [2]. We used the best practices outlined in [3] to compute precision, recall, and F1-score across the results obtained from all 3 samples on both EC2 instance types (Figure 3). In order to compute concordance, we used truth set VCFs from here. The hap.py tool was used with the RTG tools VCFeval engine to compute concordance as follows:

docker run -i -v `pwd`:/data pkrusche/hap.py /opt/hap.py/bin/hap.py <PATH-DRAGEN-VCF> <PATH-GIAB-BENCHMARK-VCF> -o <OUTPUT-PREFIX> -f <BED-REGIONSFILE-GIAB> -r <REFERENCE-FASTA> --engine vcfeval --engine-vcfeval-path <PATH-RTGTOOLS-VCFEVAL> --engine-vcfeval-template <PATH- RTGTOOLS-SDF-REFERENCEFILE>

Figure 3 showcases DRAGEN’s award-winning high accuracy across all three samples tested.  Furthermore, the concordance metrics are consistent across both f1.4xlarge and f1.16xlarge EC2 instances. These results show that DRAGEN’s performance is consistent and deterministic across runs, samples, and FPGA instance types.

Figure 3: Precision, Recall, and F1 score for variants (indel, SNP) tested on GIAB samples across f1.4xlarge and f1.16xlarge EC2 instances. The results show high results (greater than 99%) precision, recall, and model accuracy for both insertion-deletion and single nucleotide polymorphisms.

Figure 3: Precision, Recall, and F1 score for variants (indel, SNP) tested on GIAB samples across f1.4xlarge and f1.16xlarge EC2 instances. The results show high (greater than 99%) precision, recall, and model accuracy for both insertion-deletion and SNPs.

Cleaning Up

If running DRAGEN in your own environment, terminate all the resources created in the steps above to avoid incurring future charges.

Conclusion

The DRAGEN v4.0 DNA pipeline was run for three samples (GIAB HG002, HG003, and HG004) across two EC2 instance types (fx1.4xlarge, f1.16xlarge) after which cost, efficiency, and concordance of small variants were measured. DRAGEN v4.0 demonstrated exceptional accuracy and efficiency across samples and FPGA instance types. The DRAGEN v4.0 DNA pipeline completed analysis and QC metrics for all callers and an extensive number of additional targeted callers within ~95 mins or ~48 mins (f1.4xlarge or f1.16xlarge, respectively), representing a significantly faster and more comprehensive analysis compared to the GATK best practices pipeline [1]. High concordance was also demonstrated, with DRAGEN v4.0 consistently showing the same award-winning accuracy across runs, samples, and instances.

The content and opinions in this blog are those of the third-party author and AWS is not responsible for the content or accuracy of this blog.

Customer Success Stories

Find out more about how customers are using DRAGEN to accelerate genomic insights –

  • Munich Leukemia Laboratory – Munich Leukemia Laboratory uses DRAGEN on AWS Infrastructure to reduce turnaround time from 24 hours to 4 hours.
  • UMCCR Genomics Platform Group – DRAGEN pipelines on Illumina Connected Analytics enable the Genomics Platform Group to digitally transform their sequencing workflows.
  • Basepaws – Basepaws, a Zoetis company, leverages the Illumina DRAGEN Bio-IT Platform to develop direct-to-consumer DNA kits that provide cat owners with insights about their pets’ health.

References:

  1. Betschart, Raphael, et al. “Comparison of calling pipelines for whole genome sequencing: an empirical study demonstrating the importance of mapping and alignment.” bioRxiv (2022).
  2. Zook, J.M., McDaniel, J., Olson, N.D. et al. An open resource for accurately benchmarking small variant and reference calls. Nat Biotechnol 37, 561–566 (2019). https://doi.org/10.1038/s41587-019-0074-6
  3. Krusche, P., Trigg, L., Boutros, P.C. et al. Best practices for benchmarking germline small-variant calls in human genomes. Nat Biotechnol 37, 555–560 (2019). https://doi.org/10.1038/s41587-019-0054-x
Shyamal Mehtalia

Shyamal Mehtalia

Shyamal Mehtalia is a Principal Engineer at Illumina Inc. He is currently part of the Product Development group focusing on Customer Collaboration and Innovation, with a focus on the DRAGEN Bio-IT Platform. He drives key technical customer engagements for enabling Illumina’s secondary analysis solutions and provides support to Sales and Commercial organizations. Shyamal joined Illumina in May 2018 as part of the Edico Genome Corp. acquisition, which invented the DRAGEN platform where he served as Director of Operations for three years. Previously, he worked as a leader in firmware development in the mobile wireless space. Shyamal holds a Master’s degree in Electrical Engineering from Texas A&M University.

Duke Tran

Duke Tran

Duke Tran is a Staff Bioinformatics Scientist at Illumina Inc. He is a part of the Product Development group focusing on Customer Collaboration and Innovation with a focus on the DRAGEN Bio-IT Platform. Duke drives pipeline development and maintenance for DRAGEN and serves as a technical expert for sales and commercial organizations. Duke has over 25 years of experience in the biotechnology, medical device, and diagnostic industries. He holds a Master’s degree in Bioinformatics and Medical informatics from San Diego State University.

Heejoon Jo

Heejoon Jo

Heejoon Jo is a Senior Software Engineer at Illumina Inc. He is a part of the Product Development group focusing on Customer Collaboration and Innovation with a focus on DRAGEN. Heejoon drives pipeline development and maintenance for DRAGEN and provides technical supports for sales and commercial organizations. Heejoon has extensive experience in Cancer Genomics, applying statistical methods. He holds a Master’s degree in Biostatistics in University of North Carolina at Chapel Hill.

TAGS: , ,
Olivia Choudhury

Olivia Choudhury

Olivia Choudhury, PhD, is a Senior Partner Solutions Architect at AWS. She helps partners, in the Healthcare and Life Sciences domain, design, develop, and scale state-of-the-art solutions leveraging AWS. She has a background in genomics, healthcare analytics, federated learning, and privacy-preserving machine learning. Outside of work, she plays board games, paints landscapes, and collects manga.

Jason McCloskey

Jason McCloskey

Jason McCloskey has over 15 years of experience in software, hardware, and chemistry development with life sciences enterprises. He has worked at AWS since January of 2019. While at AWS, he has worked with large public sector partners in the areas of enterprise architecture, security, governance at scale and high-performance computing. He currently serves as Senior SA and Genomics Product Specialist supporting Illumina. Jason has a BS in Biomedical Engineering with an emphasis in Biochemical Engineering from the University of Southern California and has done graduate work in Nano Engineering, Bioinformatics and Continuum Mechanics at UCSD.

Sujaya Srinivasan

Sujaya Srinivasan

Sujaya Srinivasan is a Solutions Architect specializing in Genomics and Life sciences. She has a strong background in both technology and bioinformatics, and has more than a decade of experience working in oncology, clinical genomics and pharma. She is passionate about using technology to accelerate research and discovery in life sciences, genomics and precision medicine.