Resources & FAQ

Q: What does the Genomics Tertiary Analysis and Data Lakes Using AWS Glue and Amazon Athena solution do?

A: The solution creates a scalable environment in AWS to prepare genomic data for large-scale analysis and perform interactive queries against a genomics data lake. This solution demonstrates how to build, package, and deploy libraries used for genomics data conversion; provision data ingestion pipelines for genomics data preparation and cataloging; and run interactive queries against a genomics data lake.

Q: Can I modify the solution to work with my own genomics data, queries, and notebooks?

A: Yes, you can modify the solution to fit your particular needs, for example, by adding new AWS Glue jobs and crawlers to ingest, prepare, and catalog your data; and new Jupyter notebooks and Amazon Athena queries to perform data analysis. Each change will be tracked by the CI/CD pipeline, facilitating change control management, rollbacks, and auditing.

Q: What bioinformatics tools are used for data preparation?

A: This solution demonstrates how to use third-party bioinformatics tools to prepare data for ingestion into a genomics data lake. The example provided uses Hail, from the Broad Institute, to read genomics variant data in a Variant Call File (VCF) format into a Spark data frame for processing. The solution also demonstrates how to build a third-party tool, for example, Hail, from source using AWS CodeBuild and deploy it to an Amazon S3 bucket for use in an AWS Glue Job.

Q: What bioinformatics datasets are used in the solution?

A: This solution deploys the ClinVar dataset, a portion of the 1000 Genomes dataset, and an individual 1000 Genomes VCF into the solution data lake bucket. These datasets are used to demonstrate how to ingest, prepare and analyze genomics data using AWS Glue and Amazon Athena. Finally, a Jupyter notebook is provided that demonstrates how to create a drug response report from within a Jupyter notebook.

Q: Can I deploy the solution in any AWS Region?

A: No, this solution uses the AWS CodePipeline service which is currently available in specific AWS Regions only. Therefore, you must launch this solution in an AWS Region where this service is available. For the most current availability by Region, see AWS service offerings by Region.