1000 Genomes Project and AWS
The 1000 Genomes Project is an international research effort coordinated by a consortium of 75 companies and organizations to establish the most detailed catalogue of human genetic variation. The project has grown to 200 terabytes of genomic data including DNA sequenced from more than 1,700 individuals that researchers can now access on AWS for use in disease research. The 1000 Genomes Project aims to include the genomes of more than 2,662 individuals from 26 populations around the world, and the NIH will continue to add the remaining genome samples to the data collection this year.
The dataset containing the full genomic sequence of 1,700 individuals is now available to all via Amazon S3. The data can be found at: s3.amazonaws.com/1000genomes
Accessing 1000 Genomes Data
AWS is making the 1000 Genomes Project data publicly available to the community free of charge. Public Data Sets on AWS provide a centralized repository of public data hosted on Amazon Simple Storage Service (Amazon S3). The data can be seamlessly accessed from AWS services such Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Elastic MapReduce (Amazon EMR), which provide organizations with the highly scalable compute resources needed to take advantage of these large data collections. AWS is storing the public data sets at no charge to the community. Researchers pay only for the additional AWS resources they need for further processing or analysis of the data. Learn more about Public Data Sets on AWS.
All 200 TB of the latest 1000 Genomes Project data is available in a publicly available Amazon S3 bucket.
You can access the data via simple HTTP requests, or take advantage of the AWS SDKs in languages such as Ruby, Java, Python, .NET and PHP.
Analyzing 1000 Genomes Data
Researchers can use the Amazon EC2 utility computing service to dive into this data without the usual capital investment required to work with data at this scale. AWS also provides a number of orchestration and automation services to help teams make their research available to others to remix and reuse.
Making the data available via a bucket in Amazon S3 also means that customers can crunch the information using Hadoop via Amazon Elastic MapReduce, and take advantage of the growing collection of tools for running bioinformatics job flows, such as CloudBurst and Crossbow.
The NIH National Center for Biotechnology Information (NCBI), a division of the National Library of Medicine at NIH:
- ftp6.ncbi.nlm.nih.gov (for IPv6 access)
- 1000 Genomes : NCBI/NLM/NIH (via Aspera)
The European Bioinformatics Institute (EMBL-EBI), with support from the Wellcome Trust:
Education Grants Program
Educators, researchers and students can apply for free credits to take advantage of the utility computing platform offered by AWS, along with Public Datasets such as the 1000 Genomes Project data. If you're running a genomics workshop or have a research project which could take advantage of the hosted 1000 Genomes dataset, you can apply for an AWS Grant.