Category: AWS Batch
Building High-Throughput Genomics Batch Workflows on AWS: Workflow Layer (Part 4 of 4)
Aaron Friedman is a Healthcare and Life Sciences Partner Solutions Architect at AWS
Angel Pizarro is a Scientific Computing Technical Business Development Manager at AWS
This post is the fourth in a series on how to build a genomics workflow on AWS. In Part 1, we introduced a general architecture, shown below, and highlighted the three common layers in a batch workflow:
- Job
- Batch
- Workflow
In Part 2, you built a Docker container for each job that needed to run as part of your workflow, and stored them in Amazon ECR.
In Part 3, you tackled the batch layer and built a scalable, elastic, and easily maintainable batch engine using AWS Batch. This solution took care of dynamically scaling your compute resources in response to the number of runnable jobs in your job queue length as well as managed job placement. (more…)
Building High-Throughput Genomic Batch Workflows on AWS: Batch Layer (Part 3 of 4)
Aaron Friedman is a Healthcare and Life Sciences Partner Solutions Architect at AWS
Angel Pizarro is a Scientific Computing Technical Business Development Manager at AWS
This post is the third in a series on how to build a genomics workflow on AWS. In Part 1, we introduced a general architecture, shown below, and highlighted the three common layers in a batch workflow:
- Job
- Batch
- Workflow
In Part 2, you built a Docker container for each job that needed to run as part of your workflow, and stored them in Amazon ECR. (more…)
Deep Learning on AWS Batch
Thanks to my colleague Kiuk Chung for this great post on Deep Learning using AWS Batch.
—-
GPU instances naturally pair with deep learning as neural network algorithms can take advantage of their massive parallel processing power. AWS provides GPU instance families, such as g2 and p2, which allow customers to run scalable GPU workloads. You can leverage such scalability efficiently with AWS Batch.
AWS Batch manages the underlying compute resources on-your behalf, allowing you to focus on modeling tasks without the overhead of resource management. Compute environments (that is, clusters) in AWS Batch are pools of instances in your account, which AWS Batch dynamically scales up and down, provisioning and terminating instances with respect to the numbers of jobs. This minimizes idle instances, which in turn optimizes cost.
Moreover, AWS Batch ensures that submitted jobs are scheduled and placed onto the appropriate instance, hence managing the lifecycle of the jobs. With the addition of customer-provided AMIs, AWS Batch users can now take advantage of this elasticity and convenience for jobs that require GPU.
This post illustrates how you can run GPU-based deep learning workloads on AWS Batch. I walk you through an example of training a convolutional neural network (the LeNet architecture), using Apache MXNet to recognize handwritten digits using the MNIST dataset. (more…)
Creating a Simple “Fetch & Run” AWS Batch Job
Dougal Ballantyne, Principal Product Manager – AWS Batch
Docker enables you to create highly customized images that are used to execute your jobs. These images allow you to easily share complex applications between teams and even organizations. However, sometimes you might just need to run a script!
This post details the steps to create and run a simple “fetch & run” job in AWS Batch. AWS Batch executes jobs as Docker containers using Amazon ECS. You build a simple Docker image containing a helper application that can download your script or even a zip file from Amazon S3. AWS Batch then launches an instance of your container image to retrieve your script and run your job. (more…)
