AWS Open Source Blog

Tag: HPC

Remote visualization in HPC using NICE DCV with ParallelCluster

NICE DCV is an AWS-owned high performance remote display protocol, which specializes in 2D/3D interactive streaming over the internet or a local network (e.g., WiFi). With the power of NICE DCV we can seamlessly connect to our remote session running either in the cloud or data center via internet from a local laptop. We can […]

Managing AWS ParallelCluster SSH users with AWS OpsWorks

In a previous article, we highlighted the potential for deploying a local LDAP server to provide a mechanism for managing a multi-user AWS ParallelCluster deployment with low administrator overhead. If we want our cluster users to access or manage other AWS resources, it’s preferable to control their access via AWS Identity and Access Management (IAM). […]

Using multiple queues and instance types in AWS ParallelCluster 2.9

Since its release as an officially supported AWS tool and open source project in November 2018, AWS ParallelCluster has made it simple for high performance computing (HPC) customers to set up easy-to-use environments with compute, storage, job scheduling, and networking in the cloud in one cohesive package. These clusters can cater to a wide variety […]

Managing AWS ParallelCluster SSH users with OpenLDAP

A common request from AWS ParallelCluster users is to have the ability to deploy multiple POSIX user accounts. The wiki on the project GitHub page documents a simple mechanism for achieving this, and a previous blog post, “AWS ParallelCluster with AWS Directory Services Authentication,” documents how to integrate AWS ParallelCluster with AWS Directory Service. However, […]

diagram of host machine, container, code, and datasets and checkpoints

Why use Docker containers for machine learning development?

I like prototyping on my laptop, as much as the next person. When I want to collaborate, I push my code to GitHub and invite collaborators. And when I want to run experiments and need more compute power, I rent CPU and GPU instances in the cloud, copy my code and dependencies over, and run […]

VMD over NICE DCV.

Deploying an HPC cluster and remote visualization in a single step using AWS ParallelCluster

Since its initial release in November 2018, AWS ParallelCluster (an AWS-supported open source tool) has made it easier and more cost effective for users to manage and deploy HPC clusters in the cloud. Since then, the team has continued to enhance the product with more configuration flexibility and enhancements like built-in support for the Elastic […]

How to run AWS ParallelCluster from AppStream 2.0 and share S3 data

High Performance Computing (HPC) cluster administrators typically need a way to let their users to create HPC clusters quickly and easily from a common Windows desktop, while enforcing security, isolation, scalability, and cost effectiveness. This important step could be part of a wider user workflow, or an established procedure followed by HPC users to start […]

Amazon API Gateway for HPC job submission

AWS ParallelCluster simplifies the creation and the deployment of HPC clusters. Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. In this post we combine AWS ParallelCluster and Amazon API Gateway to allow an HTTP interaction with the scheduler. […]

example of a Mach 3 external flow calculation.

Scale HPC Workloads with Elastic Fabric Adapter and AWS ParallelCluster

中文版 – In April, 2019, AWS announced the general availability of Elastic Fabric Adapter (EFA), an EC2 network device that improves throughput and scalability of distributed High Performance Computing (HPC) and Machine Learning (ML) workloads. Today, we’re excited to announce support of EFA through AWS ParallelCluster. EFA is a network interface for Amazon EC2 instances […]

AWS Parallel Cluster graphic

AWS ParallelCluster

中文版 Orchestration software has played a key role in cluster bring-up and management for decades. Dating back to solutions like SunCluster, PSSP, and community solutions such as CFEngine, the need to launch many resources together to enable large parallel applications continues to be a vital part of the High Performance Computing (HPC) environment. AWS has […]