Artificial Intelligence
Author: Ananth Raghavendra
Ananth Raghavendra is a Senior Solution Architect at Amazon Web Services (AWS) with a focus on EC2 Spot, Graviton2, and EC2 related services such as Auto Scaling Group. Ananth has been passionate about helping customers with their cloud journey solving for their performance problems with a cost efficiency. Ananth has more than 25 years of experience in development, architecture, DevOps, and infrastructure.
Run multi-node inference workloads using Ray LLM Deep Learning Containers
Learn how to deploy one large language model across two GPU nodes on Amazon EKS using the multi-node Ray LLM Deep Learning Containers, with separate prefill and decode nodes behind one OpenAI-compatible endpoint. The pre-built container ships the full inference stack, so you configure the deployment instead of assembling it.
Simplify and support your TorchServe workloads using Ray Serve Deep Learning Containers
TorchServe is no longer maintained, leaving teams to own the entire GPU inference stack. The AWS Ray Serve Deep Learning Container is a supported, pre-tested container with the framework, GPU drivers, and serving layer already assembled. This post walks through deploying a vision-language model on Amazon EKS using the Ray Serve DLC on a single GPU node.