Scale distributed AI workloads with Ray and Anyscale on Amazon EC2

If you are building multimodal artificial intelligence and machine learning (AI/ML) pipelines on Amazon Elastic Compute Cloud (Amazon EC2) that process text, video, and audio, you have likely hit at least one of these challenges. Scaling from a single instance to distributed compute without rewriting code, managing costs across training, fine-tuning, and inference, and keeping up with an evolving network of models, frameworks, and hardware are among the biggest challenges AI engineers face today. These challenges compound when each pipeline stage has distinct compute, networking, and storage requirements, and multiple teams own separate parts of the workflow.

This post shows how the Ray-powered Anyscale application on AWS helps you tackle these challenges. You will learn how to: deploy Ray across three AWS compute options based on your operational requirements, optimize costs by matching heterogeneous instance types to each pipeline stage, and scale from prototype to production without code changes.

How Anyscale builds on Ray

Ray Core is the distributed computing foundation underlying the entire framework. It provides primitives for parallelizing Python workloads across clusters, handling task scheduling, fault tolerance, and autoscaling. Ray AI libraries are built on top of Ray Core. As a result, custom workloads that don’t fit a standard pattern can still use the same distributed infrastructure.

Ray Data ingests and transforms datasets at scale, feeding processed data into training, tuning, and inference stages. Ray Train orchestrates distributed multi-node jobs with built-in fault tolerance, supporting foundation model fine-tuning across architectures such as Llama and Mistral with automatic selection of data-parallel, model-parallel, or pipeline-parallel strategies. Ray Tune parallelizes hyperparameter experiments across large clusters. Ray Serve runs production inference endpoints with dynamic batching, automatic scaling, and multi-model deployment support. Ray RLlib supports distributed reinforcement learning workloads, including reinforcement learning from human feedback (RLHF) for large language model (LLM) alignment.

Anyscale on AWS

Many AWS customers already run self-managed Ray clusters on Amazon EC2. Anyscale builds on that foundation as a fully managed service that deploys within your AWS account, integrating with your existing Amazon Virtual Private Cloud (Amazon VPC) networks, AWS Identity and Access Management (IAM) roles, and compliance frameworks. The application handles cluster management, autoscaling, and cost controls so you can focus on workloads rather than infrastructure.

AWS and Anyscale together address three dimensions of production AI/ML: cost-efficient infrastructure spanning the full Amazon EC2 portfolio, a Ray-on-Anyscale stack that scales reliably from experimentation to production, and an integrated framework that reduces friction at every stage of development. With Anyscale, you can map each pipeline stage to the appropriate instance type. Intel and AMD-based instances handle preprocessing, tokenization, and feature extraction. AWS Graviton-based instances cover preprocessing, worker nodes, embedding generation, and lightweight inference where price-performance matters. NVIDIA GPU instances and AWS Trainium handle training and fine-tuning. AWS Inferentia serves high-throughput inference endpoints. Amazon EC2 Spot Instances run automatically for batch training, preprocessing, and hyperparameter sweeps.

Anyscale deployment architecture on AWS

Three deployment options provide flexibility based on operational requirements:

Amazon EC2 with Elastic Fabric Adapter (EFA) provides direct compute control and high-performance inter-node networking.

Amazon Elastic Kubernetes Service (Amazon EKS) with EFA supports containerized workloads.

Amazon SageMaker HyperPod delivers managed large-scale foundation model training with built-in cluster resiliency.

Regardless of the deployment path, Ray libraries handle distinct pipeline stages. Ray Data orchestrates preprocessing across CPU-only instances, AWS Graviton-based instances, GPU instances, and Spot Instances. Ray Train, Ray Serve, and Ray RLlib handle distributed training, inference serving, and reinforcement learning on NVIDIA GPU instances and AWS AI chips.

Figure 1: Anyscale on AWS architecture

Anyscale runs a hosted control plane that sends instructions to an operator deployed in your AWS environment. The data plane (Amazon EC2 instances, storage, and networking) stays in your account, and Anyscale manages Ray cluster lifecycle without direct access to your data.

Option A: Amazon EC2

Amazon EC2 supports the full pipeline without constraint. This makes it the recommended default for teams running EC2-based workloads who want to scale into AI/ML without retooling their operations. Ray Data runs preprocessing on cost-efficient instance types including Spot Instances and Graviton processors. Ray Train handles distributed training on GPU and Trainium instances with EFA. Ray Tune runs hyperparameter search across CPU and GPU instances. Ray Serve deploys inference on Inferentia or Graviton. Ray supports reinforcement learning through RLlib on GPU instances, and RLHF workflows run through frameworks such as OpenRLHF.

Anyscale manages instance lifecycle, provisioning and terminating nodes based on demand. Amazon Simple Storage Service (Amazon S3) handles datasets, checkpoints, and model artifacts. Amazon FSx for Lustre provides parallel file system throughput for training jobs with repeated large-dataset reads. EFA is available on supported instance types for multi-node training.

Option B: Amazon EKS

Amazon EKS runs the same full pipeline within a Kubernetes environment. This makes it the right choice if your organization already manages Kubernetes and wants Ray to share the same infrastructure and tooling. The Anyscale operator deploys Ray nodes as pods, and Ray libraries run identically to the Amazon EC2 path at the application layer.

Karpenter handles node autoscaling, provisioning GPU nodes for training, Graviton nodes for preprocessing and inference, and Spot Instances for batch workloads. Ray Serve benefits from Kubernetes-integrated zero-downtime upgrades for production inference deployments. The tradeoff: Amazon EKS adds container orchestration complexity and requires custom configuration for the Anyscale operator.

Option C: Amazon SageMaker HyperPod

Amazon SageMaker HyperPod is purpose-built for resilience-critical stages where multi-week runs on large GPU or Trainium clusters make node failures expensive. It continuously monitors cluster health, automatically replaces unhealthy nodes, and resumes from the last checkpoint without manual intervention. HyperPod supports both Slurm-orchestrated and Amazon EKS-based clusters, and now supports the full model development lifecycle including inference deployment. Amazon SageMaker Studio integration adds experiment tracking and model management.

Native integrations shared by EC2, EKS, and HyperPod

Networking and storage

EFA provides low-latency, high-bandwidth inter-node communication for distributed training on all three deployment paths. Amazon S3 is the primary store for datasets, checkpoints, and artifacts. Ray Data pipelines read directly from Amazon S3-based data lakes. Amazon FSx for Lustre integrates with S3 for parallel file system throughput. Amazon Elastic File System (Amazon EFS) covers shared file access for workspaces and notebooks.

Security and compliance

Amazon Virtual Private Cloud (Amazon VPC) network isolation, IAM role-based access control, and AWS Key Management Service (AWS KMS) encryption at rest and in transit apply regardless of deployment path. Data stays within your AWS account and encryption keys remain under your control. This architecture supports compliance with frameworks including HIPAA and SOC 2 without changes to Ray workload code.

Observability

Anyscale monitoring provides workload-specific dashboards covering training progress, pipeline throughput, and request latency. Grafana dashboards are accessible from the Anyscale console. Amazon CloudWatch handles AWS metrics, logs, and alarms. Prometheus supports custom metrics collection for teams with existing monitoring pipelines.

Each deployment path follows the same workflow: Ray Data feeds processed data to Anyscale Runtime, Ray Train orchestrates training, Ray Serve handles inference through Amazon API Gateway, and Ray RLlib supports reinforcement learning.

Cost optimization and provisioning

Right-sizing compute per stage

AWS Graviton-based instances deliver strong price-performance for CPU-bound Ray workloads across preprocessing, embedding generation, and lightweight inference. PyTorch inference on Graviton3 C7g instances shows up to 50% cost savings versus x86 equivalents (Tomaz Berisa, n.d.). Graviton4 adds 30% better compute per core and 75% more memory bandwidth than Graviton3 (AWS, n.d.). Graviton also runs small language models (1–7B parameters) cost-efficiently for generative AI inference.

Intel-based instances provide strong x86 price-performance for tokenization, batch transforms, and workloads requiring broad framework compatibility. AMD-based instances deliver strong performance for CPU-intensive preprocessing, feature engineering, and inference workloads requiring maximum throughput per core. NVIDIA GPU instances provide high throughput for distributed training, fine-tuning, and large-model inference.

AWS Trainium2 delivers 30–40% better price-performance than comparable GPU instances for training at scale (AWS, n.d.). AWS Inferentia2 provides up to 4x higher throughput and 10x lower latency than its predecessor for production inference. Ray auto-detects accelerator types, so Anyscale routes each workload stage to the appropriate hardware without manual configuration.

Spot Instances with Ray fault tolerance

Anyscale keeps head nodes on On-Demand and runs workers on Spot with automatic fallback. Ray Data retries failed tasks on replacement nodes. Ray Train resumes from checkpoints, achieving up to 60% savings on distributed training. Ray Serve proactively migrates replicas on the 2-minute preemption warning.

Capacity Blocks and On-Demand Capacity Reservations

For scheduled training runs requiring capacity availability, Amazon EC2 Capacity Blocks reserve 1–64 accelerated instances for up to six months, bookable eight weeks in advance. Each block supports up to 512 GPUs or 1,024 Trainium chips, placed in Amazon EC2 UltraClusters with petabit-scale networking. Anyscale machine pools reference capacity reservations directly with priority-ordered fallback: Capacity Block, then Spot, then On-Demand. On-Demand Capacity Reservations (ODCRs) provide open-ended capacity availability.

Combined provisioning strategy

Anyscale runs two-layer autoscaling: Ray Serve scales replicas on request metrics while the cluster autoscaler provisions and terminates nodes. The recommended purchasing stack starts with AWS Compute Savings Plans, saving up to 66% across Graviton, GPU, Trainium, and Inferentia. Layer on Spot for fault-tolerant batch jobs, Capacity Blocks for scheduled training, and On-Demand for burst.

Getting started: From prototype to production

Your path follows four phases, each building on the last without code changes:

Phase 1: Validate locally (1–2 days). Start with Ray on a single Amazon EC2 instance or your laptop. Build your data pipeline and training loop, and validate model quality before touching distributed infrastructure.

Phase 2: Scale data (1–2 weeks). Deploy Ray Data with AWS Graviton and Amazon EC2 Spot Instances to process your full dataset at scale, with Spot Instances reducing preprocessing costs by up to 90% (AWS, n.d.).

Phase 3: Scale training (2–4 weeks). Choose your deployment option: Amazon EC2 for maximum flexibility, Amazon EKS for Kubernetes-native environments, or Amazon SageMaker HyperPod for resilience-critical runs. Scale training across multiple GPUs or AWS Trainium instances with EFA.

Phase 4: Deploy inference (1–2 weeks). Deploy Ray Serve for model serving with autoscaling configured to your traffic patterns.

The same code runs across all four phases. Moving from a single instance to a multi-node cluster requires only configuration changes managed by Anyscale.

Conclusion

Ray and Anyscale on AWS provide a practical path from prototype to production across the full AI/ML pipeline. Three deployment options address different operational requirements: Amazon EC2 for maximum flexibility, Amazon EKS for Kubernetes-native environments, and Amazon SageMaker HyperPod for resilience-critical training. With heterogeneous clusters mixing Amazon EC2 Spot Instances, AWS Graviton, NVIDIA GPUs, AWS Trainium, and AWS Inferentia, you can match compute to each pipeline stage. The same Ray code runs from a single instance to a multi-node cluster without modification, and Anyscale handles cluster management, fault tolerance, autoscaling, and resource orchestration.

Your next steps depend on where you are today:

• Already running Ray on AWS – deploy Anyscale through the AWS Marketplace for managed autoscaling, cost controls, and governance.
• New to distributed ML – start with the Anyscale documentation on a single Amazon EC2 instance, then scale out.
• Evaluating Anyscale – explore the Ray quickstart for architecture guides and tutorials.

For more on the AWS services covered in this post, see the documentation for Amazon EC2, Amazon EKS, Amazon SageMaker HyperPod, AWS Trainium, AWS Inferentia, and AWS Graviton.

About the authors