AWS Partner Network (APN) Blog
Run Apache Spark up to 10x faster with DataPelago on Amazon EKS
By: John Janakiraman, VP of Engineering – DataPelago Inc.
By: Hany Mostafa, Senior Solutions Architect – AWS
By: Nikhil Marrapu, Senior Solutions Architect – AWS
 |
| datapelago |
 |
Spark infrastructure costs are a significant bottleneck in modern AI and analytics pipelines, which is a direct consequence of running today’s data workloads on compute frameworks designed for earlier CPU-only architectures.
As AI workloads grow, teams need faster, more efficient ways to process data without rewriting their existing applications. Enterprises across social media, AI-powered analytics, ecommerce, and financial services have adopted DataPelago Accelerator for Spark (DPA-S) to address this challenge.
As an AWS ISV Partner, DataPelago accelerates Apache Spark on Amazon Elastic Kubernetes Service (Amazon EKS) for these customers, which means they deliver up to 10 times faster execution and up to 80% cost reduction without changes to their existing applications, security configurations, or data pipelines.
In this post, you’ll learn how customers accelerate Apache Spark workloads on Amazon EKS with DataPelago, review production performance results, and find out how to get started with a proof of concept on your own infrastructure.
Why Spark acceleration has been hard
Spark was designed for CPU-only environments. Data centers today look different. CPUs sit alongside GPUs, FPGAs, and custom accelerators. The processing power is there, but Spark can’t reach it.
The barrier is a programming model gap. Spark is built to run JVM tasks on legacy CPUs. Getting workloads onto GPUs or vectorized CPUs means rewriting application logic for a different execution model. Data engineers focus on data movement and orchestration rather than low-level processor-specific optimization. Infrastructure teams can provision GPU-enabled Amazon Elastic Compute Cloud (Amazon EC2) instances but they can’t change the Spark runtime to use them. Organizations pay for heterogeneous compute and run Spark as if the only option is a CPU.
Customers use DataPelago to close this gap with no application rewrites and no retooling needed. The system accelerates workloads without manual intervention.
How DPA-S works
DPA-S runs on DataPelago Nucleus, a universal data processing engine designed to accelerate structured and unstructured data on supported hardware. It places a domain-specific instruction set architecture between the Spark physical plan and the underlying hardware. This architecture means Spark workloads can run across CPUs and GPUs without the application knowing which processor handles which operation.
The following diagram shows the DataPelago Nucleus architecture, a single processing engine that sits between data workloads—analytics and generative AI—and the underlying hardware, supporting structured and unstructured data across CPUs, GPUs, and FPGAs:
Figure 1: DataPelago Nucleus architecture
The Spark Catalyst optimizer (the query planner that converts SQL and DataFrame operations into an executable plan) produces a physical plan as it normally does. DataPelago changes none of it. Nucleus intercepts at the physical plan stage. It evaluates every operation for acceleration, including scans, filters, joins, aggregations, and sorts. For each operation, Nucleus determines at runtime which processor will be fastest based on the data. Some operations route to GPU; others stay on CPU with vectorized execution. If a stage can’t be accelerated, Nucleus falls back to standard Spark automatically with no errors. Cluster topology, data formats, security, and monitoring remain untouched. The following diagram illustrates this architecture:
Figure 2: CPU and GPU infrastructure
Performance beyond cuDF
As Jensen Huang noted at COMPUTEX, “Data processing consumes the vast majority of cloud spend today. All of it should be accelerated.” When benchmarked against NVIDIA cuDF, the GPU-accelerated library powering the Spark RAPIDS Accelerator, DataPelago Nucleus demonstrated measurable performance gains across key operation types.
The DataPelago team ran the benchmark comparison on an AWS P5.48xlarge instance (192 vCPUs, 2 TB RAM, and 8x NVIDIA H100 GPUs). The team conducted measurements using DataPelago Nucleus v3.0 and cuDF v25.04.00. The benchmark covered a suite of physical operators that form the building blocks of AI and analytics workloads. Here are four of the key results:
- Projection and filtering – With 3.7–10.5 times higher throughput than cuDF. Nucleus fuses multiple Spark operations into single GPU kernels, reducing the extra kernel launches and memory copies that limit cuDF on integer-heavy workloads.
- Aggregation – 3.3–4.5 times higher throughput across the tested data types, including float32 and float64 operations common in financial and scientific workloads.
- Memory-intensive operations – Up to 4.5 times faster on hash joins and aggregations. Nucleus manages GPU high-bandwidth memory dynamically, avoiding the spill to CPU cycles that degrade cuDF on large datasets.
- Variable length string processing – Up to 38.6 times faster for hash joins on text data. This performance gap is particularly relevant for generative AI pipelines, where unstructured text is the dominant data type. Nucleus uses purpose-built string hashing that keeps performance GPU in-house in cases where cuDF might fall back to slower execution paths.
For AWS customers running GPU instances, these gains translate to significantly more useful work from the same hardware.
Deploying on AWS
Customers run DataPelago on AWS for self-managed Apache Spark on Amazon EKS. DataPelago is deployed as a Spark plugin during cluster setup, requiring no data migration, application rewrites, or application recertification. For deployment instructions, see the DataPelago deployment guide for Amazon EKS.
The integration works at the executor level. DataPelago adds a pre-execution hook that decides which stages to accelerate. Spark drivers, resource managers, monitoring, and scheduling continue as before. The only change: Jobs finish faster on fewer Amazon EC2 resources.
Data stays in Amazon Simple Storage Service (Amazon S3) in Parquet, Delta, ORC, or other standard formats, fully compatible with Spark’s built-in authentication and encryption. According to DataPelago, the solution is SOC 2 Type 2 and ISO 27001 certified. DataPelago doesn’t access, copy, or move customer data, and the solution avoids vendor lock-in.
You can procure DataPelago in AWS Marketplace for streamlined billing. For guidance on running Spark workloads on Amazon EKS, refer to this Containers Blog post: Best practices for running Spark on Amazon EKS.
Production results across industries
Customers across social media, AI-powered analytics, ecommerce, and financial services have adopted DataPelago to accelerate their Spark workloads. In production, customers have achieved 2–10 times faster performance and 50–80% lower costs. In micro-benchmarks, Nucleus reaches up to 60 times faster speedup on memory- and compute-intensive operations such as joins, aggregations, and sorts.Four outcomes are consistent across these deployments:
- Performance gains 2–4 times faster in production – DataPelago accelerates compute- or memory-intensive operations such as joins, aggregations, and sorts by routing them to optimized execution paths on CPUs or GPUs. Customers running daily extract, transform, and load (ETL), enrichment, and model validation pipelines have reduced the number of vCPU hours needed to complete these jobs on the same cluster configurations.
- Cost reduction of 50–80% – Faster job completion means fewer compute hours consumed per workload. Because DataPelago reduces the active runtime of Spark jobs without changing cluster size, customers pay for less compute time on the same EC2 instances. Some customers have also been able to downsize their clusters while maintaining equivalent or better performance, further reducing costs.
- Seamless deployment at scale – In enterprise environments with diverse workloads where legacy approaches caused Spark jobs to fail during runs, DataPelago’s Smart Acceleration helped prevent job failures by intelligently reverting individual queries and stages to built-in Spark when acceleration is not possible. This alleviated the all-or-nothing risk of full job
acceleration and confirmed teams depending on these jobs had data available on schedule - Production deployment in hours to days – DataPelago deploys as a Spark plugin (JAR file) with no code changes, no data migration, and only minimal configuration settings. Most customers complete a proof of concept in 3–4 weeks and move to production shortly after.
Getting started
DataPelago Accelerator for Spark is available now in AWS Marketplace.
You can deploy DataPelago into your existing Spark on Amazon EKS environments. You select the workloads, complete benchmarks with and without DataPelago’s accelerator, and measure the difference. Most customers finish a proof of concept in 3–4 weeks. Deployment itself takes hours to days.
To get started, explore DataPelago in AWS Marketplace or contact the team to discuss a proof of concept and measure your Spark acceleration on your own workloads.
Visit DataPelago to explore how DataPelago can accelerate your Spark workloads and how to request a proof of concept.
DataPelago – AWS Partner Spotlight
DataPelago is an AWS Advanced Partner and AWS Data & Analytics Competency Partner that that accelerates Apache Spark and other open source frameworks for generative AI and analytics workloads. DataPelago Accelerator for Spark deploys as a plugin into existing Spark environments with zero code changes, no data migration, and no disruption to security policies, delivering up to 10x faster execution and up to 80% lower compute costs across CPUs and GPUs.