Posted On: Jun 19, 2023
We are excited to announce the addition of Volcano and Apache Yunikorn as job schedulers when running EMR on EKS using Spark operator and spark-submit. Amazon EMR on EKS enables customers to run open-source big data frameworks such as Apache Spark on Amazon EKS. Using a custom job scheduler for Spark jobs enables fine-grained capacity management and faster pod provisioning at scale.
The default Kubernetes scheduler handles the placement of individual pods, while maintaining constraints such as available capacity, resource requests and limits, and node affinity. However, it does not support scheduling based on jobs. With this new feature, customers have the option to use Apache Yunikorn and Volcano to schedule EMR on EKS Spark jobs with Spark operator and spark-submit. Customers can leverage features like gang scheduling, queue management, preemption, fair-share scheduling in these schedulers, which helps to deliver high scheduling throughput and optimized capacity.
To learn more about this feature, please visit the custom job scheduling section of our documentation. Volcano and Apache Yunikorn as Kubernetes custom scheduler are supported on Amazon EMR on EKS 6.11 release and above, and available in regions where Amazon EMR on EKS is currently available.