Posted On: May 4, 2023
We are excited to announce that Amazon EMR on EKS now supports vertical autoscaling, a feature to automatically tune the memory and CPU resources of EMR Spark Applications to adapt to the needs of the provided workload, offering a simplified mechanism for customers to tune resources, enhance reliability and optimize costs. Amazon EMR on EKS enables customers to run open-source big data frameworks such as Apache Spark on Amazon EKS without having to manage application provisioning themselves.
EMR Spark allows users to configure the amount of Memory and CPU cores that it will utilize. However, tuning these values has until now been a manual process for customers that can be complex. For instance, too little memory can result in out-of-memory exceptions but allocating too much can result in over-spending on idle resources. Vertical autoscaling automatically scales the memory and CPU allocated to an EMR Spark application based on its real-time and historic resource utilization. This simplifies the process of tuning resources and optimizing costs for an application while helping improve its reliability.
To learn more about this feature, please visit the AWS Big Data Blog post: Improve reliability and reduce costs of your Apache Spark workloads with vertical autoscaling on Amazon EMR on EKS. Refer to the Using vertical autoscaling with Amazon EMR Spark jobs section of the EMR on EKS documentation for additional details.Vertical autoscaling is supported on Amazon EMR on EKS 6.10 release and later, and available in all regions where Amazon EMR on EKS is currently available.