Amazon EMR Managed Scaling is now Application Master placement aware
Today, we are excited to announce a new enhancement in EMR Managed Scaling that improves application resiliency and scales the cluster based on executor and ApplicationMasters demand by adding support for Yarn Node Labels. Amazon EMR by default ensures that the processes that controls running jobs and needs to stay alive for the life of the job (ApplicationMasters) can run on both core and task nodes. However, many customers specially who use Spot Instances to run task nodes choose to run ApplicationMasters only on On-Demand core nodes to ensure running jobs do not fail if application masters running on Spot Instances are interrupted. With today’s launch, EMR Managed Scaling will now scale the clusters based on the demand for the individual AM’s or executors requests as defined by YARN node labels. Intelligently scaling the cluster based on AM's or executors demand leads to better performance, utilization and lower cost.
As part of today’s launch, with EMR release 7.2 and later, Amazon EMR will also let you specify application level YARN node labels expressions by market type i.e. On-Demand vs Spot. Previously, customers were only able to specify YARN node labels expressions by node type level i.e. Core vs Task. Now, with this new enhancement customers will have the additional flexibility to better suit the needs of your cluster workloads.
This feature is available with Amazon EMR release 7.2 and above in all the AWS Regions where Amazon EMR Managed Scaling is available. Review our Managed Scaling documentation to learn more.