Amazon EMR now supports Managed Scaling – automatically resizing clusters to lower cost
Amazon EMR now supports Managed Scaling, a new feature that automatically resizes your EMR cluster for best performance at the lowest possible cost, without the need to specify scaling policies. You can reduce up to 60% cost compared with fixed-size clusters by setting the minimum and maximum compute resource limits for a cluster.
Previously, you could manually scale cluster size or leverage EMR Automatic Scaling by customizing scaling rules based on CloudWatch metrics. However, these approaches require in-depth understanding of application frameworks and workloads patterns; EMR Automatic Scaling supports instance groups only. EMR Managed Scaling applies to both instance groups and instance fleets. You can seamlessly scale Spot Instances, On-Demand Instances, and instances that are part of a Savings Plan all within the same cluster.
Amazon EMR Managed Scaling is available on Apache Spark, Apache Hive, and YARN-based workloads on Amazon EMR version 5.30.1 and above. You can use this feature in the following AWS Regions: US East (N. Virginia and Ohio), US West (Oregon and N. California), South America (São Paulo), Europe (Ireland, London, Paris, Stockholm, and Frankfurt), Canada (Central), and Asia Pacific (Singapore, Sydney, Mumbai, Tokyo, Seoul, and Hong Kong).
To get started, see the following resources:
- Introducing Amazon EMR Managed Scaling - Automatically Resize Clusters to Lower Cost in the AWS Big Data Blog
- EMR Managed Scaling in the Amazon EMR Management Guide