Posted On: May 6, 2024
Amazon EMR Serverless is a serverless option in Amazon EMR that makes it simple for data engineers and data scientists to run open-source big data analytics frameworks without configuring, managing, and scaling clusters or servers. Today, we are excited to announce detailed performance monitoring of Apache Spark jobs with Amazon Managed Service for Prometheus, allowing you to analyze, monitor, and optimize your jobs using job-specific engine metrics and information about Spark event timelines, stages, tasks, and executors.
Apache Spark provides detailed performance metrics for the driver and executors for jobs such as JVM heap memory, GC, shuffle information etc. These metrics can be used for performance troubleshooting and workload characterization. Amazon Managed Service for Prometheus is a secure, serverless, fully-managed monitoring and alerting service. With EMR Serverless integration with Amazon Managed Service for Prometheus, you can now monitor these performance metrics for multiple applications/jobs in a single view, making it easier for centralized teams to monitor these metrics to identify performance bottlenecks, historical trends etc.
This feature is generally available on EMR release versions 7.1.0 and later and in the following AWS Regions: US East (N. Virginia, Ohio), US West (Oregon), Europe (Stockholm, Paris, Frankfurt, Ireland, London), South America (São Paulo) and Asia Pacific (Tokyo, Seoul, Singapore, Mumbai, Sydney). To get started, visit the Monitor Spark metrics with Amazon Managed Service for Prometheus page in the Amazon EMR Serverless User Guide.