Posted On: Nov 20, 2023
Announcing general availability of two new capabilities to enhance monitoring and debugging of AWS Glue jobs: AWS Glue serverless Apache Spark UI and AWS Glue observability metrics. AWS Glue serverless Spark UI is a new capability that enables you to get detailed information about your AWS Glue Spark jobs. This launch allows you to see the details of any AWS Glue Spark job run in AWS Glue Studio. With AWS Glue serverless Spark UI, you can get information about scheduler stages, tasks, and executors. AWS Glue observability metrics provide additional insights into job reliability, performance, throughput and resource utilization. These two new capabilities enables you to debug common error classes with root cause, analyze aggregate worker performance, and monitor data skews in your Glue jobs.
You can use AWS Glue Studio job monitoring to view both serverless Spark UI and observability metrics. Serverless Spark UI is accessible from a new ‘Spark UI’ tab on your Glue job’s page in Glue Studio. You can also navigate to the job run monitoring page in AWS Glue Studio to view both serverless Spark UI and observability metrics. AWS Glue serverless Spark UI is fully managed and there are no infrastructure for you to manage. AWS Glue observability metrics are published to Amazon CloudWatch.
AWS Glue serverless Spark UI and observability metrics are available when using AWS Glue 3.0 and later, and is available in all commercial regions that AWS Glue is available in.
To learn more, visit our documentation on serverless Spark UI and observability metrics.