Posted On: Feb 13, 2024
Today, Amazon EMR on EC2 announced an enhancement to the DescribeStep and ListStep API to allow customers to view up to 10,000 completed steps within the last 7 days, an increase from the previous limit of 1,000 steps. The service will also continue supporting the retrieval of 1,000 steps from the past two months. These API improvements provide greater visibility into completed steps and makes it easier for you to monitor and manage your steps workflow.
Amazon EMR is a cloud big data platform for data processing, interactive analysis, and machine learning using open-source frameworks such as Apache Spark, Presto and Trino, and Apache Flink. This improvement simplifies the experience for customers with a larger number of short running steps by providing a streamlined way to review and reconcile status of their steps to ensure proper step execution and help with debugging. If the cluster receives over 10,000 steps within a 7-day period, inactive user-submitted steps in COMPLETED, CANCELLED, or FAILED state will be removed in a chronological order, starting with the oldest ones.
This improvement is supported on all EMR releases and is available in all the AWS Regions where Amazon EMR is available. No further action is needed from your end. To learn more about viewing steps, please refer to the Viewing steps, and API DescribeStep documentation.