How do I troubleshoot a failed step in Amazon EMR?

Short description

Amazon EMR identifies and returns the root cause of step failures for steps submitted using the Step API operation. Amazon EMR 5.x and later also returns the name of the relevant log file and a portion of the application stack trace through API.

Note: You can use the following information to troubleshoot an Amazon EMR step of any application. For information specific to failed Apache Spark steps, see How do I troubleshoot a failed Spark step in Amazon EMR?

Resolution

Note: For descriptions of the types of step logs, see Check the step logs.

View step logs using the AWS Management console

For more information, see To view failure details using the AWS console in Enhanced step debugging.

View step details using the AWS CLI

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent version of the AWS CLI.

Use the describe-step command as shown in the following example. In the following command, replace cluster-id and the step-id with the correct values for your use case.

aws emr describe-step --cluster-id j-1K48XXXXXHCB --step-id s-3QM0XXXXXM1W

For more information, see To view failure details using the AWS CLI in Enhanced step debugging.

View step log files on the master node using SSH

For more information, see View log files on the master node.

View log files archived in Amazon S3

1.    Open the Amazon Simple Storage Service (Amazon S3) console.

2.    Select the S3 bucket specified as the S3 log URI where log files are archived.

3.    Navigate to the following path and download the log file object: cluster-id/steps/step-id/.

For more information, see View log files archived to Amazon S3.

View step logs in the debugging tool

Note: The debugging tool isn't turned on automatically in Amazon EMR. For information on turning on the debugging tool, see Turn on the debugging tool.

For information on viewing step logs in the debugging tool, see View log files in the debugging tool.

---

Related information

How do I cancel an Amazon EMR step?