I set my Amazon Elastic MapReduce cluster to archive log files to an Amazon S3 bucket, but I couldn't find the container logs in the S3 bucket at the path I specified. Why?

The Hadoop 2 log aggregation feature deletes the container log files from the local disk after uploading container log files into HDFS, so the EMR log pusher can't upload the logs to S3.

Because Hue depends on the Hadoop2 log aggregation feature, when Hue is installed on an EMR cluster, the log aggregation feature turns on automatically.

This issue had been fixed since AMI release 4.3.0; if possible, use AMI release 4.3.0 or newer.

For versions of the AMI version older than 4.3.0, disable log aggregation, or do not install Hue, if possible.

EMR, S3 logs, log aggregation

Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2016-04-29