Posted On: Jun 13, 2023
We're excited to announce the ability to control container log rotation when running Apache Spark jobs in EMR on EKS. Amazon EMR on EKS enables customers to run open-source big data frameworks such as Apache Spark on Amazon EKS. Customers can now enable container log rotation to avoid excessive log files impacting pod execution.
For long running Spark jobs like Spark streaming, customers can run into low disk space issues which may cause performance degradation or job failure, and might get forced to manually delete logs from Kubernetes pods. With this release, customers can now enable container log rotation and customize how many log files will be kept in the Spark driver/executor pods, including setting the maximum size of each log file.
To learn more about container log rotation, please visit the Container Log Rotation section of our documentation. Container log rotation for Apache Spark is supported on Amazon EMR on EKS 6.11 release and above, and available in regions where Amazon EMR on EKS is currently available.