Posted On: Jun 24, 2021
Amazon EMR on Amazon EKS announced support for Custom Images, a new capability that enables customers to customize the Docker container images used for running Apache Spark applications on Amazon EMR on EKS. Custom images enables you to install and configure packages specific to your workload that are not available in the public distribution of EMR’s Spark runtime into a single immutable container. An immutable container promotes portability and simplifies dependency management for each workload and enables you to integrate developing applications for EMR on EKS with your own continuous integration (CI) pipeline.
Until today, EMR on EKS enabled you to provide your application dependencies in a just in time fashion where you submit a dependency from an external storage service such as Amazon S3 as part of job submission. Now, with custom image support, you can create a self-contained docker image with the application and its dependencies for each use-case. For example, you can create a custom image for data engineers that includes a specific Java version and certificates required by the application, and a separate image for data scientists that includes different dependencies such as proprietary libraries or specific Python dependencies. Data engineers and data scientists can then use their application specific custom image in EMR on EKS jobs. With the new functionality, you no longer need to maintain, update or version externally stored libraries, and your big data applications can be developed using the same DevOps processes that your other containerized applications are using.
Using a custom image for your EMR on EKS job is easy. You can specify the custom image that you want to use in your job using the parameter spark.kubernetes.container.image in your Start-Job-Run API. You can also use the parameters spark.kubernetes.driver.container.image and spark.kubernetes.executor.container.image to specify a different image for Spark driver and executor pods.
To get started, see our AWS News launch blog and our overview demo video. Custom image is supported for all Amazon EMR on EKS releases and is available in all regions where Amazon EMR on EKS is available. Additionally, you can learn more by reading our documentation or visit the Amazon EMR on Amazon EKS detail page.