Posted On: Mar 24, 2023
We are excited to announce support for defining where a Jupyter Enterprise Gateway (JEG) pod can be deployed when running interactive Spark workloads using managed endpoints. Amazon EMR on EKS enables customers to run open-source big data frameworks such as Apache Spark on Amazon EKS. Amazon EMR on EKS customers setup and use a managed endpoint (available in preview) to run interactive workloads using integrated development environments (IDEs) such as EMR Studio.
Until now, customers running Jupyter notebooks via managed endpoints had no control over the instance type to deploy their managed endpoint on. If a managed endpoint is deployed on a spot instance, it could run into a situation where access to Jupyter notebooks is lost via the managed managed endpoint, creating an issue customers have to work around. With this feature, customers now have full control on where the JEG pod will be deployed, including the ability to specify an on-demand instance via a managed or self-managed node group.
To learn more about this feature, please visit our documentation. JEG pod placement for managed endpoints is supported on Amazon EMR on EKS 6.10 release and above, and available in all regions where Amazon EMR on EKS is currently available.