Posted On: Apr 19, 2024
Today, we are excited to announce that customers will now be able to use Apache Livy to submit their Apache Spark jobs to Amazon EMR on EKS, in addition to using StartJobRun API, Spark Operator, Spark Submit and Interactive Endpoints. With this launch, customers will be able to use a REST interface to easily submit Spark jobs or snippets of Spark code, retrieve results synchronously or asynchronously while continuing to get all of the Amazon EMR on EKS benefits such as EMR optimized Spark runtime, SSL secured Livy endpoint, programmatic set-up experience etc.
Apache Livy is a service that enables easy interactions with the Spark cluster over the REST interface. Prior to today’s launch, customers when running their batch or interactive workloads from their local environment over a REST interface were required to either use the StartJobRun API or Interactive endpoints. This resulted in customers required to make changes to their applications. Now, with today’s launch, we have simplified the experience where customers can easily create a Livy endpoint and use that endpoint to submit Apache Spark jobs to their Amazon EMR on EKS clusters over a REST interface. Additionally, for enhanced security, customers can also restrict access to the Livy endpoint by configuring authentication through one of the supported protocols such as Kerberos, Custom Auth.
To learn more and get started, please visit our Apache Livy section documentation. Amazon EMR on EKS with Apache Livy is available with the EMR 7.1 release in the regions where Amazon EMR on EKS is currently available.