Getting started with Amazon EMR

Amazon EMR allows you to process vast amounts of data using open-source tools such as Apache Spark, Hive, Flink, Trino, and more. Simply choose your preferred EMR deployment model:

• EMR Serverless: Run applications without managing clusters and automatically scale resources up and down based on your workload
• EMR on EC2: For control over cluster configuration, including instance types and custom AMIs.
• EMR on EKS: Consolidate analytics with your other Kubernetes-based applications on a shared Amazon EKS cluster.

Amazon EMR supports a wide range of frameworks and languages, allowing you to build everything from standard ETL pipelines to large-scale generative AI data preparation.

Languages: Use Python (PySpark) for data science and machine learning, SQL (via Hive or Trino) for analytical queries, or Java and Scala for high-performance Spark applications.

Frameworks: Build and run applications using Apache Spark for large-scale data processing, Apache Flink for real-time streaming, Trino for fast SQL across multiple data sources, and Apache Hudi or Iceberg for managing transactional data lakes.

To begin processing, your data must be accessible to Amazon EMR. While Amazon S3 is the standard storage layer for EMR applications, you have several high-speed methods to move data from your local environment or other AWS services.

• Direct Uploads: For immediate processing, upload objects directly to Amazon S3 using the AWS Management Console, CLI, or SDKs.
• High-Speed Connectivity: Use AWS Direct Connect to bypass the public internet and establish a private, dedicated network connection from your data center to AWS. This provides consistent bandwidth and reduced latency for large-scale transfers.
• Real-Time Streaming: Use Amazon Data Firehose or Amazon Managed Streaming for Apache Kafka (MSK) to feed data directly into your EMR applications as it is generated, enabling near real-time analytics.
• Zero-ETL Integrations: Analyze data from Amazon Aurora or Amazon Redshift using Zero-ETL features, which allow EMR to access operational data without the need for manual pipeline construction.
• Hybrid Access: If your data resides in a local Hadoop HDFS environment, you can use the S3 Connector to read data directly into EMR or sync specific datasets for cloud-based processing.

Amazon EMR offers a streamlined deployment experience, whether you are running a one-time job or a continuous production pipeline.

• Launch via EMR Studio: Open your EMR Studio notebook and attach it to a Serverless Application or an existing EC2 Cluster. With one click, you can execute your Spark or Hive code in a fully managed environment.
• Serverless: If using EMR Serverless, submit your job via the console, CLI, or API. EMR automatically provisions the exact compute and memory needed, scaling up to handle peaks and down to zero when finished.
• Launch via SageMaker Unified Studio: Within SageMaker Unified Studio, you can open a serverless notebook and instantly connect it to an EMR Serverless application or an EMR on EC2 cluster. 

EMR provides visibility into your data pipelines with built-in tools that help you identify bottlenecks and optimize costs automatically.

Monitor job progress and cluster health through the EMR Management Console, AWS CLI, or SDKs. EMR provides native integration with Amazon CloudWatch for real-time metrics, logs, and automated alerting.

Access the live and persistent Spark UI or Tez UI directly from the console—debug live jobs in real time and even after a serverless job is finished—to review execution plans and DAGs (Directed Acyclic Graphs).