AWS Big Data Blog
Category: Expert (400)
Use MSK Connect for managed MirrorMaker 2 deployment with IAM authentication
January 2026: This post was reviewed and updated for accuracy. MSK Connect configurations required for cross-account and same-account replication have been added to show different use cases. March 2025: This post was reviewed and updated for accuracy. MSK Replicator now makes it easier to set up cross-Region and same-Region replication without running MirrorMaker 2. Read AWS News Blog […]
Ingest streaming data to Apache Hudi tables using AWS Glue and Apache Hudi DeltaStreamer
In today’s world with technology modernization, the need for near-real-time streaming use cases has increased exponentially. Many customers are continuously consuming data from different sources, including databases, applications, IoT devices, and sensors. Organizations may need to ingest that streaming data into data lakes built on Amazon Simple Storage Service (Amazon S3). You may also need […]
Best practices to optimize cost and performance for AWS Glue streaming ETL jobs
AWS Glue streaming extract, transform, and load (ETL) jobs allow you to process and enrich vast amounts of incoming data from systems such as Amazon Kinesis Data Streams, Amazon Managed Streaming for Apache Kafka (Amazon MSK), or any other Apache Kafka cluster. It uses the Spark Structured Streaming framework to perform data processing in near-real […]