AWS Big Data Blog
Category: Thought Leadership
Powering agentic AI with real-time streaming data on AWS
Agentic AI applications now observe, reason, and act on streaming data in production. This post presents three architecture patterns that form a unified streaming backbone for the agentic AI era: streaming feature engineering with real-time inference, event-driven agent invocation, and real-time context synchronization.
Trace cascading decision failures with a blame graph on Amazon OpenSearch Service
When multiple AI agents collaborate on a decision and get it wrong, standard logs can’t tell you which agent caused it. This post shows how to build a blame graph on Amazon OpenSearch Service and Amazon Bedrock that measures influence between agents and walks backward from a failed decision to find the root cause.
Deliver Apache Kafka data to streaming tables for Apache Iceberg with Amazon MSK Express brokers
Announcing delivery to streaming tables on Apache Iceberg for Amazon MSK Express brokers, a fully managed capability that continuously materializes your Kafka streaming data as queryable Iceberg tables on Amazon S3 Tables. No connectors, Flink jobs, or custom consumers to manage, and no code to write.
Migrate from Apache Solr to Amazon OpenSearch Serverless
In this post, you will learn why now is the time to take advantage of the ease of operations and native AI capabilities of OpenSearch Serverless, and migrate from Solr.
How Stifel built a modern data platform using AWS Glue and an event-driven domain architecture
In this post, we show you how Stifel implemented a modern data platform using AWS services and open data standards, building an event-driven architecture for domain data products while centralizing the metadata to facilitate discovery and sharing of data products.
Accelerate analytics and AI innovation with the next generation of Amazon SageMaker
We are excited to announce the general availability of SageMaker Unified Studio. In this post, we explore the benefits of SageMaker Unified Studio and how to get started.
Cross-account data collaboration with Amazon DataZone and AWS analytical tools
In this post, we will cover how you can use Amazon DataZone to facilitate data collaboration between AWS accounts.
How ANZ Institutional Division built a federated data platform to enable their domain teams to build data products to support business outcomes
ANZ Institutional Division has transformed its data management approach by implementing a federated data platform based on data mesh principles. This shift aims to unlock untapped data potential, improve operational efficiency, and increase agility. The new strategy empowers domain teams to create and manage their own data products, treating data as a valuable asset rather than a byproduct. This post explores how the shift to a data product mindset is being implemented, the challenges faced, and the early wins that are shaping the future of data management in the Institutional Division.
Run high-availability long-running clusters with Amazon EMR instance fleets
In this post, we demonstrate how to launch a high availability instance fleet cluster using the newly redesigned Amazon EMR console, as well as using an AWS CloudFormation template. We also go over the basic concepts of Hadoop high availability, EMR instance fleets, the benefits and trade-offs of high availability, and best practices for running resilient EMR clusters.
How Volkswagen Autoeuropa built a data solution with a robust governance framework, simplifying access to quality data using Amazon DataZone
This second post of a two-part series that details how Volkswagen Autoeuropa, a Volkswagen Group plant, together with AWS, built a data solution with a robust governance framework using Amazon DataZone to become a data-driven factory. Part 1 of this series focused on the customer challenges, overall solution architecture and solution features, and how they helped Volkswagen Autoeuropa overcome their challenges. This post dives into the technical details, highlighting the robust data governance framework that enables ease of access to quality data using Amazon DataZone.









