AWS Big Data Blog
Author: Matthias Rudolph
Matthias Rudolph is a Delivery Consultant at AWS Professional Services with 8+ years’ experience in shipping production systems. From IoT platforms and data pipelines to enterprise generative AI solutions. He likes to dive into the messy middle: integrating AI into real enterprise environments with legacy APIs, security perimeters, and data quality challenges.
Run Apache XTable in AWS Lambda for background conversion of open table formats
In this post, we explore how Apache XTable, combined with the AWS Glue Data Catalog, enables background conversions between open table formats residing on Amazon S3-based data lakes, with minimal to no changes to existing pipelines, in a scalable and cost-effective way.
Run Apache XTable on Amazon MWAA to translate open table formats
In this post, we show you how to get started with Apache XTable on AWS and how you can use it in a batch pipeline orchestrated with Amazon Managed Workflows for Apache Airflow (Amazon MWAA). To understand how XTable and similar solutions work, we start with a high-level background on metadata management in an OTF and then dive deeper into XTable and its usage.