AWS announces support for Apache Iceberg V3 deletion vectors and row lineage
AWS now supports deletion vectors and row lineage as defined in the Apache Iceberg Version 3 (V3) specification. These new features are available with Apache Spark on Amazon EMR 7.12, AWS Glue, Amazon SageMaker notebooks, Amazon S3 Tables, and the AWS Glue Data Catalog.
These Iceberg V3 capabilities help customers build petabyte-scale data lakes with improved performance for data modifications and functionality to easily track changed records. Deletion vectors write optimized delete files that speed up data pipelines and reduce data compaction costs. Row lineage provides metadata fields on each record to track changes with a simple SQL query, eliminating the computational expense of finding small changes in large tables.
Get started creating V3 tables by setting the table property to 'format-version = 3' in the CREATE TABLE command in Spark or a SageMaker notebook. To upgrade existing tables, simply update the table property in metadata with the new format version. When you do this, AWS query engines that support V3 will automatically begin to use deletion vectors and row lineage.
Iceberg V3 deletion vectors and row lineage are now available in all AWS Regions where each respective service/feature—Amazon EMR, AWS Glue, SageMaker notebooks, S3 Tables, and AWS Glue Data Catalog—is supported. To learn more about AWS support for Iceberg V3, visit Apache Iceberg V3 on AWS, and read the blog post.