How Vanderbilt University scales digital archive discovery with Amazon S3 Metadata

Managing massive digital collections is hard. When you’re preserving decades of historical content and adding new materials daily, making that content discoverable matters more than the storage itself. Vanderbilt University Library discovered this firsthand while managing their extensive digital archives, including the renowned Vanderbilt Television News Archive (VTNA).

Amazon S3 Metadata accelerates data discovery by automatically capturing metadata for objects in your general purpose buckets and storing it in read-only, fully managed Apache Iceberg tables that you can query. When you add, update, or remove objects in your buckets, S3 Metadata automatically refreshes the corresponding metadata tables to reflect the latest changes.

In this post, we explain how Vanderbilt University uses S3 Metadata to organize, query, and accelerate discovery of data across their growing digital collections, enabling researchers worldwide to access critical historical resources more efficiently.

The challenge: Finding content in a 220-terabyte archive

The Vanderbilt Television News Archive, established in 1968, houses the world’s most extensive collection of broadcast news, used by the U.S. Library of Congress and researchers globally. By 2018, the archive had grown to 200 TB, expanding by 4 TB annually as staff captured 3.5 hours of content every night.

Storage was only part of the equation. Library staff faced a four-year backlog in creating abstracts for broadcasts and needed to restructure how they stored and accessed assets. As Vanderbilt expanded beyond VTNA to include other digital collections, like the Slave Societies Digital Archive chronicling the trans-Atlantic slave trade, efficient metadata management became critical.

Query metadata for intelligent discovery

Vanderbilt migrated their digital archives to Amazon S3 for durable, scalable storage. With S3 Metadata enabled, the library can now query across all their objects without building or maintaining a separate metadata system.

Understanding S3 Metadata tables

Every object in S3 carries metadata that describes it. There are four types of object metadata in S3:

• System-defined metadata — automatically captured by S3, including an object’s creation time, size, storage class, and encryption status
• Custom metadata — custom key-value pairs set at upload time, such as a department name or project code
• Object tags — key-value labels you can add, modify, or delete at any time, integrating with IAM policies, lifecycle rules, and cost allocation
• Annotations — rich, large-scale business context attached to any object at any time, supporting formats like JSON, XML, and YAML

S3 Metadata brings all four types together into queryable, fully managed Apache Iceberg tables. S3 Metadata provides three table types:

Journal table (required) — The journal table captures object-level changes—uploads, deletions, and metadata updates—in near real-time. Each metadata table configuration requires a journal table, which records changes from the moment you create it. Because this table is queryable, you can audit changes to your bucket through simple SQL queries. To minimize storage costs, you can enable journal table record expiration (minimum 7-day retention).

Live inventory table (optional) — The live inventory table provides a queryable inventory of all objects and their versions in your bucket. When you enable it, S3 performs an initial backfill that scans existing objects. This process takes a minimum of 15 minutes, with larger collections (by object count) taking hours. After backfilling completes, updates are typically reflected within one hour. AWS charges for backfilling; buckets with more than one billion objects also incur a monthly fee.

Annotation table (optional) — The annotation table tracks the latest annotations on objects in your bucket and makes annotation content directly queryable. Each row represents a single annotation on a specific object version. Use the annotation table to discover and retrieve objects based on annotation values. For example, you can query it to find all objects with a specific annotation key, retrieve annotation content matching a predicate based on object size or key prefix, or correlate annotations across objects.

S3 stores the metadata tables in a managed table bucket named aws-s3, with S3 handling all maintenance tasks including compaction and snapshot management. To query your metadata, integrate your table bucket with Amazon SageMaker Lakehouse, which uses the AWS Glue Data Catalog to make your tables discoverable. Optionally, use AWS Lake Formation for fine-grained access control. You can then query using Amazon Athena, Amazon EMR, Amazon Redshift, or open-source tools like Apache Spark, Trino, and DuckDB.

Real-world applications at Vanderbilt

Accelerate content discovery across historical archives

With S3 Metadata tables, Vanderbilt can now query metadata across their entire 220 TB archive without scanning individual objects. Researchers quickly identify broadcasts by date range, topic, or custom metadata tags, reducing search time from hours to under a minute.

For example, Vanderbilt uses Athena to analyze how their collections grow over time. The following query aggregates objects by month to show the number of items and total size added to a collection:

SELECT
    date_trunc('month', last_modified_date) AS month,
    COUNT(*)                                AS number_of_items,
    SUM(size)                               AS total_size,
    ROUND(SUM(size) / 1073741824.0, 4)      AS total_size_gb
FROM "b_ltp-scua"."inventory"
WHERE is_delete_marker = FALSE
GROUP BY date_trunc('month', last_modified_date)
ORDER BY month ASC;

This gives the library a clear view of collection growth trends, helping them plan storage capacity and report on acquisition rates. The results can be exported to dashboards or fed into automated reporting workflows.

The metadata tables enable complex queries that would have been impractical with traditional approaches, for example, finding all broadcasts from a specific date range where object tags or user-defined metadata indicate a particular presidential administration or historical event.

Analyze multiple archives and ensure governance

As Vanderbilt expanded to manage multiple digital archives, the library adopted S3 Metadata tables for cross-collection analytics. The library can now query metadata across VTNA, the Slave Societies Digital Archive, and other collections to understand storage patterns, identify optimization opportunities, and ensure compliance with data governance policies.

“We’re often asked how many files were added on a specific date or how quickly a specific collection is growing,” says Dale Poulter, Director of Library Technology Access Services at Vanderbilt University Library. “S3 Metadata tables allows us to do these tasks with a single query and without the need to touch any of the actual files themselves. This not only saves time, but it also allows us to better respond to questions in real time.”

“Using S3 Metadata for a collection search now takes less than a minute — pulling a collection listing that would have taken an hour in the past. We’ve also been able to locate individual files in under five minutes that previously could have taken several hours to find,” Poulter adds.

Reduce operational overhead and enable cost optimization

Before S3 Metadata, generating a complete inventory of objects across multiple buckets required repeated ListObjects API calls leading to a slow, expensive process at Vanderbilt’s scale. S3 Metadata tables eliminated this overhead entirely, giving the library a continuously updated, queryable inventory without the need to list files across buckets multiple times.

Looking ahead, the same metadata tables position Vanderbilt to make data-driven decisions about storage lifecycle policies. By querying fields like storage_class and last_modified_date, the library can identify candidates for transition to lower-cost tiers like S3 Glacier Flexible Retrieval as their optimization strategy matures.

Key benefits

Improved research access: Collection searches that once took an hour now complete in under a minute, and locating individual files has gone from a multi-hour effort to less than five minutes. Researchers query metadata across decades of archives instead of scanning individual objects.

Operational efficiency: Eliminating manual file-by-file searches frees library staff to focus on content cataloging and researcher support. Metadata queries drive automated workflows that prioritize content for processing, helping the library keep pace with daily additions.

Scalability for growth: As Vanderbilt continues adding new archives and expanding existing collections, S3 Metadata Tables scale automatically without requiring infrastructure changes or manual metadata management.

Foundation for AI/ML innovation: Rich, queryable metadata lays the groundwork for advanced use cases. For example, by combining S3 Metadata with services like Amazon Rekognition or Amazon Bedrock, libraries could analyze and classify content across multimedia archives, enabling new forms of computational scholarship.

Get started with S3 Metadata tables

Organizations managing large digital collections can follow Vanderbilt’s approach:

1. Migrate to S3. Move your digital assets to S3 for durable, scalable storage integrated with AWS analytics. Vanderbilt used AWS DataSync for a one-time migration of 100 TB from on-premises storage, and AWS Storage Gateway for live data that needed continuous access during migration.
2. Enable S3 Metadata. Configure metadata table creation for your buckets through the AWS Management Console, CLI, or SDKs. S3 creates a journal table automatically with every configuration. You can optionally enable a live inventory table for a complete object inventory. When enabling live inventory tables, S3 scans existing objects to populate initial metadata expect a minimum of 15 minutes for smaller buckets, with larger collections taking hours.
3. Query with AWS analytics services. Integrate your AWS managed table bucket with SageMaker Lakehouse (via AWS Glue Data Catalog), then query your metadata tables. You can also use natural language to search objects by their metadata using agents in Amazon SageMaker Unified Studio, or any IDE with the S3 Tables MCP server. Optionally, enable AWS Lake Formation for fine-grained access control.
4. Automate workflows. Build automated processes that use metadata queries to route content for processing, apply lifecycle policies, or trigger notifications.

S3 Metadata is available in supported AWS Regions. For pricing details, see the S3 pricing page.

Conclusion

Vanderbilt University Library’s experience demonstrates how Amazon S3 Metadata can transform the management of large-scale digital collections. S3 Metadata tables replace manual cataloging with queryable, near real-time discovery, making vast archives accessible to researchers, students, and the public.

Whether you’re managing a media archive, a research data lake, or an enterprise content repository, S3 Metadata gives you the tools to understand, organize, and unlock the value in your data at any scale.

Ready to get started? Visit the S3 Metadata documentation to learn more. Go build!