AWS Storage Blog

Category: Compute

Orchestrating multi-agent AI architectures with Amazon S3 Files

​​​​Organizations are moving beyond single-model AI toward multi-agent architectures. In these systems, agents offload intermediate results to files rather than carrying everything in the prompt, because a large prompt inflates cost and degrades quality. A model’s context window is finite, so files become working memory that persists after a session ends. In multi-agent systems, a […]

Hybrid ML inferencing on Amazon EKS with Amazon FSx for NetApp ONTAP and on-premises NetApp

Machine learning (ML) models used for inference on Kubernetes are often several gigabytes in size. When these models are embedded in container images, images become oversized and pod scheduling slows. More critically, inference pods are inherently stateful. Model weights, tokenizer files, compiled GPU kernels, and runtime caches must persist across pod restarts, node failures, and […]

FSxZ featured image

Optimize your self-managed PostgreSQL data warehouse with Amazon FSx for OpenZFS

A data warehouse is the analytical backbone of a modern enterprise, consolidating data from disparate sources into a single, authoritative view that enables complex queries, trend analysis, and confident decision-making. In financial services, this means sharper regulatory reporting, faster fraud detection, and deeper customer understanding. The operational reality is demanding. Enterprises juggle multiple source databases […]

s3-annotations-header-image

Track healthcare data lineage in real time with Amazon S3 annotations

Organizations that store sensitive data need to answer a simple question: where did this data come from, and what happened to it? In healthcare, this isn’t optional. Regulations like HIPAA, GDPR, and SOX require organizations to produce this information on demand. When an auditor asks, for example, “Show me every transformation that touched patient record […]

Amazon S3 Replication

Accelerate Amazon S3 Replication with automated S3 Batch Operations parallelization

As data volumes grow, organizations must move large datasets between storage locations to meet compliance requirements, optimize performance, and help meet data sovereignty requirements. However, migrating petabytes of data presents significant challenges: lengthy transfer times, complex coordination of parallel operations, data integrity verification, and substantial engineering overhead. Automation reduces resource consumption and operational complexity during […]

Simplify compliance-driven Amazon S3 data movement with multi-criteria filtering

Organizations routinely need to move a subset of their stored data from one location to another, such as a compliance audit that requires all PDFs from a specific quarter, a company reorganization that splits one tenant’s records into an isolated bucket, or a regulatory mandate that demands financial documents older than 7 years be archived […]

Amazon S3 Express One Zone thumbnail

How Turso built a transactional database using Amazon S3 Express One Zone

Turso is a transactional, cloud database platform built on SQLite, serving developers and enterprises that need lightweight, high-density databases at the edge and in the cloud. Because each database is just a file, a single Turso compute node can host millions of them with no cold start. Turso delivers these databases in a Bring Your […]

Figure 1: Solution architecture diagram

Replicate Amazon S3 bucket configurations across AWS Regions with AWS Step Functions

Many organizations operate thousands of Amazon S3 buckets in a single AWS Region, each with its own configuration accumulated over the years. Some were created manually in the AWS Management Console and others by scripts that are no longer actively maintained, provisioned by different business units with their own policies, lifecycle rules, encryption, and tags. […]

Scalable cross-cloud data migration to Amazon S3 with distributed rclone

Migrating petabytes of data across cloud providers is one of the most operationally demanding tasks an organization can take on. At this scale, simple transfer approaches break down. Teams lose track of what has been copied and what has failed. Transfers stall and require constant manual intervention to restart. In some cases, teams need to […]

Amazon S3

Implement single-exchange tokens for short-lived Amazon S3 presigned URLs with Terraform

Organizations across industries use signed URLs to grant temporary, credential-less access to private resources such as receipts, medical or financial records, legal files, or confidential reports. However, signed URLs can be reused by anyone until they expire, creating security risks if a URL is shared or inadvertently disclosed. This risk can be mitigated by vending […]