Build AI-powered file classification with AWS Transfer Family

Organizations that receive files from external partners through SFTP face a persistent operational challenge: routing each file to the correct downstream system. Invoices, contracts, images, CSVs, and reports all arrive in a single landing zone, and each requires a different destination. The traditional approach—pattern-matching on file names with regular expressions—is inherently fragile. It relies on naming conventions that partners frequently violate and fails silently when formats change.

AWS Transfer Family and Amazon Bedrock together replace brittle naming-convention rules with content-aware routing that adapts automatically as partners change formats.

In this blog post, we show how to build a production-ready architecture that replaces brittle file name-based routing with AI-powered content classification. Amazon Bedrock analyzes actual file content to determine categorization, removing naming-convention dependence.

Limitations of file name-based routing

Consider a typical scenario: a regex script from several years ago inspects file names and routes files accordingly. This approach presents several failure modes:

• Naming convention drift – Partners change their export formats without notification. A file previously named INV_2024_001.csv arrives as payment_request_june.xlsx.
• Format mismatches – A partner transmits a PDF where the system expects a CSV. The script, unable to read the file content, routes it to a default folder or drops it entirely.
• New partner onboarding – Each new vendor requires manual regex updates, creating an ever-growing maintenance burden.
• Silent failures – Misrouted files generate no alerts. The error surfaces only when downstream teams report missing data, often days later.

File name-based routing breaks down because it has no visibility into what a file actually contains.

Solution overview

The architecture combines Transfer Family for secure file ingestion through SFTP, Amazon Bedrock (using Amazon Nova Lite) for AI-powered content classification, Amazon Textract for PDF text extraction, and Amazon EventBridge with Amazon Simple Queue Service (Amazon SQS) for event-driven orchestration. The following diagram illustrates how file reception, content analysis, and intelligent routing work together.

Figure 1: Architecture diagram showing AWS Transfer Family ingesting files using SFTP, Amazon EventBridge capturing upload events, Amazon SQS buffering classification requests, AWS Lambda extracting content and invoking Amazon Bedrock (Amazon Nova Lite) for AI-powered classification, and routing files to destination prefixes based on content type

The component responsibilities are as follows:

• Amazon Bedrock (Amazon Nova Lite) – Classifies content types: text analysis for documents and multimodal analysis for images.
• Amazon CloudWatch – Monitors Lambda execution metrics and triggers alarms on errors or dead-letter queue (DLQ) depth, providing operational visibility into the classification pipeline.
• Amazon EventBridge – Captures the “Transfer Family File Receive” event (containing partner username, protocol, file path, and bytes transferred) and routes it to the SQS queue.
• AWS Lambda – Extracts content (text files directly, PDFs using Amazon Textract, images through multimodal), invokes Amazon Bedrock for classification, and routes files to the appropriate destination prefix.
• Amazon Simple Notification Service (Amazon SNS) – Sends notifications to the operations team when files are routed to human review or when classification confidence is below the configured threshold.
• Amazon Simple Queue Service (Amazon SQS) – Buffers classification requests, provides automatic retry (up to three attempts), and routes failed messages to a DLQ for investigation.
• Amazon Simple Storage Service (Amazon S3) – Stores three buckets:
  • Archive bucket – Retains untouched copies of original files for audit and recovery purposes.
  • Destination bucket – Organizes classified files into separate prefixes (invoices for downstream processing).
  • Landing bucket – Receives incoming files under the raw-incoming/ prefix.
• Amazon Textract – Extracts text from PDF documents, including scanned documents using OCR.
• AWS Transfer Family – Provides the SFTP endpoint for partner file uploads, with logging and user management.

The complete infrastructure is deployable using the AWS CloudFormation template described later.

Classification strategy by file type

Different file formats require distinct content extraction approaches before classification can occur:

• Text-based files (CSV, XML, JSON, TSV, TXT) – The Lambda function reads the file content directly from Amazon S3, limiting extraction to the first 10 KB. This truncation is intentional for classification purposes; the initial content provides sufficient signal. The model can examine column headers in a CSV and identify it as a financial transaction log without processing 50,000 rows.
• PDF documents – PDFs require content extraction prior to classification. The Lambda function invokes Amazon Textract to perform text extraction (including OCR for scanned documents), then passes the extracted text to Amazon Bedrock.
• Images – For image-based files (scanned contracts, receipts, product photographs), the Lambda classifier uses the multimodal capabilities of Amazon Nova Lite through the Amazon Bedrock Converse API. The image is downloaded from Amazon S3 and sent directly to the model alongside a text prompt asking it to describe and identify the document content.

This approach removes the need for a separate image analysis service: Amazon Nova Lite can read text within images, interpret form layouts, and distinguish between a scanned invoice and a scanned contract in a single API call, routing them to the same destination prefixes as text-based files.

As of August 2026, images larger than 3.75 MB (the Amazon Bedrock Converse API limit) are skipped and routed to human review for manual classification.

Classifier Lambda function

A single Lambda function handles the file types and implements the complete routing logic. This design choice prioritizes straightforward operation and debuggability over premature optimization.

Key design decisions for the classifier Lambda include:

• Amazon Nova Lite – Selected for its speed and cost-efficiency. Because the classifier uses the model-agnostic Amazon Bedrock Converse API, you can switch to any model available in Amazon Bedrock – including Anthropic Claude, Meta Llama, or future releases – by updating the BedrockModelId parameter. No code changes are required.
• 10 KB content cap – The cap minimizes token costs while providing adequate classification signal.
• Graceful degradation – If model response parsing fails, the file is classified as REJECTED rather than causing a Lambda error.

The Lambda code is a compressed Python file, available in the Deploy CloudFormation template section later in this post.

Confidence scoring and human review

AI classification is not infallible. When the model returns confidence below the configured threshold (default: 80%), the file is routed to a human-review/ prefix with an SNS notification to the operations team.

Event-driven orchestration with EventBridge and Amazon SQS

Transfer Family emits lifecycle events directly to EventBridge upon file upload completion. The event payload includes the server-id, user-name, bucket-name, and object key, which the solution uses to trigger the classification pipeline through Amazon SQS with full partner context, automatic retry logic, and burst handling.

This approach offers the following benefits:

• Burst handling – Amazon SQS absorbs traffic spikes (end-of-month batch uploads) and delivers messages at a controlled rate using Lambda event source mapping
• Automatic retry – Failed messages are retried up to three times before moving to the DLQ, with exponential backoff
• Batch processing – Lambda event source mapping is configured with BatchSize: 10 to process multiple files per invocation, reducing cold starts and cost at high volumes

Prerequisites

To deploy the solution, you must have the following:

• Amazon Bedrock model access enabled for Amazon Nova Lite in your target AWS Region.
• An SSH public key for the SFTP user.
• (Optional) An email address for human review notifications.
• Download the Lambda code classifier.zip and upload it to an S3 bucket of your choice.

Deploy CloudFormation template

The following table summarizes the resources you deploy using the CloudFormation template.

To deploy the template, complete the following steps:

1. Download the CloudFormation template.
2. Edit line 520 of the template. Change the S3Bucket value from <some-unique-deployment-bucket-name> to the name of the S3 bucket where you uploaded the Lambda code (classifier.zip).
3. On the CloudFormation console, in the navigation pane, choose Stacks, then Create stack, With new resources (Standard). Choose Upload a template file, upload the YAML file, and then choose Next.
4. On the Specify stack details page, provide the following information:
   1. For Stack name, enter a name for your stack.
   2. For BucketnamePrefix, enter your bucket prefix. (for example, fileclassification).
   3. For SftpPublicKey, enter the SSH public key.
   4. For Public or Private, select the endpoint.
      1. For Private, enter the VPC ID, Subnet IDs, and Security Group.
      2. For InternetFacing, enter the VPC ID, Subnet IDs, Security Group, and Elastic IP Allocations IDs.
   5. For BedrockModelId, leave empty to auto-select the Amazon Nova Lite.
   6. For ReviewNotificationEmail, enter your optional email address.
5. Choose Next.
6. For Capabilities, select the acknowledgement check box, then choose Next.
7. Choose Submit.

Test file transfer and classification

Complete the following steps to validate the file transfer and classification:

1. On the Amazon S3 console, go to <bucket-name>-classified to check if the bucket is empty.

Figure 2: Empty S3 bucket

2. On the Transfer Family console, open the server-id you created, and in the endpoint details, copy the endpoint information (<server-ID.server.transfer.region.amazonaws.com>).
3. Open a terminal and enter the following command:

sftp -i private-key-file <user-name>@server-ID.server.transfer.region.amazonaws.com

4. After you complete the connection, transfer the files.

Figure 3: File transfer

5. Return to the Amazon S3 console and check if the files were transferred and classified in the <bucket-name>-classified bucket.

Figure 4: S3 bucket contains new folders

6. Open the folders to confirm the files were transferred.

Figure 5: Folder containing transferred files

Production considerations

The following issues are commonly encountered in production deployments:

• File encoding – Not all files are UTF-8. Partners in different Regions might transmit Latin-1, Windows-1252, or Shift-JIS encoded content. Use chardet for detection or, at minimum, errors=’replace’ when decoding to prevent classification failures.
• Extension mismatches – Binary files (for example, .xlsx) might be named with text extensions (.csv). Validate file signatures (magic bytes) before assuming the extension is accurate.
• Large files – A 2 GB data export doesn’t require full content extraction; the 10 KB range read is sufficient.
• Amazon Bedrock rate limits – Default quotas are generous for most workloads, but accounts running multiple AI workloads might encounter throttling. Request quota increases proactively and implement exponential backoff with jitter.

Security considerations

For enterprise deployments, the following security controls should be evaluated and applied based on your organization’s compliance requirements.The CloudFormation template scopes all IAM permissions to specific resource ARNs, including the Lambda function’s access to Amazon Bedrock, S3 buckets, and SNS topics. Amazon Bedrock does not use customer inputs for model training, and the Lambda function sends only a 10 KB content sample (or up to 3.75 MB for images), limiting data exposure. For additional hardening, consider enabling SSE-KMS on S3 buckets, restricting SNS publishing principals, and enabling CloudTrail data events for object-level audit logging.

Continuous improvement

Prompt-based classification enables iterative refinement without code changes:

• Log human review decisions – Track how reviewers classify files the model was uncertain about.
• Refine the classification prompt – Add few-shot examples for ambiguous categories. If the classifier consistently confuses purchase orders with invoices, include concrete examples of each.
• Externalize the prompt – Store the classification prompt in Parameter Store, a capability of AWS Systems Manager, or Amazon DynamoDB, so the operations team can tune behavior without deployments.
• Monitor accuracy metrics – Track classification accuracy, confidence distributions, and human review rates over time.

Classification prompt tuning

The classification prompt is the most impactful element of this architecture. A well-tuned prompt reduces human review rates, improves accuracy, and avoids misclassifications, all without code changes.

The guidance in this section is specific to file classification use cases. For general prompt engineering techniques applicable to Amazon Bedrock models, refer to Prompt engineering guidelines.

Add few-shot examples

When the classifier consistently confuses two categories, add concrete examples directly to the prompt:

Here are examples of each category:

INVOICE example: "Invoice #12345, Date: 2026-01-15, Amount Due: $15,000.00,
Payment Terms: Net 30, Vendor: Acme Corp, Bill To: Global Enterprises Ltd"

CONTRACT example: "This Master Service Agreement is entered into as of January 1, 2026,
by and between Provider (CloudTech Solutions Inc.) and Client (Global Enterprises Ltd).
NOW THEREFORE, in consideration of the mutual covenants herein, the parties agree..."

REPORT example: "Q2 2026 Performance Report - Executive Summary: Overall system availability
achieved 99.97% during Q2. Mean Time to Resolve improved 23% quarter-over-quarter."

DATA_FILE example: "timestamp,server_id,cpu_utilization,memory_usage_mb,disk_io_ops,
network_in_bytes,status,region,instance_type
2026-06-09T00:00:00Z,srv-001,23.4,4096,1245,89234567,running,us-east-1,m5.xlarge"

When to update your prompt

Consider updating your prompt in the following cases:

• Human review rate exceeds 5% consistently
• New file types or categories are introduced

Clean up

To avoid incurring future charges, first delete the files from the S3 buckets and then on the CloudFormation console, delete the stack that deployed this environment. As a result, the created resources will be deleted.

Conclusion

File name-based file routing is an inherently brittle approach that fails silently—the worst category of failure in production systems. An AI-powered classifier built with Transfer Family, Lambda, and Amazon Bedrock provides content-aware routing that handles new formats gracefully, communicates uncertainty through confidence scoring, and operates at minimal cost.

The recommended adoption path is incremental: begin with the single partner integration that generates the most support tickets, validate the reduction in misrouted files, then extend coverage across partner integrations.

Deploy this solution in your own AWS account using the CloudFormation template above, and let us know how it works for your classification use case in the comments below.