S3 Provisioner - Automated AWS S3 Bucket Infrastructure for ML Workloads

S3 Provisioner by Axon Tech Labs automates the complete lifecycle of AWS S3 bucket provisioning via CloudFormation. Purpose-built for machine learning workloads, it transforms a simple YAML configuration into production-ready S3 infrastructure - complete with ML-optimized folder structures (130+ folders), automated lifecycle policies for cost optimization, and least-privilege IAM policies.

Deploy across multiple AWS accounts, tenants, environments, and regions with consistent results. Choose between two deployment patterns: shared buckets (multiple ML solutions in one bucket) for cost efficiency, or dedicated buckets (one per solution) for isolation. Every step - from configuration to deployment - is documented in human-readable HTML reports for compliance and team visibility.

Designed for ML engineers, data engineers, DevOps teams, and platform architects who need consistent, repeatable S3 infrastructure across ML projects, environments, and regions.

Key Capabilities:

• Validated Provisioning: Move from YAML configuration to deployed infrastructure with built-in schema validation, structural checks, and isolated test deployments to verify stability and resource limits before going live.
• ML-Ready Orchestration: Built-in logic for ML solution environments. Automatically provision 130+ folder hierarchies organized by ML pipeline phase (data, models, notebooks, artifacts, code, config), clone master-solution structures to specific solutions, and manage .gitkeep files at various directory levels to maintain repo-to-S3 consistency.
• Security and Compliance: Automatically generate least-privilege IAM policy documents tailored to your specific infrastructure needs, ensuring your storage remains secure from day one.
• Drift and Change Management: Maintain full visibility with the ability to preview projected infrastructure changes and detect discrepancies between your live environment and your defined configuration.
• Lifecycle Management: Streamline the entire bucket lifecycle with 4 built-in profiles (ml-optimized, compliance, development, none). From initial template uploads and stack creation to comprehensive tear-down actions that clean up stacks, buckets, and objects simultaneously.

19 Actions:

1. validate-config - Validate configuration YAML template before deployment
2. create-policy - Create a least-privilege IAM policy document for provisioning S3 infrastructure
3. create-prov-template - Generate the CloudFormation template for provisioning S3 infrastructure
4. validate-prov-template - Perform a syntax and structural check on the generated provisioning template
5. test-deploy - Execute a test deployment with an isolated suffix to verify stability and permissions
6. prep-master - Provision the S3 bucket with master-solution folder structure for ML projects
7. create-bucket - Deploy the S3 bucket infrastructure and associated resources to AWS
8. show-changes - Preview projected infrastructure changes before deploying
9. check-drift - Detect discrepancies between your live environment and defined configuration
10. deploy-solution - Add a new ML solution subtree by cloning the master-solution folder structure
11. deploy-folders - Recreate all folders and .gitkeep files for an existing ML solution
12. upload-template - Upload the CloudFormation provisioning template to S3 for team access
13. gitkeep-full - Create .gitkeep files for all folders under a solution
14. gitkeep-none - Remove .gitkeep files from all folders under a solution
15. gitkeep-partial - Remove .gitkeep files from folders two levels below root
16. purge-bucket - Remove all .gitkeep files from bucket
17. delete-bucket - Delete S3 bucket and all associated resources directly
18. delete-cfn-stack - Delete S3 CloudFormation stack related to provisioning
19. tear-down - Perform a complete S3 infrastructure cleanup (stack + bucket + objects)

How It Works:

1. Configure: Define your desired S3 infrastructure in a simple YAML file
2. Execute: Run the Docker container with your config mounted
3. Review: Generate your CloudFormation template and IAM policies, then validate before deploying
4. Deploy: Deploy to AWS via CloudFormation for immediate, reliable resource creation

Technical Requirements:

• Docker 20.10 or later
• AWS account with S3 and CloudFormation permissions
• AWS credentials (access key or IAM role)
• 512 MB RAM minimum