Data Repository for Growing Businesses

Cloudtech will design and implement a data storage and sharing solution with ETL and a data lake. Specifically, this packaged solution will gather data from up to five (5) data sources into Amazon S3, performing ETL using AWS Glue. The solution will also include monitoring, alerting, and logging using AWS-native tooling.

Solution Stages:

Requirement Gathering and Design:

• Identify data sources, types, and volume
• Determine user access requirements and data sharing needs
• Define ETL requirements.
• Develop the architecture diagram

Implementation:

• Set up data storage
• Configure ETL processes
• Set up monitoring, alerting, and logging Testing and Validation
• Perform unit testing of ETL jobs
• Validate data accuracy

Deployment:

• Assist customer to deploy to production
• Monitor performance and security

Training and Documentation:

• Provide training and knowledge transfer
• Deliver architecture and process documentation

Maintenance and Support:

• Establish support and maintenance plans
• Schedule reviews for optimization and scalability

Price: $7,500 (For additional data sources and/or other modifications, custom pricing is available.)

Estimated AWS cost: $32/mo; $389/yr

Timeline: 2 weeks (customer infrastructure, communication, and business processes may impact timeline.)

Assumptions and limitations: AWS Glue; DPUs for Apache Spark job (10), DPUs for Python Shell job (0.0625) AWS S3; S3 Standard storage (1 TB per month), PUT, COPY, POST, LIST requests to S3 Standard (10000), GET, SELECT, and all other requests from S3 Standard (10000), Data returned by S3 Select (0 GB per month), Data scanned by S3 Select (0 GB per month) DT Inbound: Internet (50 GB per month), DT Outbound: Not selected (0 TB per month).

Pricing assumes the following criteria. Implementations falling outside of these assumptions can incur additional professional services and AWS costs. 1/Customer has an existing AWS account with a basic OU structure in place. 2/Customer employs at least one full-time technical resource that can assume the role of AWS administrator, facilitate any needed data transfer functions, and provide answers to current infrastructure and architecture questions. 3/Customer is transferring data from a maximum of three (3) distinct data sources. 4/Volume of data to be transferred is 300 GB or less.