Overview
Migration of data environment and related functionality to AWS Glue and implement archiving policy for Raw files. ▪ Check for possible optimization of ETL jobs during migration and align with client on the same. ▪ Data Quality checks that are present in the current environment need to be implemented into the new AWS Glue jobs. ▪ Glue jobs performance needs to beat or meet SLA of current workload. ▪ Migrate Current data and historical data from Hadoop in Cloudera on prem environment to AWS Cloud. ▪ Check any re-engineering required for Autosys jobs to ensure Orchestration integration is intact. ▪ Configuration and Setup of business applications for scheduling ▪ Ensure all the common services required for AWS Environment with respect to IAM, Alerts and Monitoring
Highlights
- Infrastructure Readiness • AWS Development Instance in Place ** • ETL and BI Instances in place ** • Connectivity between tools have been established ** ▪ Build out Development Target AWS environment ▪ Build out user security structure
- Target Data Store(S3) ▪ Converted Data Store in Development environment ▪ Converted access layer ▪ Converted and new and Unit tested ETL code ▪ ETL Scheduler changes complete for ETL code ▪ Connectivity changes in place for dev and test servers ** ▪ Historical Data Migration process set up and tested ** ▪ Finalized End User security structure
- Instantiate new data model for Target Data Store ▪ Build new pipes to feed into Target Data Store (S3) ▪ Build and test History Data Migration Process ▪ Connectivity Changes for downstream data feeds ▪ Convert Business logic into transformation rules