Data Quality Pilot – Medium

Approach:

After the Data Quality Analysis and Metadata Collection engagement, using the client’s data quality technology, Adastra will implement data quality cleansing and validation rules and store the results in a table that can be used for repeatable processing. AWS Glue will be used for data wrangling and consolidation and data can be persisted in Amazon S3.

Exceptions will be exported to an Excel file for manual remediation. Amazon QuickSight or a client-provided business intelligence/visualization tool will be used to create a data quality dashboard.

• Scope: Approximately 60-100 data elements, depending on the outcomes of the DQ analysis
• Duration: 10 -12 Weeks

Assumptions/Dependencies:

• Adastra’s DQ Analysis and Metadata Collection have been completed prior to the start of the engagement
• Adastra resources will have access to the client’s DQ technology:
  • The scope will be fixed and agreed to prior to the start of the engagement
  • Business SMEs will be available and engaged for review @20%
  • Production quality data must be available
  • The environment must be provided for data access
  • Data stewards have been identified and engaged
  • The solution will not be deployed to QA, user acceptance testing (UAT) and production environments, as this is a pilot engagement

Client Requirements:

• Data quality technology acquired, installed and configured
• Access to production quality data
• Access to Amazon QuickSight or equivalent BI/visualization tool

Next Steps:

• Production deployment
• Extension of the Data Quality Pilot scope

Activities:

• Define and validate data quality rules
• Design and implement data quality rules in DQ technology
• Design and implement orchestration and workflow processes
• Implement DQ exception export processing
• Documentation

Deliverables:

• Requirements document
• Code base for repeatable one-time data quality cleansing and validation rules
• Data quality exceptions document
• Run book
• Development guide
• Data quality dashboard on Amazon QuickSight

Outcomes:

• Pilot data quality solution
• Repeatable DQ rules in code base
• Data quality measurement and monitoring framework