Data Validation Harness

Data Validation Harness (DVH) is a comprehensive toolkit designed for a wide range of data testing and validation use cases. Its modular and loosely coupled design ensures adaptability to the data testing and reconciliation needs of organizations, regardless of scale, domain, or complexity. DVH also offers a fully configurable and customizable automation framework, identifying potential risks and challenges across data workflows.

**How Does it Work? **

For Glue-based pipelines, it offers interfaces to create data quality rules using DQDL (Data Quality Definition Language) within the Rule Builder section. Various rulesets can also be created in Glue Databrew.

It provides granular control over test coverage, allowing custom unit tests to be created using Deequ and Pytest libraries. These tests can be applied across S3, Athena, Redshift, and Glue pipelines.

Third-party frameworks such as Great Expectations, IceDQ, and Pytest can also be integrated with Amazon S3, Amazon Athena, Amazon Redshift, and AWS Glue for more customized testing.

The following use cases are supported by DVH for data validation and testing:

**1. Foundational Capabilities **

• Data Validation
• Metadata Validation
• Data Integration
• Report Validation
• Test Suite Management
• Test Case Repository
• Code Coverage

2. Intermediate Capabilities

• Special Character Testing
• Data Reconciliation
• Data Transformation Testing
• Data Quality Assurance
• Dependency Impact Analysis
• Data Dictionary & Automated Documentation

3. Advanced Capabilities

• Data Encryption Testing
• Data Masking Testing
• Continuous Integration and Deployment (CI/CD)
• Test Data Generation
• AI/ML Testing