Posted On: Sep 20, 2023
AWS Glue Data Quality can now identify records that have the CustomSQL rule type, enabling customers to isolate problematic records from the good records to allow only high-quality data flows into their data lakes and warehouses.
AWS Glue Data Quality automatically calculates statistics, recommends quality rules, monitors data quality, and alerts users when it detects a decline in quality. Customers can create rules using over 25 pre-defined data quality rule types to spot and address problematic data. One of these rule types is the CustomSQL rule type, which empowers customers to utilize SQL for crafting intricate business rules to identify quality issues. Until today, the CustomSQL rule type identified the presence of problematic records without identifying them specifically. With this new release, the CustomSQL rule type can pinpoint the specific records responsible for rule failures, flagging them for isolation. This feature enables customers to create robust business rules in SQL, detect and isolate problematic records, and guarantee the delivery of high-quality data to their data repository.
To learn more, visit our documentation and the blog.
This feature is available in all commercial AWS Regions where AWS Glue is available.