In my current project, I am not using Delpha Data Quality, but in a previous project, I used it for approximately 1.5 years to check for anomalies and address data quality issues in our data.

My main use case for Delpha Data Quality revolved around checking data quality issues as we fetched data from multiple sources and, after all transformations, loaded the data into the Snowflake database. Before loading the data into the final application, we checked for data quality issues such as not null conditions, uniqueness, completeness, and correctness. To check all anomalies and data qualities in the source data, we used the tool.

In my previous project, we were fetching data from multiple sources like CSV, Excel, and SQL Server. First, we loaded the data into Snowflake staging tables, which contained raw data only. Before moving the data from the staging layer to the bronze layer, we used Delpha Data Quality to perform data quality checks. For example, in customer data where the customer ID serves as the primary key, we validated whether it had a valid customer ID or if it contained a null customer ID. This example represents just one aspect of our data quality checks across many columns in customer data and sales data. Delpha Data Quality performs these checks and generates a score, and we defined a threshold where a score greater than 90% indicates good data quality. Consequently, if the score exceeds 90%, we move the data from the staging area to the bronze layer. If the score is lower, we stop the process and fix the issues in either the staging or source data before proceeding.

Setting up and interpreting the data scores in Delpha Data Quality is straightforward due to its inbuilt functionality. When we define and evaluate any table, we specify the tests to perform on various columns, such as not null or unique checks. Delpha completely automates the scoring process, which generates the final score based on these parameters.

The best feature I appreciate about Delpha Data Quality is the data score, which evaluates the data based on specific parameters or dimensions, ultimately generating a score for each column and table. This score assists our data stewards in determining whether the data is suitable for further downstream applications. It is a crucial metric for deciding if the data is good or requires remediation, which I find to be a great feature.

Delpha Data Quality not only highlights issues in data but also provides suggestions for improvements, indicating the main areas to focus on for applying fixes. This aspect is another significant feature I discovered in Delpha Data Quality.

Delpha Data Quality has positively impacted my organization by replacing traditional tools such as Informatica for test case execution. Previously, we relied on Informatica for testing all tables and columns, but adopting Delpha Data Quality means it performs tests and generates scores for all Snowflake tables, allowing us to determine the quality of data effectively, a feature not available in our prior tools.

On the UI side, some improvements could be made. In projects involving multiple tables in Snowflake, having a dashboard feature to provide a centralized view of each table and column would be beneficial, allowing data stewards and quality assurance teams to monitor and identify areas needing attention more easily.

While we used Delpha Data Quality with Snowflake, I understand that it is not compatible with all databases. Enhancing integration capabilities, particularly with databases such as Azure SQL Server or Databricks, would improve its functionality, as the current integration process is not seamless across all cloud providers.

I have been working in the data engineering field for over ten years.

Delpha Data Quality is stable.

Delpha Data Quality exhibits good scalability. Initially, we set it up with one database, Snowflake, but we have since integrated it with additional databases without encountering any performance issues, thanks to its cloud-based architecture, allowing seamless scaling as our data volume grows.

Customer support from Delpha Data Quality was excellent during the initial setup. Our infrastructure team received full support, which was crucial as the setup process was time-consuming. However, once in operation, we experienced no significant issues.

I previously used Informatica for data quality checks and switched to Delpha Data Quality because it offered features such as generating final scores and providing recommendations. Informatica would highlight issues without offering the same level of insight into how to address those problems, leading us to rely more heavily on Delpha Data Quality for smoother data pipeline operations.

The pricing, setup cost, and licensing were managed by our client infrastructure team. As developers and users of Delpha Data Quality, we did not handle its setup. The infrastructure team purchased licenses and oversaw the initial setup.

There has been a notable return on investment, especially regarding time reduction. Previously, addressing issues could take four to five hours, but with Delpha Data Quality, relying on its recommendations has shortened the troubleshooting time to approximately one to 1.5 hours when we confirm a quality score above 90%, leading to fewer failures.

The pricing, setup cost, and licensing were managed by our client infrastructure team. As developers and users of Delpha Data Quality, we did not handle its setup. The infrastructure team purchased licenses and oversaw the initial setup.

After switching to Delpha Data Quality, we experienced notable improvements in time efficiency. For instance, when using Informatica for data quality checks, any data failure could take four to five hours to troubleshoot. However, with Delpha Data Quality, we save considerable time. When the data quality score exceeds 90%, we can proceed without stopping the data pipeline, reducing troubleshooting time to approximately one to 1.5 hours.

I chose a rating of 8 out of 10 because over the 1.5 years I used it, I explored many features. However, tracking all tables and the associated data quality checks can be cumbersome, especially with a large number of test cases spread across various tables. This leads to difficulties in monitoring all test cases in one place, requiring us to check the final scores for each object individually.

The governance and security of Delpha Data Quality are commendable. Integration with databases such as Snowflake or SQL Server involves fetching data without storing it, ensuring the security of our data during testing. Moreover, the AI capabilities not only highlight data quality issues but also provide recommendations for necessary fixes.

The accuracy and reliability of Delpha Data Quality are impressive. During the initial setup, I performed manual data quality checks and compared them with Delpha Data Quality outputs. In 99% of cases, the results from manual tests matched those from Delpha Data Quality, reflecting its high accuracy.

My advice to potential users of Delpha Data Quality is that it is ideal for organizations dealing with large data sizes, such as petabytes or gigabytes, and for critical data architectures. Due to the relatively high licensing cost, smaller organizations or projects with limited data might find it less beneficial and should consider evaluating other options.

I believe I have shared all relevant details regarding the features I have explored in Delpha Data Quality, and I feel satisfied with the insights provided. My overall review rating for Delpha Data Quality is 8 out of 10.