Incremental data models have cut full refresh time and support trusted executive reporting
What is our primary use case?
I am currently working with dbt and Snowflake together. We use dbt for data transformation purposes. We obtain the data and store the raw data directly into Snowflake, then perform all transformations using dbt to prepare the data for reporting purposes.
We use dbt's modular SQL models. In dbt, we do not use full refresh or full data refresh. We have incremental strategies in place that only compute or transform incremental data, which operates in a CDC architecture. This approach is very fast, and we use it on a daily basis. We have scheduled all our dbt models using Airflow to run according to the scheduled time.
We use dbt's testing framework and the inbuilt functionality of dbt testing. For example, we use dbt's built-in tests to identify not null values and determine how many not null columns and values exist in each column. Beyond the built-in functionality, we have written custom SQL scripts to create external test cases on our models.
We ensure that incorrect or incomplete data does not go into the reporting layer because it can impact the business. We always perform dbt tests on our landing or raw data to ensure the correctness and completeness of the data before loading it into the final reporting layer. These reports are used by higher management, so we ensure that incorrect data is not published into the reporting layer for the Power BI reports.
We use dbt's documentation site generator. In dbt, we have YML file functionality, which can be used for creating documentation for each model. Whenever we make modifications to a model, we always update the YML file so we can track the history of how frequently we change the model. When new team members join, they can refer to this documentation to understand the data lineage and the data transformation strategy of the project.
What is most valuable?
dbt is very fast compared to the traditional tools. Previously, I worked on SSIS, which is provided by Microsoft, and data transformation took a considerable amount of time when dealing with large amounts of data. Since dbt works on the ELT architecture rather than the ETL architecture, it is much faster than traditional data transformation tools.
Previously, we were using SSIS packages, which were very slow. Recently we migrated all our SSIS packages to dbt models. After the migration, we moved the data from SQL Server to Snowflake. Previously, our data pipeline took around two days to load complete data when performing a full refresh. Since we migrated from SSIS to dbt model architecture, it takes around four hours only to complete a full refresh. The client is now happy because our downtime was drastically reduced when we perform a complete refresh of the data.
What needs improvement?
I am not very familiar with dbt's version control system.
I cannot identify any improvements in dbt because I am still exploring more functionality. I have been working with dbt for only three years, so I am exploring more functionalities and cannot see any limitations or improvement areas at this time.
In the past, I used the seed functionality, which is used to load raw files, individual files, or static files into the database. Going forward, if dbt can improve or implement more features on the seed side, that would be beneficial, especially when we have large files available that take time to load the data into Snowflake database.
For how long have I used the solution?
I have been working with dbt for the last three years.
What do I think about the stability of the solution?
I have not experienced any crashes, performance issues, or anything regarding stability and reliability.
What do I think about the scalability of the solution?
I find dbt very scalable.
How are customer service and support?
The dbt support team is very responsive. Whenever we have any issues on the dbt side, we always reach out to them. We did not face any challenges in the initial setup. I would rate the technical support a nine out of ten.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
Previously, we were using SSIS packages, which were very slow. Recently we migrated all our SSIS packages to dbt models. After the migration, we moved the data from SQL Server to Snowflake. Previously, our data pipeline took around two days to load complete data when performing a full refresh. Since we migrated from SSIS to dbt model architecture, it takes around four hours only to complete a full refresh. The client is now happy because our downtime was drastically reduced when we perform a complete refresh of the data.
The main reason we decided to switch to dbt is performance. As mentioned earlier, every quarter we perform a full refresh, and that refresh took considerable time on SQL Server. Since we had to migrate because our data is very large and growing daily, we adopted dbt because Snowflake is very fast. In Snowflake, the storage layer and the computation layer are separate, which is not present in the SQL Server traditional database. That is why we moved from SQL Server to Snowflake and from SSIS to dbt.
How was the initial setup?
We evaluated Databricks as well, but ultimately the client wanted to adopt Snowflake and dbt technologies only.
What about the implementation team?
We took help from Snowflake directly, the Snowflake company, for the Snowflake side. The dbt side is maintained or set up by our infrastructure team.
What was our ROI?
Since we migrated from SSIS to dbt model architecture, it takes around four hours only to complete a full refresh. The client is now happy because our downtime was drastically reduced when we perform a complete refresh of the data.
What's my experience with pricing, setup cost, and licensing?
The pricing, setup cost, and licensing cost are managed by our infrastructure teams. As data engineers, we are not familiar with these details.
I need to check with my infrastructure team on whether we purchased dbt through the AWS Marketplace or directly from the local vendor.
Which other solutions did I evaluate?
Since dbt has a license cost, if a company is small and does not have much budget, they can explore other tools because there are other tools that provide the same functionality at a lower cost. If an organization is small, they can explore other products as well.
What other advice do I have?
I am currently working with Power BI, Tableau, Python, Databricks, Snowflake, and PySpark in the current project. I would rate my overall experience with dbt a nine out of ten.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Good performance in BigQuery, but the focus on monetization disappoints
What do you like best about the product?
dbt core for data transformations on BigQuery
What do you dislike about the product?
The change in the business model to focus on monetization has been notable.
What problems is the product solving and how is that benefiting you?
Generate data in a structured manner and with version control in ETLs is easy using SQL. This tool facilitates the handling and organization of information during the extraction, transformation, and loading processes.
Effortless Data Transformation with Easy Setup and Integration
What do you like best about the product?
DBT is a data building tool that is very easy to setup, to use and we are using it every day for our data transformation. It is very easy to integrate and leverage the tool with lots for features.
What do you dislike about the product?
Sometimes, it experiences server downtime.
What problems is the product solving and how is that benefiting you?
DBT is solving where our business analyst has data spread out in many different tables in our warehouse. Using DBT, I made datamart where I gathered all that information together so our BAs can get all the information they need from one single table.
Structured data workflows made effortless with dbt
What do you like best about the product?
The largest benefit of dbt to me is that it provides structure to data work. I use it regularly with the BigQuery and version control tools. The integration is comfortable and teamwork is facilitated. It did not add any delay during implementation and the feature set enables one to reuse logic rather than rewriting it. It has minimized the number of errors and saved me time on the review and updates.
What do you dislike about the product?
The negative side about dbt is that it becomes rigid when projects expand. Minor modifications in some cases need more readjustments than anticipated, and this makes me slow down. The problems of debugging failures are not always evident, particularly to more novice team members and this has an impact on the speed of delivery. Clean source data is also used in implementation and hence when inputs are messy, it only adds more workload rather than making it easy.
What problems is the product solving and how is that benefiting you?
Before using dbt, our changes were far between and difficult to handle. At this point, all things go in the same way, which is advantageous to the entire team. The coordination between systems was eliminated through integration and implementation provided a sense of ownership. I can perceive fewer errors, more harmonious work, and a higher level of trust in products. It has made daily work less stressful and less value building oriented.
Reliable Data Automation and Trustworthy KPIs
What do you like best about the product?
What I appreciated most about DBT was its capability to automate the creation of form data models, allowing me to trust the data. I felt confident that the KPIs displayed were accurate, thanks to transformation logic that had been thoroughly tested and addressed, which I found particularly valuable.
What do you dislike about the product?
The learning curve could be smoother, and the user interface would benefit from some enhancements.
What problems is the product solving and how is that benefiting you?
My priority is to ensure that the strategic decisions I make are grounded in reliable and consistent data. DBT enables this by providing a column that transforms data into clear metrics, eliminating any mistrust in the data. This is achieved without requiring its own visualization, allowing the focus to remain on the quality of the data model. As a result, the agility and speed of reporting are significantly improved.
Streamlined Data Transformations with Room for Debugging Improvement
What do you like best about the product?
What I like most about dbt is that it brings software engineering best practices to SQL-based data transformations, making our SQL code base maintainable at scale. It has a clear model structure like staging, intermediate, and reporting layers. It provides macros and ref macros that make logic reusable, and the dependencies are really easy to understand. I appreciate its good collaboration with Git and integration with version control. Dbt has a strong documentation background, providing an auto-generated documentation site, so everyone is aware of what's happening in the project. The initial setup of dbt is really easy thanks to its great documentation, and it's available for almost all major data warehouses.
What do you dislike about the product?
One of the pain points is debugging and error troubleshooting. Error messages can really be vague, making it difficult to pinpoint which part of the core caused the failure. Also, large models are painful to debug. Query plan visibility inside dbt would be really helpful. Step by step execution for failed models would also be helpful.
What problems is the product solving and how is that benefiting you?
dbt provides a standard structure for our code base, eases data transformation with Jinja templating, organizes SQL scattered across tools, offers version control with Git, and includes data quality tests, making transformations maintainable and dependencies clear.
I can manage my own dependencies using dbt.
What do you like best about the product?
dbt runs well on Redshift, since that is what was mentioned over and over again in the notes; however, dbt simply compiles the SQL and the warehouse itself handles the heavy lifting. Using Git and Version Control for Data Models, is nice because it keeps the data model from exploding. dbt also integrates with our AWS infrastructure without requiring tears. The speed is sufficient, as it simply passes the work to the database; although, having the transformation logic in one location is helpful.
What do you dislike about the product?
The cost is becoming increasingly expensive and considering dbt is essentially a fancy SQL Compiler. dbt also has poor performance when handling un-structured data (although this may be due to Redshift); I'm unsure, everything seems to blend together. Additionally, the learning curve is very steep if you are not familiar with Jinja and setting-up YAML files properly.
What problems is the product solving and how is that benefiting you?
dbt allows us to scale the analytics engineering work so we are not running ad-hoc SQL scripts on a laptop. dbt separates the compute and storage logic, allowing us to define the "what", while it determines the "how". dbt automatically manages the dependency graphs, which is great, as I cannot handle tracking those manually.
Streamlined Data engineering and built-in lineages
What is our primary use case?
dbt is used for data transformation and data engineering with multiple data transformations and engineering functions. It is also used for orchestrating data engineering pipelines. An example of this is ingesting data from Azure Blob or S3 sources and then transforming it into different layers in the data platform.
What is most valuable?
The best features of dbt include lineage and Jinja templating languages that make it easy for creating pipelines.
The built-in lineage feature provides a good understanding of the several layers where data is being loaded in dbt, allowing visibility from different layers into the end product.
dbt has positively impacted version controlling as it has different version control steps involved. The specific improvements seen with version control in dbt are that it has helped trace the data lineage, enabled faster trace and rollbacks, and enabled safe collaboration at every scale, which has improved data quality.
A return on investment has been seen from using dbt as the time has reduced while utilizing dbt in the form of data pipelines and ETL scripting. There is operational efficiency achieved, and data quality and governance have also been achieved with modular SQL and version controlling, which reduced duplication of data and data errors.
What needs improvement?
dbt is not as stable as preferred, as it has had a few outages in the current year itself, so improvement should be made in the outages section as it is not stable.
The copilot in dbt is not very comfortable for users, and my team has already tried using it but opted to move off from the dbt copilot to other copilots such as GitHub.
Improvement is needed in the tool itself in terms of the copilot, in terms of covering outages, in terms of testing, and in terms of quality reasons related to governance and collaboration.
For how long have I used the solution?
dbt has been used for about a year.
What do I think about the stability of the solution?
dbt is not as stable as preferred, as it has had a few outages in the current year itself, so improvement should be made in the outages section. Overall, dbt is stable.
What do I think about the scalability of the solution?
In terms of scalability, dbt has improved the scalability of the organization depending on different dimensions for team size, data, and complexity of transformations.
How are customer service and support?
The customer support from dbt was good and was identified and resolved by the customer support team when reached out to.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
Initially, multiple solutions such as Talend Studio and Informatica were utilized for different projects before switching to dbt.
How was the initial setup?
The experience with pricing, setup cost, and licensing was that it was straightforward for the pricing setup and also on the licensing part for dbt.
What was our ROI?
A return on investment has been seen from using dbt as the time has reduced while utilizing dbt in the form of data pipelines and ETL scripting. There is operational efficiency achieved, and data quality and governance have also been achieved with modular SQL and version controlling, which reduced duplication of data and data errors.
What's my experience with pricing, setup cost, and licensing?
dbt was purchased through the AWS Marketplace.
Which other solutions did I evaluate?
Before choosing dbt, other options were evaluated, but dbt was the preferred choice as it was an open-source solution that was already on the track.
What other advice do I have?
My advice to others looking into using dbt is that it is a good tool for having ETL or ELT transformations done. To begin with, a pilot project can be added with modular SQL or modeling, Git workflows, and a standardized project structure from source, staging, intermediate, to the mart layers, which will optimize performance. I would rate this solution a seven out of ten.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
User-Friendly Data Modeling with Seamless Integration
What do you like best about the product?
DBT is great for organizing data models. It is user friendly, integrates well with other tools, and they had a great onboarding process.
What do you dislike about the product?
In dbt Cloud, I cant work on two different branches at the same time in different browsers.
What problems is the product solving and how is that benefiting you?
DBT allows us to take raw data from many sources and output it in clean, easy to use output tables that are used in our bi tool.
We finally found a solution for easier management of data models
What do you like best about the product?
The interesting fact about dbt is that it simplifies the process of managing data pipelines. It was implemented successfully and I depend on it on a daily basis and hence my frequency of use is high. The amount of features such as model testing, documentation, and version control is especially appreciated by me. It has minimized errors in our conversion processes and has simplified the process of teamwork a lot and has helped the team maintain pipelines which are uniform and structured across projects.
What do you dislike about the product?
The thing I dislike with dbt is that it may be difficult to troubleshoot model errors. The features are good, and error messages are not always helpful in disclosing the problem. High frequency of use implies that such moments have the capacity of derailing workflows since I use it frequently. There is responsive customer support but edge-case fixes are not always immediately available, so the team occasionally has to check outputs before proceeding.
What problems is the product solving and how is that benefiting you?
Dbt has resolved the problem of inaccurate or inconsistent transformations within our workflows. It has simple implementation and I use it frequently hence my usage frequency is also high. It has many features that can be used to test and keep track of the version that helps in uncovering errors at the earlier stages. It is lean cooperation throughout the team, reduced manual checks that have to be done multiple times, and ensures our data is reliable and can be used in reporting and business decisions.
