dbt Platform

What do you like best about the product?

I simply love how I can write regular sql select statements and then dbt takes care of translating those into views and tables within bigquery w/o requiring constant supervision. After loading large amounts of data into the warehouse we use dbt to prepare those datasets for looker and tableau, and creating partitions or clusters in the config file is as simple as I could have hoped so our queries complete quickly. I've been working a crazy long shift today and I am literally falling asleep sitting here on the couch now and I don't think I would be able to write some sort of epic post... it's just easier to say it works.

What do you dislike about the product?

The cloud ui is annoying. When I'm looking at the result set of a query it defaults to cutting off rows and attempting to copy/paste results from the small browser window is a real pain. And on top of that, I'm stuck with a bad keyboard on this laptop and typing out all these repetitive yaml files has made me want to toss it in the trash. Compiling jinja errors also suck for debugging purposes since the error message is usually so vague that it won't even tell you which line caused the issue.

What problems is the product solving and how is that benefiting you?

Prior to using dbt, our group ran various "random" saved queries and miscellaneous scripts everywhere; and we had absolutely no clue which table to use for reporting. Now, we take raw data from mongo, ga4 and cloud storage and push that into bigquery and use dbt to clean up in one central location, so everybody uses the same numbers. Our previous pipeline would frequently fail under heavy loads, forcing us to run jobs overnight but now our transformation process completes much quicker than before and we can run them multiple times per day while both the analytics and engineering groups work during the same time.

We have a FinOps use case, which means we are processing billing data from cloud providers. We use dbt to create data models around billing and convert them into insights where we can find cost optimization, and also a reporting function, ensuring that the cost in the cloud infrastructure is attributable to the team, the organization, or whoever is responsible for that resource.

When we started adopting agentic engineering, we were looking at a tool that could help us do some verification. dbt came up as the leading tool that could support the workflow that we have envisioned with the team because we would not want to use AI without a capability to do verification and to do proper testing on data models.

The capability for doing proper testing in dbt is quite recent. One thing that dbt still lacks is native support for knowing the coverage. In traditional software engineering, once you have a tool packaged as a testing tool, you immediately have a number beside that tool. If you do testing, it can tell you how much of the source code is covered by tests. A good coverage could be that 90% of all tests or 90% of the code is covered by tests, and you can see that bad code potentially has 20% uncovered tests. In the case of dbt, this number is not something that they support out of the box, and you have to be creative to extract this number.

I have been using dbt for about a year.

So far, I have not experienced any stability issues, but I can attribute this to our workflow being more automated. Our commands, when we issue dbt, are predictable. We rarely have to run a dbt command ourselves, and if we have to do that, we have playbooks, which means everything is documented whenever we need to do something with dbt. When you are issuing dbt commands, you have a tendency to alter active data, which depends on if you alter the datasets, the reports which upstream users are looking at, whether management or those using our FinOps tooling, will be affected. We have streamlined the process of running dbt to avoid executing things that could accidentally be an issue in production.

For our small team of three engineers, dbt is super stable at the moment. When thinking about scalability, every engineer has to have their own workspace. If everyone is playing around with dbt commands, there is a high tendency that there will be congestion. If I update a model, another engineer updates a model, and then another engineer updates the same model, then it is going to be an issue. Every head needs to have its own separate workspace, which means every workspace will have to have test data and the same baseline schema. This is the current setup I see for dbt, so as long as you understand that this is the condition for using dbt, every engineer that needs to work on the codebase needs to have their own individual workspace. This is quite different from traditional software engineering product teams, where people can share a test environment. For dbt, this is not the case, so everyone contributing to the code needs to have an individual workspace, which makes it less scalable than other tools, but this is really because dbt is a data-related tool, not the fault of the system.

There is an active community, but to be honest, I just have not had time to engage or ask around. I think when the problem is medium to high for us, there will be a bit of pressure, but for now, this is more of a wishlist rather than something that is restricting us.

Being already familiar with the DevOps stack, so CI/CD, it is not a high bar for learning for me. I can imagine that some people could feel a bit intimidated by it because I know some colleagues who have been working in data engineering and they have difficulty adopting or understanding how you set up a tool such as dbt. I would say it depends on the background and the skills inside the software engineering team.

We are unfortunately, or fortunately, depending on the perspective, not using dbt Cloud, which is the paid option. We are using the open-source version, which is dbt Core. For our use case, this is sufficient at the moment, but we will see eventually if there is really something interesting that we would like to leverage in their paid offering. For now, using the open-source version is sufficient for us.

I did read about a few other tools such as Bruin, or maybe some teams are actually using just Python testing, but those solutions are not as close to what you would be using if you are developing the data engineering part of FinOps, which is closely tied to SQL as a language. The other tools are more geared toward other programming languages, and dbt is quite unique in that it is the only one supporting testing with SQL as a first-class language.

I have been working in my current field for more than 15 years.

We take care of maintenance in terms of checking compatibility of upgrades. For example, dbt released a newer version in Rust that is a bit faster in terms of performance, but it is not yet compatible with our AWS data analytics stack, so we cannot switch to Rust even though we would prefer to. We have to wait for the connector to AWS if someone builds that. This is the problem with open source: if someone does not build it, potentially we are stuck with Python. But it is still working for now, so in terms of maintenance, if you want to maybe go for the state of the art, that could be a use case to be on a paid subscription to dbt, and then you tell them you would prefer this to be supported in AWS so we can merge to the latest version in Rust and adopt a faster workflow.

Amazon Web Services (AWS)

What do you like best about the product?

It’s very simple to use. All we really need to know is SQL, and it’s easy to create different types of materializations in it. Most importantly, it has a lineage feature.

What do you dislike about the product?

The UI could use a bit of improvement. After opening different models, it starts to look a little messy and cluttered. Cortex can be added by default to add models using AI.

What problems is the product solving and how is that benefiting you?

It helps me create different types of models in Snowflake or any other warehouse. I can write simple CTE-based SQL, and dbt generates the underlying code and creates the tables in Snowflake or other databases.

What do you like best about the product?

Great UI—it's easy to see the lineage and the data flow.

What do you dislike about the product?

Lineage isn’t exactly easy to follow when it’s rendered. I wish there were a better explorer so I could view the lineage properly.

What problems is the product solving and how is that benefiting you?

dbt is our primary data transformation tool, it is helping us model data to out needs

What do you like best about the product?

Easy SQL-based data transformations and testing.

What do you dislike about the product?

Large projects can become difficult to manage.

What problems is the product solving and how is that benefiting you?

DBT automates data transformations, testing, and documentation, improving data quality and pipeline reliability, and development efficiency.

What do you like best about the product?

its capabilities to put features of development pipelines into the data pipelines. The interface is fairly easy to use. Its scalability and performance, as it can handle very big workloads by pushing it into the DW. It works wonderfully integrated with Snowflake. Pricing is predictable.

What do you dislike about the product?

It requires very high skills to use effectively. Should have much more AI capabilities.

What problems is the product solving and how is that benefiting you?

Cleaning and preparing the data for all other processes

What do you like best about the product?

I like the amount of variety that dbt offers because of the Jinja code and the inbuilt functions. The incremental models are very well built-in, offering lots of capabilities at a layer beyond what's in the data warehouse, like Redshift. The initial setup of dbt is very straightforward, which I find really helpful.

What do you dislike about the product?

I guess the development cycle of dbt is slower as a result. Writing YAML file descriptions and the actual code for every single model can lead to a slower development cycle.

What problems is the product solving and how is that benefiting you?

dbt allows us to handle data with software engineering best practices, letting us write tests, store code in a git repository, and manage it like a software project.

What do you like best about the product?

I use DBT for a lot of our sql models to version control. It's easy to use and helpful

What do you dislike about the product?

So far there really hasnt been anything wrong with the platform. You need admin that know what they're doing

What problems is the product solving and how is that benefiting you?

Data model version control and merging updates into snowflake tables on schedules

What do you like best about the product?

I find dbt very simple to use and get started with. Anyone with some SQL and YAML files can use it. It solves many problems related to data transformations, data quality, and data lineage. I like dbt Cloud for scheduling jobs, which is crucial for me, and I appreciate that I can schedule jobs in various environments based on my use case and get alerts too. I also enjoy the new AI feature, dbt-copilot, which helps with development, understanding code better, and writing dbt macros, which can be challenging. The initial setup was easy as it mostly required SQL expertise, and the training sessions were helpful. Overall, I love this tool.

What do you dislike about the product?

I like most use cases with dbt, but sometimes I have to write the entire code just to compile it. It can improve in terms of showing errors early before compile time. For example, if I am selecting from a model and a column is extra, dbt could flag it before compile and run actions.

What problems is the product solving and how is that benefiting you?

I use dbt for data transformations, quality testing, and lineage on top of warehouses. It simplifies starting up as it just needs SQL and YAML files. With dbt Cloud, I can schedule jobs and get alerts. The dbt-copilot aids development, particularly with writing dbt macros.