Dagster Labs

A unique workflow I have developed is that I have completely automated all the pipelines by using sensors that Dagster Labs provides by default. This is my main use case and a unique flow that I have developed. Since automating with Dagster Labs, I save around 90% of manual effort. I monitor Dagster Labs jobs once a week to check whether everything is working fine, but other than that, I never open Dagster Labs to see how jobs are processing.

What makes the Python-based orchestrator in Dagster Labs stand out for me compared to others is hosting Dagster Labs on Snowflake , which is a very useful feature because due to the heavy pipelines that I run, Snowflake eases my tasks by scaling up and down. Connecting to different sources, I mostly use Spark jobs for submitting Spark jobs directly from Dagster Labs.

Dagster Labs has positively impacted my organization by increasing efficiency because I automated all my pipelines using sensors in Dagster Labs, which reduced a lot of manual effort and less manual intervention. Unless there is a major issue with data or something else, there is no need for manual intervention, and everything goes smoothly.

Smaller pain points include that after I switched from native Dagster Labs to Snowflake, some of the GraphQL APIs started failing; they load only when I load on the browser for the first time or when I have to clear the cache. Sometimes the GraphQL APIs silently fail, which shows the health of the pipelines that have been run in the last 24 hours; I haven't debugged much into it because it's not impacting much, but it could provide some useful metrics.

Dagster Labs handles scheduling and monitoring of pipelines fairly well; however, one issue while scheduling jobs is that I can't trigger another job from another job. If job A succeeds, I can't directly trigger job B from job A. The only possibility I see is using schedule sensor-based jobs, where I poll the current job, and once that current job succeeds, I trigger another job. It's an alternate way, but it would be useful if there was the functionality of triggering job B from job A.

My advice for others looking into using Dagster Labs is to first understand all the functionalities that Dagster Labs offers; take a little bit of time to try and explore all the features, and then it's better to go into implementation. That way, you can design your pipelines better.

I chose eight out of 10 because of the retry issue I mentioned earlier; I would have rated it 10 out of 10 if there was an option to retry the pipeline from the point of failure. It would be great if the feedback items could be released in upcoming versions.

My main use case for Dagster Labs  is for orchestrating my data processes, like data pipelines and other automations related to data. The main thing I use it for is to build data pipelines because it has many good features such as lineage and auto parent detection that help significantly.

I will share the flow where I have used Dagster Labs . For example, there is a source database where data is populated every millisecond. From that source, I process that data and ingest it into my database. After ingesting, I process the data, transform it, and perform many transformations that my business requires, such as transformations used in our business cases and other stakeholders' preferences for formatted data. Then I build up my data marts. There are several data marts, and after the data mart is completed, another process in that same pipeline emails the required data to the stakeholders.

Dagster Labs has positively impacted my organization, and I have noticed various outcomes.

I have seen specific outcomes or improvements since using Dagster Labs, particularly in time saving. Previously, I had to use Airflow  where I had to manually group or manually create the lineage from upper stream to lower stream, which was cumbersome because I had multiple processes, tens or hundreds of them. Manually building up the upper stream to lower stream lineage was difficult. Dagster Labs helps significantly in this case because it reads the manifest.json file and easily generates the lineage. I can code it and not worry about the lineage building process because Dagster Labs handles it.

The unique aspect of my use case with Dagster Labs is that it picks up the manifest.json from DBT. I actually use DBT with Dagster Labs. My main use case is running DBT commands using Dagster Labs and integrating Python with it.

The best features Dagster Labs offers are that it automatically reads the manifest.json and generates the lineage. It has a unique UI where I can see the code and control the schedules, runs, and everything else.

I want to know more about the backfill features. I believe that the backfill features in Dagster Labs are quite complex. If the design team creates something that makes the backfilling process much easier, it would be very helpful. Additionally, AI integration is very relevant nowadays, and integrating AI data tools with Dagster Labs might be a good feature to add.

Dagster Labs can be improved by making the backfilling features simpler and less complex, which would be very efficient.

Needed improvements include making the backfilling process easier and less complex. I would also suggest that while Dagster Labs has a UI, the integration with DBT does not have a stable version that integrates well with DBT Core. I had some problems with this integration. Since DBT is one of the main tools for transformation, I would suggest that Dagster Labs should have an updated version that is stable with DBT Core.

I want to continue using Dagster Labs. The improvements I have pointed out—integration with DBT in a stable, updated version, and reduced complexity in backfilling—would help significantly.

I have been using Dagster Labs for nearly eight months.

Regarding Dagster Labs' AI capabilities and the accuracy and reliability of output, I believe it is acceptable because this is a workflow orchestration tool. If AI is integrated into designing orchestration, I do not see any harm.

I would advise others looking into using Dagster Labs to first find the stable version compatible with the tools they are integrating with. I rate this product 7.5 out of 10.

I used Dagster Labs  in a previous assignment in 2024, where I was working on a POC that required DBT and Dagster Labs  integration for ETL pipeline orchestration. In that project, I worked mainly on DBT, and we selected Dagster Labs for the orchestration purpose so that we could create an ETL pipeline with orchestration and scheduling jobs.

The main use case I worked on for Dagster Labs was the DBT with Dagster Labs integration, where all our ETL jobs were built into DBT. There was a need to create a lineage and also the orchestration, and for the orchestration purpose, we used Dagster Labs.

Dagster Labs helped with lineage and orchestration in my project by allowing us to see what the impacted upstream and downstream systems were, which was a limitation in DBT. It enabled us to create dependencies in DBT, and I saw the flexibility of Dagster Labs where we could directly integrate it seamlessly with the DBT tool. All the ETL jobs could be imported directly into Dagster Cloud, and we could create the dependencies and schedule the jobs based on the frequency decided by the business, and it was quite seamless.

The best features Dagster Labs offers mainly include error handling, job scheduling, and lineage visualization, with all three being quite easy to manage. After fixing the code, we do not have to rerun the pipeline again; it automatically restarts. There is also an option to put data validation and data contracts within Dagster Labs. If there is any data quality issue that happens from the source, we can put the data validation checks over there, and Dagster Labs can easily check that and based on the results or the outcome of it, Dagster Labs can run the pipeline. The data lineage part is quite visible, making it easy to see what is the source, what is the target, and what are the in-between intermediate layers we have put in place for the pipeline.

The data contract or data validation part is a great feature because it has enabled us to do our quality checks at the runtime and that has saved a lot of manual efforts from the engineering team.

Dagster Labs has impacted my organization positively by enabling faster project delivery since it has reduced a lot of manual efforts, especially the manual scheduling part. With Dagster Labs, we have scheduled all our jobs and orchestration automatically, and we did not have any data lineage tool or orchestration tool before. After onboarding Dagster Labs, we have achieved some good results.

In terms of project delivery, it has reduced the effort significantly because a lot of manual things were automated, resulting in our timelines being met on time. Earlier, it was getting delayed, but we have made our deliveries right on time, so that was the really outstanding outcome.

There are multiple products from Dagster Labs: Dagster Cloud, Dagster Labs, and sometimes it is quite confusing to choose which one for what purposes. I know there are some licensing buckets for small organizations, medium organizations, and bigger organizations, so the modeling and the cost estimation part could be more intuitive, making it easier for beginners to understand how much time and money they would save by onboarding Dagster Labs in their project.

I deducted two points because I faced some challenges working with the Git  repository integration with Dagster Labs due to some security keys or some weird issue. I had to get in a call with Dagster Labs subject matter expert, and we really struggled for two to three weeks because we could not establish the secure connection through that.

I have been working in the Data and AI industry for almost 14 years now.

Dagster Labs is quite stable.

Dagster Labs is quite scalable, allowing us to scale it to enormous sizes.

The customer support was good; I had a weekly call with them when I was running into some issues, and they helped me out promptly.

I would rate the customer support nine out of 10.

We were not using any previous solution, so that was the first experience, having onboarded Dagster Labs.

The initial setup is quite good. I am really impressed.

The documentation is quite clear, and the integration is seamless as I mentioned, such as with DBT, which was good. The user experience was also good.

I do not have the numbers with me, but we have definitely reduced the manual efforts, resulting in one or two fewer employees needed and some productive tasks being accomplished. We have saved some time over there.

The pricing, setup cost, and licensing were quite intuitive but not as clear for the data architect to understand what would be the cost estimation when onboarding Dagster Labs.

We were thinking of using Airflow , but after comparing it with Dagster Labs, Dagster Labs came out as the winner, leading us to decide to go ahead with it.

One of the user experiences was that earlier DBT was a back-end tool, and Dagster Labs is something we can actually show to stakeholders with all the job run logs. It is easy to monitor all your running jobs and also what has been completed and what has errored out, and also the error status messages are quite descriptive. If there is any error that occurs, you can look into the error log details and then easily debug that and identify the error and fix it. If there is any failure that happens at one checkpoint, we do not need to restart the job from the start; from that checkpoint itself, we can resume the job after fixing the code, and it was quite seamless.

I would advise others to start using Dagster Labs and explore their functionalities and features because it is easy to onboard, and their cloud-native features are really awesome. It has a great lineage with all the ETL tools and also with GCP  and other cloud platforms, so I would definitely recommend others to start using it.

From the security perspective, everything was good, and I do not have any feedback on that, to be honest.

It met all the organization's requirements. I would rate this review an overall 8 out of 10.

The main use case for using Dagster Labs  is to utilize ETL processes such as extract, transform, and load because we already have raw data, and we perform all the necessary transformations, including feature engineering and mathematical calculations, and then we load it to the front end so that customers can see how their data has evolved into something useful, allowing them to make actionable decisions.

Let me consider a scenario where a customer has raw and unreadable data. We need to identify features first. For example, a customer may have IP addresses with last seen timestamps along with raw security logs. During the initial feature engineering phase, we check for use cases such as detecting if a user on the customer side is using unauthorized resources. To accomplish this, we need to perform feature engineering on several attributes, such as the user's authorization level and the objects they are accessing. For a second example, we can detect if a user is authorizing company access after work hours by using the last seen attribute. We can perform feature engineering on such scenarios.

That covers my main use case, but we are also currently trying to use the DLT tool by using Dagster Labs  so that we can extract the data as well.

The best features Dagster Labs offers are that it is very beginner-friendly, the UI is very nice, and we can backfill most of the processes. The backfill feature is the best feature I would consider.

When any new user comes in, they can really see the UI and do all sorts of things rather than going into code-level work. When it comes to backfill, if an error occurred in the past and we want to retrieve the data from that point, we can do that using backfill, which makes it good for our workflow.

Dagster Labs has positively impacted our organization by making it much easier to create data pipelines, helping us create new use cases, and allowing us to transform all of the data we have into features, which Dagster Labs has greatly assisted with.

I can say that since using Dagster Labs, we have saved time, increased productivity, and also cut down on some third-party vendor applications, which saves us money in infrastructure costs and other expenses.

Dagster Labs is currently providing a Dagster University course, but those offerings are somewhat high-level. When a user comes in and tries to learn Dagster Labs, there need to be different kinds of projects that they can understand. I do not think that Dagster University course has that.

Also, when we are doing something in Dagster Labs, we need to know that when I make a change in the UI, it needs to be reflected in the code, so as a developer, I can check on the code and understand the workflow. The UI needs to be connected to the code in real-time.

I have been using Dagster Labs for the past four to five months.

Dagster Labs is stable.

Dagster Labs's scalability is good because you can easily create different pipelines for different purposes, which is beneficial.

I would advise others looking into using Dagster Labs to utilize all of the features provided by Dagster Labs, as it has many great offerings. I would rate this product a seven out of ten.