Dataiku Trial

My main use case for Dataiku  is general; I create ETL pipelines and then automate everything using that, along with ML modeling. These are the major use cases that I have for Dataiku . On a daily basis, we use Dataiku for ad hoc analysis for following the product lifecycle.

For one of my use cases with Dataiku, we are using it where the data resides in Snowflake  and the expectation is to orchestrate and automate a complete CI/CD pipeline, with the final data residing on S3 . In between, there are multiple logics and transformations that we have to build in. Along with that, we are supposed to do all the DQ checks, data quality framework, and data governance. We automated everything using Dataiku, and now the project is live, with overall efficiency being very good.

Automating that workflow with Dataiku increased the overall productivity of the team compared to the tasks that we used to do earlier using other ETL tools. Dataiku has optimized that, and data visualization became easy. The checkpoints that Dataiku provides, such as analyzing the data and finding the outliers, became easy, and sharing the data sets became easy as well. Now, with the visual recipes, even people who can't code can also do the transformation, so overall, it is a good tool.

We have created a few visual recipes that are not only limited to the project; we created a package so that they don't have to code that part of logic again and again. We have provided them as a recipe, which is a good thing.

The best features Dataiku offers include the data analysis part, the ETL, and the overall orchestration part. We can create a recipe and share it with others without having to code that again and again, and we can create an application and a dashboard in one single place. These are the very good features of Dataiku.

I find myself and my team relying the most on the data analysis part of Dataiku. We use it to visualize the data, find the outliers, and it helps us very well.

Dataiku has positively impacted my organization as most of our projects have been migrated to Dataiku, and now people are relying on it as a go-to tool for all our data use cases. This migration has led to measurable improvements, as most of the projects have been migrated and the overall efficiency has increased. Most people who used to do tasks manually are now working on automating that.

Dataiku can be improved from the dashboard perspective because right now it is very restricted, and I feel that can be improved. API integration and other aspects can also be enhanced, but I am pretty impressed with the rest of it.

I have been using Dataiku for four years now.

Dataiku is stable.

Dataiku's scalability is pretty good; I can scale the projects very easily, and clear guidance is given as well. I have no issues with that.

I need to stress upon the part about customer support because there are some product issues we have identified and raised with customer support, but sometimes the response is delayed, so that can be improved.

I previously used a different solution before Dataiku, and the other solution was not cloud-based; they were local, which made the license cost higher.

My experience with pricing, setup costs, and licensing is good because that was managed by my IT team, and overall it was seamless with clear guidance given.

We have a direct link with Dataiku; we did not purchase it through the AWS Marketplace .

I cannot share the numbers regarding return on investment.

My experience with pricing, setup costs, and licensing is good because that was managed by my IT team, and overall it was seamless with clear guidance given.

Before choosing Dataiku, I evaluated other options, specifically Databricks .

My advice to others looking into using Dataiku is to first understand the product, which is very important. You should first see what your use case is, what Dataiku is offering, and understand that it is a tool meant not only for coders but also for higher management, as they can do drag and drop to easily perform transformations without needing to write code. Dataiku is a tool for everyone. I would rate this product an 8.5 out of 10.

My main use case for Dataiku  is mostly based on the client's data where we look into life sciences data, mostly aligned to claims, campaign measurement, campaign targeting, IQVIA, LAD Epsilon  data, and Komodo for instance.

Apart from this, I'm basically working on creating an end-to-end pipeline as a bundled unit project, which has been the overall case. We primarily work on Next Best Engagement and Next Best Actions, more or less aligned to the healthcare side, while sometimes working on the consumer front and on the professionals front, meaning healthcare professionals (HCP ).

A specific project I built in Dataiku  was on HCP  campaign measurement. Our day-to-day cycle involves ingestion of data from our S3 , which is the client's S3  storage. We fetch the data, perform some visual recipes to bring it onto Dataiku DSS, make preliminary changes, preprocess the data, do some data preparation, and perform feature engineering to have the final model ready dataset for modeling.

We create multiple iterations of the model where Dataiku is of great help, allowing us to try multiple modeling iterations with different hyperparameters, saving a lot of time and providing a visual overview for everyone to understand how the data is performing. Once the modeling is done, we push the data downstream through an API or use MLOps for productionization, either via CI/CD pipeline or just simple scenario triggers such as sending an email once a job gets done. This primarily results in our day-to-day activity.

The best features Dataiku offers include primarily the visual recipes, which ease data preparation greatly. It is very easy now to handle small tasks where you need to understand the shape of data; instead of writing a query, you can just use a visual recipe to create the views. You can also have multiple intermediate views, which is significantly helpful for larger streams, especially during reverse engineering.

Additionally, the automation piece and scenario triggering has been a boon for me, as my projects often involve weekly or monthly reporting. Everything is set up so that we just need a human in the loop to ensure everything follows properly, with time-based triggers automatically generating and sending reports to stakeholders.

Furthermore, the integration capabilities and the ability for multiple team members to access the same projects concurrently enhance collaboration, making it quite beneficial for data scientists such as myself as we progress in our careers.

Dataiku has positively impacted my organization, specifically in one project where we performed migration from AWS  to Dataiku, speeding up the solution by close to 40%. We completed tasks that used to take 10 days in just four days. Moreover, the architecture costs associated with AWS  were reduced by almost 70%, which was a significant benefit and greatly impacted our operations. This success has enabled us to pitch Dataiku to clients, who have actively incorporated it into their daily work streams, resulting in a win-win situation.

The 70% cost reduction and 40% faster delivery came primarily from the ease of use in how we were creating architecture. Since we were migrating, we leveraged the opportunity to improve and enhance the architecture. The earlier AWS architecture was hampered by multiple services leading to high costs, but moving to Dataiku streamlined everything into one platform. Consequently, the delivery time for generating reports for stakeholders decreased from 10 days to three to four days.

In terms of improvement, I cannot comment on the LLMs or the agentic view as I have not used them yet. However, I feel that better documentation is necessary. Dataiku should establish a stronger community since this is proprietary software, where users can share knowledge. Although they have some community interaction, it is often challenging to find assistance when stuck.

For example, when I was new to Dataiku and trying to use an external optimization tool such as CPLEX, I struggled with resource directory linking to a project's notebook. Detailed documentation and community discussions could have significantly alleviated these issues for users such as myself.

I have been using Dataiku for close to three and a half years.

There were a few challenges, but they were not from Dataiku's standpoint in terms of technicality; they were more related to the rapid updates where we currently work on version 10.2, and soon, we are on version 11.4, requiring things to be redone. The support for earlier projects created on the older version is something the team could look at, as it would help if there was a backup proposition in place to avoid hampering our work due to updates.

While I do not have a particular feature that surprised me, I found the plugins available in Dataiku to be very helpful. Not only can users leverage existing plugins, but we can also create our plugins based on the rules we use daily. This feature is quite handy and extends beyond just individual projects, as published plugins can be used by everyone across the board.

My advice to others considering Dataiku is to utilize the visual recipes, as they can significantly expedite project creation. Although the fundamental processes remain the same, leveraging elements in visual recipes can enhance efficiency, making it easier than writing code for basic queries, resulting in quicker execution. Dataiku encompasses everything from visualization to integrations and sharing the results, so once you dive in, it is important to familiarize yourself with the available features and make the most out of them.

I would rate this product a 9 out of 10.