Databricks Data Intelligence Platform

My usual use case for Databricks  as an end-user mostly involves exporting data. This typically entails writing directly into a web interface to get the data out, so probably with Python.

The most significant benefit Databricks  has brought to my company is the Unity Catalog. Previously, with our data warehouse, we weren't able to track where sensitive data was. The Unity Catalog has been a big improvement, even though we haven't gotten the rest right.

The user interface is very useful, especially in writing directly into a web interface.

From my perspective, the ability to export data effectively and use Python within Databricks are key valuable features.

I believe we could improve Databricks integration with cloud service providers. The impact of our current integration has not been particularly good, and it's becoming very expensive for us. The inefficiencies in our implementation, such as not shutting down warehouses when they're not in use or reserving the right number of credits, have led to increased costs.

We made several beginner mistakes, such as not taking advantage of incremental loading and running overly complicated queries all the time. We should be using ETL tools to help us instead of doing it directly in Databricks. We need more experienced professionals to manage Databricks effectively, as it's not as forgiving as other platforms such as Snowflake .

I think introducing customer repositories would facilitate easier implementation with Databricks.

I have been working with Databricks for the last six months.

As a platform, Databricks is fine. However, our implementation isn't particularly reliable.

We've suffered from the lack of professionals with previous experience, which makes it difficult to dig ourselves out of the situation we've found ourselves in.

The scalability level of Databricks at the moment exceeds our needs. It's not a problem for us.

The sky's the limit with Databricks.

I have addressed technical support about our issue with Databricks. It was the team that engaged with them, and I believe our development teams also reach out for support, though I'm not sure what level of support they get.

Previously, when using Snowflake , we had customer reps who were really knowledgeable and helped us to avoid beginner mistakes. With Databricks, it seems we could have benefited from similar support. Our implementation team had no experience and made obvious mistakes. It may be that we opted not to have that support, but I believe we should have.

Before Databricks, I used SQL Server .

The big decision to switch from SQL Server  to Databricks was motivated by the lack of auditing, lineage, and tracking sensitive data in SQL Server, along with a need for more flexibility.

I did not participate in the initial setup of Databricks.

We use a consultancy, Avanade, for our Databricks implementation. They had previously done a Databricks implementation for another part of our organization. Our implementation team lacked experience which resulted in several beginner mistakes.

So far, we're not measuring any return on investment, such as saving time, money, or resources with Databricks. We're still in the phase where our old system and the new system are running simultaneously, so everything is twice as expensive and much effort is doubled. We haven't progressed far enough yet to realize any ROI.

I believe that in terms of credits for Databricks, we're spending between £15,000 and £20,000 a month.

I think Databricks is priced correctly. If we managed our resources better, we wouldn't be paying anywhere near that amount. The issue is with our management of resources.

No other options were considered because we used the consultancy Avanade, who had done a previous Databricks implementation for another part of our organization. We used them to recreate our implementation.

I'm probably not the best person to discuss certain aspects of Databricks since I haven't explored it deeply and am not part of the team developing it.

We haven't utilized Databricks' machine learning capabilities.

From my company, data ingestion and transformation are done with Databricks, though I don't do it directly.

I don't use Databricks' features for managing data, such as data lake and warehouse operations.

Most of our current work with Databricks isn't really live yet, so measuring savings in time and money or identifying any return on investment isn't applicable right now.

I would rate this review a 7 overall.