Databricks Data Intelligence Platform

What do you like best about the product?

Ease of working with data, with scalability and high processing power. It has many integrations, which facilitates access to data from different sources. Furthermore, it is natively prepared for AI.

What do you dislike about the product?

The interface is technical and the price is steep for some products.

What problems is the product solving and how is that benefiting you?

Democratize the data in different areas and decentralize the processing, making access broader and distributing activities better.

What do you like best about the product?

I've been using Databricks as part of our data engineering workflow to build and maintain ETL pipelines, analyze large datasets, and support reporting requirements. One of the things I like most is that it brings data engineering, analytics, and notebooks into a single workspace. Instead of switching between multiple tools, I can write PySpark code, validate transformations, collaborate with teammates, and schedule jobs from the same platform. This has made day-to-day development more organized, especially when working on multiple data pipelines.

Another feature I rely on frequently is the notebook environment. It's convenient for developing and testing transformations before moving them into production. During development, I often use notebooks to inspect sample data, troubleshoot failed transformations, and validate business logic with SQL and PySpark. The ability to mix code, markdown documentation, and query results in one place also makes it easier for team members to understand the implementation during code reviews or knowledge transfer sessions.

I also appreciate the platform's scalability. Some of our data processing jobs involve millions of records, and Databricks handles distributed processing efficiently without requiring us to manage the underlying infrastructure directly. Features like cluster management, job scheduling, and integration with cloud storage reduce operational overhead. That said, cluster startup times can occasionally delay quick debugging sessions, and managing compute resources carefully is important to avoid unnecessary costs. Overall, Databricks has helped simplify large scale data processing while giving enough flexibility for both development and production workloads.

What do you dislike about the product?

While Databricks has been reliable for our data engineering workloads, there are a few areas where I think it could be improved. One challenge I've experienced is cluster startup time. When I only need to test a small code change or validate a transformation, waiting for a cluster to start can interrupt the development flow. It's not a major issue for scheduled production jobs, but during active development and debugging, those extra minutes add up.

Another limitation is cost management. Since compute resources are tied to cluster usage, it's important to monitor cluster configurations and ensure they are shut down when not needed. We've had situations where development clusters remained active longer than expected, resulting in higher cloud costs. The platform provides tools to manage this, but it still requires teams to establish good governance and usage policies. I also found that some configuration settings for jobs, permissions, and clusters have a learning curve, especially for new team members who are unfamiliar with the Databricks environment.

From a day to day perspective, debugging distributed Spark jobs can sometimes be challenging. While the logs provide useful information, identifying the exact cause of failures often requires navigating through multiple execution logs and Spark UI details. For straightforward issues this isn't a problem, but troubleshooting more complex pipeline failures can take time. Despite these limitations, none of them outweigh the benefits of the platform, and most challenges can be managed with proper cluster configuration, monitoring, and team practices.

What problems is the product solving and how is that benefiting you?

Databricks has helped address one of the biggest challenges in our data engineering workflow: processing and transforming large volumes of data efficiently. Before the data reaches reporting or downstream applications, we need to ingest data from multiple sources, apply business rules, clean inconsistent records, and create curated datasets. Databricks provides a single platform where we can develop, test, and run these data pipelines using PySpark and SQL instead of managing multiple disconnected tools. This has made our development process more consistent and easier to maintain.

A practical example is one of our daily ETL pipelines that processes data from different source systems before loading it into curated tables for reporting. We use Databricks notebooks during development to validate transformations on sample data and then schedule the same logic as production jobs. If a pipeline fails, the job history and execution logs help us identify the stage where the failure occurred, making troubleshooting more efficient than manually tracing scripts across different servers. Having notebooks, job scheduling, and cluster management in one platform has reduced the effort required to manage these workflows.

From a business perspective, the biggest benefit is faster availability of reliable data for reporting and analytics. Our team spends less time managing infrastructure and more time implementing business logic and improving data quality. While optimizing Spark jobs and monitoring cluster costs still require attention, Databricks has streamlined our daily workflow by providing a scalable environment for developing, testing, and running data pipelines. This has improved collaboration within the team and made it easier to deliver data that downstream users can trust.

What do you like best about the product?

many features with intuitive ui, great ai assistance and you easily integrate with other services. I like the facts that yo can orchestrate jobs and notebooks easily, run queries on large volumes of data and the fact that the AI genie can help you a lot understand and implement better and faster tasks.

What do you dislike about the product?

sometimes the documentation is not easily obtainable. I have also come across cases where the catalogue quick search did not yield my table, but the table existed. generally I am happy

What problems is the product solving and how is that benefiting you?

doing large scale queries, making it fast and easy to generate reports and dashboards for both technical and non technical people. bridges the gap between tech and product/business

What do you like best about the product?

It easily pulls data from the API, and within the same dataset we can manage our collections. We also have the option to write code using the AI.

What do you dislike about the product?

In our current setup, BigQuery SQL queries run with predictable costs that are easy to control. With Databricks, though, if a data engineer spins up an oversized cluster or leaves a node running after processing dealer posts or telematics logs, compute costs can ramp up quickly and may go unnoticed.

What problems is the product solving and how is that benefiting you?

For our projects, we use it to pull the source, or raw, data from the APIs and then transfer that same data into BigQuery. It essentially acts as a middleman for us.

What do you like best about the product?

I usually use it to make integrations between different data sources.

What do you dislike about the product?

I don't like the Genie Code, it's not good and I prefer to rely on other AI tools.

What problems is the product solving and how is that benefiting you?

The main issues we have solved so far are that we have migrated several workflows from another tool to Databricks, and the execution time has decreased considerably.