Databricks Data Intelligence Platform

What do you like best about the product?

I've been using Databricks as part of our data engineering workflow to build and maintain ETL pipelines, analyze large datasets, and support reporting requirements. One of the things I like most is that it brings data engineering, analytics, and notebooks into a single workspace. Instead of switching between multiple tools, I can write PySpark code, validate transformations, collaborate with teammates, and schedule jobs from the same platform. This has made day-to-day development more organized, especially when working on multiple data pipelines.

Another feature I rely on frequently is the notebook environment. It's convenient for developing and testing transformations before moving them into production. During development, I often use notebooks to inspect sample data, troubleshoot failed transformations, and validate business logic with SQL and PySpark. The ability to mix code, markdown documentation, and query results in one place also makes it easier for team members to understand the implementation during code reviews or knowledge transfer sessions.

I also appreciate the platform's scalability. Some of our data processing jobs involve millions of records, and Databricks handles distributed processing efficiently without requiring us to manage the underlying infrastructure directly. Features like cluster management, job scheduling, and integration with cloud storage reduce operational overhead. That said, cluster startup times can occasionally delay quick debugging sessions, and managing compute resources carefully is important to avoid unnecessary costs. Overall, Databricks has helped simplify large scale data processing while giving enough flexibility for both development and production workloads.

What do you dislike about the product?

While Databricks has been reliable for our data engineering workloads, there are a few areas where I think it could be improved. One challenge I've experienced is cluster startup time. When I only need to test a small code change or validate a transformation, waiting for a cluster to start can interrupt the development flow. It's not a major issue for scheduled production jobs, but during active development and debugging, those extra minutes add up.

Another limitation is cost management. Since compute resources are tied to cluster usage, it's important to monitor cluster configurations and ensure they are shut down when not needed. We've had situations where development clusters remained active longer than expected, resulting in higher cloud costs. The platform provides tools to manage this, but it still requires teams to establish good governance and usage policies. I also found that some configuration settings for jobs, permissions, and clusters have a learning curve, especially for new team members who are unfamiliar with the Databricks environment.

From a day to day perspective, debugging distributed Spark jobs can sometimes be challenging. While the logs provide useful information, identifying the exact cause of failures often requires navigating through multiple execution logs and Spark UI details. For straightforward issues this isn't a problem, but troubleshooting more complex pipeline failures can take time. Despite these limitations, none of them outweigh the benefits of the platform, and most challenges can be managed with proper cluster configuration, monitoring, and team practices.

What problems is the product solving and how is that benefiting you?

Databricks has helped address one of the biggest challenges in our data engineering workflow: processing and transforming large volumes of data efficiently. Before the data reaches reporting or downstream applications, we need to ingest data from multiple sources, apply business rules, clean inconsistent records, and create curated datasets. Databricks provides a single platform where we can develop, test, and run these data pipelines using PySpark and SQL instead of managing multiple disconnected tools. This has made our development process more consistent and easier to maintain.

A practical example is one of our daily ETL pipelines that processes data from different source systems before loading it into curated tables for reporting. We use Databricks notebooks during development to validate transformations on sample data and then schedule the same logic as production jobs. If a pipeline fails, the job history and execution logs help us identify the stage where the failure occurred, making troubleshooting more efficient than manually tracing scripts across different servers. Having notebooks, job scheduling, and cluster management in one platform has reduced the effort required to manage these workflows.

From a business perspective, the biggest benefit is faster availability of reliable data for reporting and analytics. Our team spends less time managing infrastructure and more time implementing business logic and improving data quality. While optimizing Spark jobs and monitoring cluster costs still require attention, Databricks has streamlined our daily workflow by providing a scalable environment for developing, testing, and running data pipelines. This has improved collaboration within the team and made it easier to deliver data that downstream users can trust.

What do you like best about the product?

It easily pulls data from the API, and within the same dataset we can manage our collections. We also have the option to write code using the AI.

What do you dislike about the product?

In our current setup, BigQuery SQL queries run with predictable costs that are easy to control. With Databricks, though, if a data engineer spins up an oversized cluster or leaves a node running after processing dealer posts or telematics logs, compute costs can ramp up quickly and may go unnoticed.

What problems is the product solving and how is that benefiting you?

For our projects, we use it to pull the source, or raw, data from the APIs and then transfer that same data into BigQuery. It essentially acts as a middleman for us.

What do you like best about the product?

I usually use it to make integrations between different data sources.

What do you dislike about the product?

I don't like the Genie Code, it's not good and I prefer to rely on other AI tools.

What problems is the product solving and how is that benefiting you?

The main issues we have solved so far are that we have migrated several workflows from another tool to Databricks, and the execution time has decreased considerably.

What do you like best about the product?

I really appreciate how Databricks earns its place in my work as the analytics tier for the reference architectures I propose to customers. It simplifies the process by allowing telemetry from distributed device fleets to land directly, enabling me to model aggregation, anomaly detection, and historical comparison without needing separate infrastructure. The platform's cost efficiency is a decisive factor for me, and it delivers more value than its price suggests. Retrieval speed is impressive, letting me query a warehouse live during design sessions without relying on others to run the request. The setup required far less scaffolding than anticipated, positively impacting how I prototype. Starting with Databricks didn't involve the tedious configuration I expected from a platform of its scale, so the environment was productive almost immediately. Also, the platform significantly shortens the path from an idea to a prototype that clients can see, thanks to its low cost, quick retrieval, and minimal setup requirements.

What do you dislike about the product?

Handling of unstructured content remains the weak point in my view, since building a usable search layer over message archives, PDF documents and finance spreadsheets proved awkward enough that I compromised on the design. Vector search performance disappoints as well: direct lookups return promptly, yet reasoning queries slow noticeably, which becomes apparent the moment a client is watching a demonstration.

What problems is the product solving and how is that benefiting you?

I use Databricks as the analytics tier to model aggregation and anomaly detection without extra infrastructure. It allows quick data retrieval, enabling live queries during sessions, with minimal setup and low cost, transforming how I prototype and shortening the time to show customers working solutions.

What do you like best about the product?

I really appreciate Databricks' ability to consolidate scattered infrastructure data like permitting records and contractor reporting into a single workspace, which makes spotting compliance gaps and making strategic decisions much easier. The broad data source connectivity is also a highlight for me because it bridges legacy systems with much less friction than I expected. Having Genie to pair with it has been a game changer, allowing me to build, test, and deploy data objects quickly, keeping my analysis moving without stalling. The initial setup was surprisingly simple, taking just a few hours with no major problems. These features make Databricks valuable for someone like me who needs reliable data more than deep engineering expertise.

What do you dislike about the product?

My one reservation concerns the Python editor. It serves its purpose, yet it could stand to be more robust. A richer editing experience would make the occasional bit of custom scripting feel less cumbersome when I am refining a data object. I'd like to see richer code intelligence in the Python editor, things like smarter autocomplete, inline error highlighting, and better debugging tools, so that when I write custom scripts to shape a data object, the process feels smoother and I catch mistakes before running the code rather than after.

What problems is the product solving and how is that benefiting you?

I use Databricks to consolidate infrastructure data into one workspace, spot compliance gaps, and make informed strategic decisions. Its broad connectivity and integration with Genie streamline data analysis, enabling me to focus on governance and risk evaluation without technical hurdles from fragmented legacy systems.

What do you like best about the product?

I really like the power of its engine, since it’s based on Apache Spark, especially when processing large volumes of telemetry, diagnostic logs, or performance metrics from embedded devices during stress tests. I also appreciate the support for collaborative notebooks in Python and Scala, which makes data exploration, building fast ETL pipelines, and running models much easier. The integration with Unity Catalog for governance is good as well; I like it.

What do you dislike about the product?

I don’t like the cost model because if I’m not careful with the cluster size or query optimization, the bill at the end of the month can be quite a surprise.

What problems is the product solving and how is that benefiting you?

Databricks is excellent, and my experience has been very positive. It handles an industrial volume of data that justifies its use, and it lets me transform scattered data into useful information much more efficiently. It’s a truly excellent cluster optimization tool, and it has saved us countless hours of manual processing.

What do you like best about the product?

What I like most about Databricks is that it helps our team monitor rider service operations more efficiently. We use Databricks Dashboards and SQL Warehouses to track daily revenue, service requests, customer satisfaction scores, completed services, and failed transactions in one place. It has made operational reporting much faster and allows us to identify trends without manually preparing multiple reports.

What do you dislike about the product?

One thing I found slightly challenging with Databricks was understanding how different components such as SQL Warehouses, dashboards, and permissions work together when building operational reports. Setting up dashboards for rider service metrics took some time initially. After getting familiar with the workflow, it became much easier to manage, but better onboarding guidance for first-time users would improve the overall experience.

What problems is the product solving and how is that benefiting you?

Databricks helps us centralize rider service operations analytics in a single workspace. We use it to track daily revenue, service requests, customer satisfaction scores, completed services, and failed transactions through operational dashboards. Before using Databricks, preparing reports across multiple datasets was more time-consuming. Having these business metrics available in one place has significantly improved reporting efficiency and helped our team identify operational trends much faster.

What do you like best about the product?

Its platform is great for cluster autoscaling when dealing with large Delta Lake operations. Our complex aggregations that rely on distributed queries are benefiting significantly from this optimization, which means they run much faster in the Photon engine. When using spark tables with Unity Catalog, you gain granular control over them and you can easily track the data lineage.

What do you dislike about the product?

There is limited integration with Git for notebook versions and merge conflicts occur when multiple people are working on the same version of a notebook. When displaying large amounts of data in workspace cells, the browser tab often freezes.

What problems is the product solving and how is that benefiting you?

Our team had to break free from data silos between data engineering and machine learning pipelines. Our raw storage and predictive models are combined in one managed infrastructure by Databricks.

What do you like best about the product?

I use Databricks to manage workflows and automate tasks, which has been really productive for my team over the past two years. It handles visualization tasks and manages different functions smoothly, with a high response time. I prefer Databricks for documentation, automation, data mining, and lead gen analysis, which helps in getting a higher ROI by syncing different tasks within one main parent workflow. Its speed and ease of use, along with no credit-based system, multiple integrations, and easy data extraction and sharing capabilities make it a must-have tool.

What do you dislike about the product?

I think there are not a lot as it is one of the best firm but maybe a better informational content can help in easy education as compared to the onboarding process we had but had to watch multiple videos on YouTube and read articles by the firm after it's launch and we went live with the product.

What problems is the product solving and how is that benefiting you?

I use Databricks for managing workflows and automating tasks, solving visualization issues, syncing tasks into a parent workflow, and integrating with tools. It enhances productivity, offers high response time, and supports data mining and lead gen analysis.

What do you like best about the product?

What I like most about Databricks is its unified platform for data engineering, analytics, and machine learning. It makes working with large datasets much easier and allows different teams to collaborate in a single environment. I especially like the notebook-based workspace, its scalability, and its integration with Apache Spark.

What do you dislike about the product?

What I dislike about Databircks is that the platform can come with a learning curve for new users, especially when you’re dealing with advanced configurations and large-scale data pipelines. Managing compute resources and keeping costs under control can also take careful monitoring and ongoing attention.

What problems is the product solving and how is that benefiting you?

It helps solve the challenge of managing, processing, and analyzing large volumes of data across different workflows. It brings data engineering, analytics, and machine learning into a single platform, which reduces the need to switch between multiple tools and makes it easier to keep work consistent from end to end.