Databricks Data Intelligence Platform

What do you like best about the product?

The unified platform experience is what keeps me on Databricks. Having notebooks, pipelines, SQL warehouses, ML, and governance all in one place under Unity Catalog means I’m not constantly stitching together five different tools just to get work done.

Lakeflow Pipelines (formerly DLT) makes it straightforward to build medallion-architecture pipelines, and the Photon engine delivers real performance gains on SQL workloads without requiring any code changes. Recent additions like Genie Code and background agents also show they’re serious about agentic AI—it doesn’t feel like a bolt-on copilot, because it can actually understand your data context through Unity Catalog. Serverless compute has been another big quality-of-life improvement as well, since I no longer have to wait for cluster spin-up when I just want to run quick, ad hoc queries.

What do you dislike about the product?

Cost management can be tricky—DBUs add up quickly if you’re not careful with cluster sizing and warehouse auto-scaling. The pricing model also isn’t always transparent, especially when you’re mixing serverless and classic compute.

Unity Catalog is powerful, but the initial setup and the migration from legacy HMS can be painful, particularly for large orgs with years of existing Hive metastore objects. The documentation is generally good, yet it sometimes lags behind new feature releases. On top of that, the workspace UI can feel sluggish at times, especially when you’re working with a large number of assets.

What problems is the product solving and how is that benefiting you?

Before Databricks, our data stack was fragmented — separate tools for ETL, analytics, ML, and governance. That meant constant context-switching, duplicated data, and governance gaps. Databricks consolidates all of that into one lakehouse platform. Delta Lake gives us reliable ACID transactions on the data lake, Unity Catalog handles lineage and access control across the board, and SQL warehouses let our analysts self-serve without needing a separate data warehouse product. It's cut our pipeline development time significantly and made data governance something we can actually enforce consistently instead of hoping for the best.

What do you like best about the product?

The ability to converge data engineering, data science, and analytics on a single platform without compromising on governance, performance, or flexibility is still rare in the industry. Databricks executes this exceptionally well.

What do you dislike about the product?

Reducing the spinning time of all purpose clusters and job clusters. It would be more usefula nd helpful if it starts as quick as serverless

What problems is the product solving and how is that benefiting you?

In enterprise banking, where regulatory compliance, data accuracy, and operational resilience are non-negotiable, Databricks is solving some of our most critical challenges. As a Lead Data Engineer managing end-to-end ETL pipelines, dashboard delivery, monitoring alerts, and data governance for a major banking client, the platform has become the backbone of our modern data architecture. Databricks unifies our fragmented data landscape through Delta Lake and Unity Catalog, giving us ACID-compliant transactions for reliable ETL, automated lineage for audit-ready governance, and fine-grained access controls to protect sensitive PII and financial data—all while enabling seamless schema evolution to handle the constant changes in source systems. This directly translates to faster, more trustworthy reporting: our dashboards in Power BI and Tableau now pull from a single source of truth, eliminating metric disputes between Risk, Finance, and Compliance teams. On the operational side, native alerting integrated with Slack and PagerDuty, combined with Databricks System Tables for observability, lets us proactively catch data quality issues or SLA breaches before they impact business decisions—reducing incident resolution time by over 60%. Ultimately, Databricks isn't just improving our engineering efficiency; it's enabling us to innovate responsibly in a highly regulated environment, delivering trusted insights at scale while keeping auditors confident and stakeholders aligned.

What do you like best about the product?

What I like most about Databricks is how it brings data engineering, analytics, and machine learning together in one platform. It streamlines the entire data pipeline—from ingestion and transformation through to serving—so I don’t have to rely on multiple separate tools to get end-to-end workflows done.

Its integration with Spark and Delta Lake is another big plus, making it both scalable and dependable when working with large datasets.

What do you dislike about the product?

One challenge with Databricks is cost management and visibility. Since compute is abstracted through clusters and jobs, it can sometimes be difficult to track and optimize costs without additional monitoring or governance in place.

What problems is the product solving and how is that benefiting you?

Solves the problem of fragmented data ecosystems, where data engineering, analytics, and machine learning are handled in separate tools.

What do you like best about the product?

Databricks is hands down my favorite platform for data engineering because it brings everything together in one place Spark processing, Delta Lake, and ML tools all play nice without the usual headaches. The auto-scaling clusters save tons of time on big ETL jobs, like the SAP integrations I've done, letting me focus on logic instead of babysitting resources. Unity Catalog has been a game changer for governance in our lakehouse setups too.

What do you dislike about the product?

Costs can sneak up fast if you're not watching usage closely, especially with premium features on large pipelines. The notebooks are great for prototyping but get messy in production without strict discipline. Setup for advanced stuff like custom Unity Catalog policies sometimes feels overly complex for what it delivers.

What problems is the product solving and how is that benefiting you?

Databricks tackles key data engineering headaches like scaling massive Spark jobs, data quality issues, and siloed teams by providing a unified lakehouse platform with Delta Lake for ACID transactions and reliable pipelines. When I have a large number of files or tables to process like in supply chain ETL from SAP systems it shines with optimized Delta processing, serverless compute, and Photon engine, slashing run times from days to hours while cutting costs through auto-scaling. This benefits me directly by speeding up project delivery, reducing debugging time on failures, and enabling seamless collaboration with analysts on notebooks without tool switches.

What do you like best about the product?

Genie code and the inline Assistant were the most helpful tools for me on my project. They helped me debug a 2k-line codebase and clearly explained why I wasn’t getting accurate data. It also provided a query to run in my source system (SQLMI). By running the discrepancy script in parallel on the source and target, I was able to debug the entire code much faster and improve my productivity. Overall, it cut my work time from about 8 hours down to around 1 hour.

What do you dislike about the product?

In Delta Sharing, there isn’t a catalog-level SELECT permission, and I sometimes think having that would be helpful. Also, when I use the Genie code inside a VM, it can make the website unresponsive at times. These are areas that could be improved.

What problems is the product solving and how is that benefiting you?

In one of our claims-processing migration projects, the client needed near real-time data availability for downstream applications. Previously, the architecture used Amazon Redshift as the data warehouse, with Jasper and Sisense consuming the data for reporting and analytics. However, that setup didn’t support real-time or near real-time streaming efficiently, which led to delays in data availability for downstream systems.

After migrating the platform to Databricks, we were able to substantially improve the data pipeline architecture. We implemented streaming along with optimized ETL pipelines, reducing the data refresh cycle to about 30 minutes. We also created a dedicated view that retains data from the previous run, so downstream systems always have a consistent dataset available while the next pipeline execution is still in progress.

Before, we struggled with delayed refresh cycles and a limited ability to meet near real-time data needs in our Redshift-based architecture. After moving to Databricks, we enabled faster ETL processing and improved near real-time data availability.

As a result, we reduced ETL refresh time to roughly 30 minutes and enabled near real-time access for downstream tools like Jasper and Sisense. Reliability also improved because the stable view continues to serve the previous run’s data during pipeline updates. Finally, the overall architecture became simpler by consolidating processing and analytics capabilities within Databricks.

Overall, Databricks helped us build a more scalable and efficient near real-time data processing platform, significantly improving the timeliness and reliability of analytics for the claims-processing workflow.