What do you like best about the product?

Genie code and the inline Assistant were the most helpful tools for me on my project. They helped me debug a 2k-line codebase and clearly explained why I wasn’t getting accurate data. It also provided a query to run in my source system (SQLMI). By running the discrepancy script in parallel on the source and target, I was able to debug the entire code much faster and improve my productivity. Overall, it cut my work time from about 8 hours down to around 1 hour.

What do you dislike about the product?

In Delta Sharing, there isn’t a catalog-level SELECT permission, and I sometimes think having that would be helpful. Also, when I use the Genie code inside a VM, it can make the website unresponsive at times. These are areas that could be improved.

What problems is the product solving and how is that benefiting you?

In one of our claims-processing migration projects, the client needed near real-time data availability for downstream applications. Previously, the architecture used Amazon Redshift as the data warehouse, with Jasper and Sisense consuming the data for reporting and analytics. However, that setup didn’t support real-time or near real-time streaming efficiently, which led to delays in data availability for downstream systems.

After migrating the platform to Databricks, we were able to substantially improve the data pipeline architecture. We implemented streaming along with optimized ETL pipelines, reducing the data refresh cycle to about 30 minutes. We also created a dedicated view that retains data from the previous run, so downstream systems always have a consistent dataset available while the next pipeline execution is still in progress.

Before, we struggled with delayed refresh cycles and a limited ability to meet near real-time data needs in our Redshift-based architecture. After moving to Databricks, we enabled faster ETL processing and improved near real-time data availability.

As a result, we reduced ETL refresh time to roughly 30 minutes and enabled near real-time access for downstream tools like Jasper and Sisense. Reliability also improved because the stable view continues to serve the previous run’s data during pipeline updates. Finally, the overall architecture became simpler by consolidating processing and analytics capabilities within Databricks.

Overall, Databricks helped us build a more scalable and efficient near real-time data processing platform, significantly improving the timeliness and reliability of analytics for the claims-processing workflow.

What do you like best about the product?

Databricks is great because it brings everything you need for data and AI into one place.
Instead of switching between different tools for data engineering, data cleaning, analytics, and machine learning, you can do it all in a single environment. That makes life a lot easier.

What do you dislike about the product?

Databricks is not beginner-friendly. You often need solid data engineering skills to use it effectively.
Reviews point out that while Databricks is extremely capable, it’s “a high‑end workshop” that requires expertise and is not easy for less technical teams.Databricks uses cost units (DBUs), which many people find difficult to estimate and manage.
Even expert reviews highlight that its pricing is famously complicated and can hide unexpected costs.

What problems is the product solving and how is that benefiting you?

Databricks uses the Lakehouse architecture to combine the strengths of data lakes and data warehouses into one unified platform. This means structured and unstructured data live together and are ready for analytics or machine learning.

What do you like best about the product?

What I like most about Databricks is how its features have consistently matched the evolving needs of engineering teams. Over the years, I’ve seen it grow from a solid data platform into a workspace that genuinely streamlines how we build and manage data and AI solutions. Unity Catalog has been one of the biggest improvements for us having a single place to manage permissions and lineage has removed a lot of manual steps we used to handle separately across systems. Genie AI and BI have also become part of my regular workflow; being able to generate SQL or explore datasets through natural conversations helps teams get to answers faster, especially when we’re under time pressure. The Apps capability has added unexpected value by letting us create and share simplified internal tools directly within the platform, eliminating the need to stand up extra infrastructure. And with Lakebase, we’ve been able to support more transactional-style use cases without losing the flexibility of a lake, which has made certain pipelines far easier to maintain. Altogether, these improvements have removed a lot of friction from day‑to‑day work and made the platform something I genuinely enjoy using as it continues to evolve.

What do you dislike about the product?

What I dislike about Databricks is that some of the newer AI experiences especially Genie for code generation can feel unstable at times and may lose context during longer development sessions. It disrupts my workflow when the assistant can’t retain earlier logic or maintain continuity across multiple iterations.

I’ve also noticed a gap in native connectors for certain enterprise systems like DFS, SMB shares or windows-based source systems, and platforms such as DB2 on AS/400, which many customers still rely on. Even though Databricks continues to expand its ecosystem, the lack of direct connectivity in these areas often means we need extra middleware or custom pipelines to bridge the gap.

None of these are deal-breakers, but they’re areas where the platform’s otherwise smooth experience can still feel a bit incomplete.

What problems is the product solving and how is that benefiting you?

Databricks has helped us address several long‑standing challenges in how we manage and deliver data and AI. Before adopting its newer capabilities, we were dealing with fragmented governance, duplicate datasets, and a lot of manual effort to keep permissions and lineage consistent across different systems. Unity Catalog improved this by giving us a single place to manage security and ownership, which reduced confusion across teams and noticeably cut down on rework during audits.

We also used to spend a significant amount of time helping teams explore data or draft queries. With Genie AI and BI, they can now generate SQL, summaries, and visual insights more independently. As a result, the time from a question to a usable answer has shortened, especially when we’re working under tight delivery cycles.

Another pain point was building small internal tools around our data. Setting up separate infrastructure or hosting environments created unnecessary overhead. With Databricks Apps, we can now build and share these tools within the platform itself, which saves setup time and reduces ongoing maintenance.

Finally, we struggled to support workloads that needed both the flexibility of a lake and the reliability of a database. Lakebase helped close that gap by enabling transactional‑style operations directly on our lake data, which simplified several pipelines and reduced the number of systems we have to maintain.

Overall, Databricks has moved us from juggling multiple disconnected tools to working in a more unified and predictable environment. That shift has sped up delivery, lowered operational overhead, and improved the clarity of our workflows.

What do you like best about the product?

Unified lakehouse architecture: Databricks lets me treat my data lake more like a “lakehouse,” combining data-lake flexibility with data-warehouse-like features such as ACID transactions, schema enforcement, and time travel on Delta tables. As a result, I can handle ETL, ad hoc analytics, and ML on a single stack, rather than juggling separate warehouses, lakes, and Spark clusters.

What do you dislike about the product?

The platform can feel heavy and is sometimes slow, especially when working with large notebooks or running long jobs. Databricks can also be expensive to operate, particularly if clusters are left idle or aren’t well optimized.

What problems is the product solving and how is that benefiting you?

Faster, collaborative workflows
Databricks simplifies big-data complexity by abstracting much of the Spark and cluster management, so I can focus more on logic and less on infrastructure. The built-in notebooks, jobs, and versioning make it easy to prototype quickly, collaborate with analysts and DS, and move code from experimentation into production with less rework.

Unified platform for data and AI
Databricks reduces the need for separate data-lake, data-warehouse, and ML tools by providing a single lakehouse platform where you can store, transform, and analyze data, and run ML workloads in the same place. This helps cut down on tool sprawl and makes it easier to share data and models across engineering, analytics, and data science teams.

What do you like best about the product?

What I like best about Unity Catalog in Databricks is its ability to provide centralized data governance and fine-grained access control across all data assets, making it easier to manage and secure data in a collaborative environment.

What do you dislike about the product?

I created a notebook with more than 70 cells that I use to parse XML files. When I try to debug issues using Genie, it doesn’t work properly and ends up hanging.

What problems is the product solving and how is that benefiting you?

I used Spark functions to define the XML structure dynamically, assigning mpid and mpparentid as needed. This approach has been very beneficial for me.

What do you like best about the product?

Genie Code (Databricks Assistant Agent) — I’m currently working on migrating existing workloads from ADF and SQLMI to Databricks. As part of that, I need to convert stored procedures and ADF dataflows into Databricks notebooks. Initially, we refactored all the code manually, but once Agent Mode was available in preview, we tried using it to convert the stored procedures and dataflows into Databricks PySpark code. I was impressed by the accuracy: it handled about 90% of the code conversion without errors, aside from some case-handling and similar adjustments.

Also, Lakeflow Connect helped me connect SharePoint and SFTP data to Databricks more easily.

What do you dislike about the product?

It’s not a major issue, but in my project the client asked us to generate table and column descriptions using AI in Unity Catalog. For each environment, these descriptions vary, and I have around 300 tables just in the Bronze zone. Having to click into each table and generate AI descriptions one by one is very time-consuming, and the results are not consistent across environments.

It would be much more efficient if we had an option to generate descriptions at the schema level, and if there were an information schema or system tables that stored table and column descriptions as metadata. That way, we could easily replicate them across environments. In some cases, clients also have source system documentation we could leverage to generate more accurate table and column descriptions.

What problems is the product solving and how is that benefiting you?

One of my main scenarios was migrating all the existing stored procedures and ADF dataflows into Databricks notebooks. Doing this manually took more than 6 hours to complete both the development and the validation. Later, we used Agent Mode Preview and converted over 80+ medium/complex stored procedures and 20+ ADF dataflows into Databricks notebooks. This saved more than 100+ hours, and it also generated validation scripts for each table to close out unit testing.

Apart from the Agent Assistant, we also used external volume. Previously, we relied on the Azure library for file processing in ADLS storage, but we ran into rate-limit issues, couldn’t process in parallel, and sometimes the job would abort. After we created an external volume pointing to the required ADLS container, we achieved parallel processing and faster reads and writes, instead of using custom Python code.

What do you like best about the product?

1) In our implementation, Genie Space is actively used to enable NLQ-based access across multiple data products like Finance, HR, Marketing, Sales, and Supply Chain (inventory, demand planning, and replenishment), reducing dependency on data teams for ad-hoc queries.

2) We designed separate Genie Spaces for each BU/team/data product, ensuring domain-level isolation while still supporting cross-functional querying where required (e.g., Finance + Sales joins).
Each Genie Space is carefully configured with curated data tables, business-level instructions, and semantic context, which significantly improves the accuracy of SQL generation.

3) We provide few-shot examples, guided prompts, and sample business questions tailored to each domain, helping Genie understand real business intent instead of generic query patterns.

4) In Chat Mode, business users directly ask questions in natural language, and Genie translates them into SQL and returns results, which has improved self-service analytics adoption.

5) In Agent Mode, Genie goes beyond SQL generation by creating a logical execution plan, breaking down complex queries into multiple steps before querying the underlying data.

6) We built a dedicated Anomaly Detector Genie Space, where users ask questions about cluster cost, performance issues, and inefficient workloads.
This anomaly-focused Genie analyzes long-running jobs, inefficient queries, and cluster utilization patterns, using historical workload data to identify optimization opportunities.

7) A key implementation is notebook-level analysis, where Genie highlights code issues, shows before vs after optimization, categorizes problems (performance, cost, inefficiency), and explains improvements clearly.

8) Genie also provides quantified recommendations, including expected cost savings (e.g., idle cluster reduction, query tuning impact) and workload-based optimization strategies, making it highly actionable for engineering teams.

9) We extended Genie into Genie Code integrated with Databricks AI Assistant, enabling an agentic development experience directly within our data engineering workflows.
Our team defined custom skills in Markdown (MD files) such as Coder, Tester, Mapper, and Data Generator, which are attached to Genie Code to modularize capabilities.
These skills are used to support end-to-end SDLC activities, including code generation, transformation logic creation, test case design, and synthetic data generation.

10) Genie Code operates by first creating a structured execution plan, outlining all required steps before starting any development activity.
It then breaks the plan into a detailed to-do list, executing each step sequentially (e.g., create notebook → write transformation → validate logic → optimize code).

11) During execution, Genie Code follows a human-in-the-loop model, asking for approvals at every step with options like allow once, always allow, or read-only execution.
The behavior of Genie Code is controlled through project-specific guidelines and instructions, ensuring it aligns with our coding standards, architecture patterns, and governance rules.

12) It acts as a co-developer within the workspace, assisting engineers in writing optimized code, validating logic, and ensuring best practices are followed consistently.
We are leveraging it for proactive development workflows, where Genie not only executes tasks but also suggests improvements and optimization opportunities during development itself.
This approach has enabled a “vibe coding” style of development, where engineers focus on intent while Genie handles structured execution, resulting in faster delivery, reduced manual effort, and improved overall code quality.

What do you dislike about the product?

Context limitation across Genie Spaces, also number of tables can be attached is 30 if i remember
Agent Mode reasoning depth is good but not fully autonomous
Need improvements in performance efficiency and reduce the latency

What problems is the product solving and how is that benefiting you?

1) Bridging business and data teams through NLQ
Databricks Genie solves the gap between business users and technical teams by enabling natural language access to data, reducing dependency on data engineers for everyday queries.

2) Eliminating data silos across domains
By integrating data from Finance, HR, Sales, and Supply Chain, it helps us analyze cross-domain datasets, improving decision-making for use cases like demand planning and inventory optimization.

3) Accelerating self-service analytics
With Genie Chat Mode converting NLQ to SQL, business users can independently fetch insights, significantly reducing turnaround time for reporting and analysis.

4) Handling complex analytical queries with Agent Mode
Genie Agent Mode solves complex query scenarios by breaking them into structured execution plans, which is especially useful for multi-step analytical and optimization problems.

5) Improving cost and performance visibility
Through our Anomaly Detector Genie Space, Databricks helps identify cluster inefficiencies, long-running jobs, and costly queries, giving clear visibility into platform usage.

6) Driving workload optimization and cost savings
The platform provides actionable recommendations like query tuning, cluster right-sizing, and idle resource reduction, helping us optimize cost based on actual workload patterns.

7) Enhancing code quality through notebook analysis
Genie analyzes notebook code and highlights performance issues with before/after comparisons, enabling developers to improve efficiency and follow best practices.

8) Supporting proactive development with Genie Code
Databricks enables an agentic development workflow, where Genie Code assists in planning, coding, testing, and executing tasks step-by-step, reducing manual effort.

9) Standardizing development using skill-based automation
By attaching custom skills (Coder, Tester, Mapper, Data Generator), we ensure consistent development practices and faster onboarding for new use cases.

10) Increasing overall productivity and faster delivery
Combining Genie Space and Genie Code, Databricks significantly improves developer productivity, reduces iteration cycles, and accelerates delivery of data solutions, while maintaining governance and control.

What do you like best about the product?

I like how Databricks allows not just engineers, but also data managers, analysts, data scientists, and everyone to work in a simplified and collaborative manner. That's a feature I appreciate which Databricks does well, setting it apart from competitors who are trying to offer similar capabilities. Many people have already adopted it, and it has become the de facto choice.

What do you dislike about the product?

I think the lineage and the addition of business assets, as well as how the data translates to the business layer of the bank or any other vendor, is where Databricks can improve. I don't see different departments getting connected in Databricks by the glossary or items which they use for themselves.

What problems is the product solving and how is that benefiting you?

I use Databricks to manage vast datasets from multiple sources, helping organize infrastructure and access management, and aids in some visualization tasks.

What do you like best about the product?

It's an all-in-one platform for data engineers, analysts, data scientists, and business users.

What do you dislike about the product?

It’s easy to overspend and it is a vendor lock-in.

What problems is the product solving and how is that benefiting you?

Data engineering, model training and inference, GenAI.

Databricks solves the problem of having fragmented tools across the data and AI lifecycle. Traditionally, teams would need separate platforms for data engineering, analytics, machine learning, and AI — leading to silos, duplicated work, and governance challenges.

With Databricks, data engineering pipelines, model training and inference, and GenAI development all live in one unified environment. This means data engineers can build and orchestrate pipelines, data scientists can train and deploy models, and teams can develop and serve GenAI applications — without constantly moving data or context-switching between tools.

What do you like best about the product?

As a growing Data Engineer, the community support and clear documentation of Databricks really helps me to guide through the problems. I've been managing the jobs and pipelines where failures are bound to happen, debugging with the Diagnose this error with AI feature has helped me with fasterthe failure recovery SLA. The UI is neat and makes it very easy to move between notebooks, SQL, and PySpark without much friction. Since I work with a team, collaboration is must. Sharing notebooks and iterating with teammates feels easy. I really like that I can rely on the ABAC policies to setup the Data Quality and Governance.

What do you dislike about the product?

I am not hundred percent sure if I would use the term dislike, I think it's just a personal preference. I sometimes feel the compute being used is a lot more than it should be for a simple query. Maybe the shuffle read/write that always gets involved when you're using a delta tables sometimes slows down the job.

What problems is the product solving and how is that benefiting you?

Databricks is helping our clients to manage the lakehouse and warehouse architecture in a much more structured way. We use it as the landing layer from S3 and then process data through our medallion architecture (bronze, silver, and gold) before delivering it to the final products. It’s been very effective for orchestrating daily jobs and pipelines. I also really like the asset bundles and how easily everything integrates with Git, which makes version control and deployments much smoother for the team. I am more likely to use Databricks as my go to platform for data lakehouse and warehousing.