Unified data workflows have cut ticket processing times and are driving faster business insights
What is our primary use case?
My main use case for Databricks involves the pipelines and ETL processes that we are implementing. Following the Medallion architecture with Gold, Silver, and Bronze layers, we filter the data, perform transformations, and integrate AI. Databricks has made this process significantly easier.
I worked for an airline company where they experienced substantial delays in data processing. When a passenger booked a ticket, it took 20 to 25 minutes for the transaction to reflect in the system. Using Databricks, we compressed that time from 10 to 6 minutes initially and eventually reduced it to just a few seconds. After setting up all the pipelines and leveraging Databricks features to enhance and accelerate the process, this project became truly impactful and time-based, resulting in reduced processing time and ultimately increased profit for the airline company.
What is most valuable?
The best features Databricks offers are Unity Catalog, Databricks Workflow, Databricks AI, Agentic AI, and the automated pipelines that utilize AI. The AI models are very easy to create and deploy in just a few seconds. These are helpful and user-friendly tools.
I find myself using Unity Catalog most frequently because it provides a unified governance solution for all data and AI needs on Databricks, offering centralized access control, auditing, lineage, and data discovery capabilities across the platform. The main features include access control, security compliance standard models, built-in auditing, and lineage tracking. Most of my projects have involved integrating Unity Catalog into systems and providing overall security, including a migration project to transition to Unity Catalog.
The platform's unified data intelligence capabilities allow teams to analyze, manage, and activate data at scale, leading to faster time to insights, cleaner data pipelines, and significant savings on infrastructure and engineering efforts. Databricks eliminates data silos, accelerates the time to insight, empowers all data personnel, and provides built-in governance and security. It also supports AI and ML, which is an added advantage in today's AI-driven field.
What needs improvement?
Databricks already provides monthly updates and continuously works on delivering new features while enhancing existing ones. However, the platform could become easier to use. While instruction-led workshops are available, offering more free instructional workshops would allow a wider audience to access and learn about Databricks. Additionally, providing use cases would help beginners gain more knowledge and hands-on experience.
Regarding my experience, I was initially unfamiliar with the platform and had to conduct research and learn through various videos. I did find some instruction-led classes, but several of those required payment. The platform should provide more free resources to enable a broader audience to access and learn about Databricks. The platform itself is user-friendly and easy to use without complex issues, so I believe it does not need improvement in its core functionality. Rather, supporting aspects can be enhanced.
For how long have I used the solution?
I have been working as a data engineer for four years. Initially, I was a software engineer, but my career has progressed as a data engineer over this four-year period.
What was our ROI?
Definitely. As I mentioned regarding my airline project, it was impactful because the cost was reduced by 60 to 70 percent. The company was initially using Azure Blob storage, and in Databricks, the cluster and associated infrastructure were cheaper than other platforms. This reduction in both time and money resulted in real-time impact and significant cost savings.
What other advice do I have?
For advice for others considering Databricks, it is important to start by understanding its place in the data ecosystem and how it fits into your specific needs. Key points to consider include familiarizing yourself with Databricks, learning the basics, starting with data engineering, and incorporating ETL processes. You can then dive deeper into Databricks features such as notebooks, clusters, and jobs. Achieving certification enhances your skills validation. For best practices, it is critical to optimize performance and minimize complexity while continuously learning to stay competitive in the data field. Following these steps will be very beneficial for anyone pursuing a career as a data engineer and Databricks engineer.
Databricks is a truly essential platform for data engineering needs, and I recommend it to anyone looking to advance in the data engineering field. It is a very important platform and tool for every data engineer. I encourage everyone to learn and explore this product and to maximize its potential. I rate this product a 9 out of 10.
AI Integration with the Data Lakehouse Made Databricks a Clear Choice
What do you like best about the product?
The integration of AI to the data lakehouse is the key thing which encouraged us to use databricks
What do you dislike about the product?
Databricks is more complex than spark therefore it takes more efforts to fine tune it as per business usecase
What problems is the product solving and how is that benefiting you?
As the worlds largest wealth manager blackrock has huge TBs of data processed daily via spark jobs and to derive meaningful analytics from that data which is highly flexible required so much effort and expertise but with databricks AI models it became easy
Effortless Data Insights and Governance
What do you like best about the product?
I like the Databricks Data Intelligence Platform for its data governance capabilities. The platform supports machine learning applications and offers helpful autofilling features. I also find the quick analytics code support to be a valuable aspect.
What do you dislike about the product?
I find it problematic that if the tables have two similar attributes and I need to choose another which isn't many-to-many, it can't handle that yet. Also, when I ask for the keys of a table, whether foreign or main, it's not able to provide the correct key.
What problems is the product solving and how is that benefiting you?
Databricks Data Intelligence Platform reduces the time to find relationships between tables.
Databricks -Scalable Data
What do you like best about the product?
1. Easy for data teams once set up; notebooks, SQL, and dashboards work smoothly in one place.
2. Used frequently for data engineering, analytics, and ML workloads.
3. Implementation is structured and scalable, especially on cloud environments.
What do you dislike about the product?
1. Customer support quality depends on the support tier purchased.
2. Too many advanced features can feel overwhelming for smaller teams.
3. Initial setup and architecture planning take time and skilled resources.
What problems is the product solving and how is that benefiting you?
1. Helps me make faster, data-driven decisions with scalable and trusted data pipelines.
2. Handles large data volumes reliably, supporting daily and recurring workloads.
3. Reduces time spent managing infrastructure so I can focus on insights and outcomes.
Unified Platform Enhances Data Collaboration and Processing
What do you like best about the product?
I like the Databricks Data Intelligence Platform because it's a unified, scalable platform for data engineering, analytics, and collaboration. It simplifies the process of building and running data pipelines, handles performance-intensive Spark workloads, and promotes collaboration across teams with shared notebooks and environments. It's great for managing and processing large datasets, improving performance, and providing a consistent platform for turning raw data into insights. The initial setup was easy, thanks to helpful documentation.
What do you dislike about the product?
It can improve in areas like startup time for clusters, deeper visibility into performance tuning and clearer documentation for advanced configurations.
What problems is the product solving and how is that benefiting you?
I use the Databricks Data Intelligence Platform for large-scale data processing and analytics. It simplifies building reliable data pipelines, handles Spark workloads efficiently, reduces operational overhead, and enables team collaboration with shared notebooks. It's great for turning raw data into actionable insights.
Outstanding Experience with This Software
What do you like best about the product?
Databricks data intelligence is a platform that helps in accommodating all of our business and official data and share it with different team departments so that they can analyse it and create a detailed analytics of past performances and also make required changes on it for future growth.
What do you dislike about the product?
One of the major challenge that we face while working with Databricks data intelligence platform is that you cannot use this tool with a single data scientist you will have to keep a team of professionals who can deal with large data and create multiple graphs and analytics according to available information and this complete activity involves lot of financial investment
What problems is the product solving and how is that benefiting you?
This software help us in making sure that all the data of different departments are accommodated in a same software so that access can be easier and decisions can be taken much quicker. With the help of this tool data of all the departments like finance, operations, sales and marketing are screend in one time and thoroughly interchecked too.
Excellent ML Features and Data Controls in Databricks
What do you like best about the product?
Databricks offers useful features for machine learning engineers, such as the ability to use compute pools to run workloads efficiently. In addition, it includes the Unity Catalog, which is an excellent tool for managing data access controls.
What do you dislike about the product?
The product would benefit from having more built-in functions that are specifically optimized for GenAI use cases.
What problems is the product solving and how is that benefiting you?
We successfully consolidated data from multiple ERPs onto a single platform, which enabled us to use this unified data for data engineering, reporting, machine learning, and GenAI use cases.
Impressive AI Features and Unified Lakehouse Architecture
What do you like best about the product?
The recent AI features are quite helpful and also the unified lakehouse architecture.
What do you dislike about the product?
A bit tough for those new to spark or from a non tech background especially the table view.
What problems is the product solving and how is that benefiting you?
unified data platform for all my data from gsc to bigquery to data warehouse and datalake. Super high processing speed. Easy to use and understand with a clean UI. the AI query mode is very helpful and kind of addictive lol.
Simplifies Data Analysis for Streamlined Workflows
What do you like best about the product?
Databricks is a Data Analytics tool that is used by our designing engineers to analyse all the different type of financial and operational data of our company and create a summary of everything so that understanding of companies workflow can be simple.
What do you dislike about the product?
One of the major disadvantage of databricks is that the software is completely based on programming language like Python, SQL, Scala so if the user does not have knowledge of these programming languages than it is very difficult for them to use this platform.
What problems is the product solving and how is that benefiting you?
The best part of databricks is that it can easily get integrated with cloud networks like as azure, AWS, Google Cloud etc. So, working on this platform also help us in getting access to all the data of different years and compile them to get a proper analytics of business performance.
Why I love Databricks
What do you like best about the product?
I like the integration between Spark, Delta Lake, and notebooks, which helps optimize and scale data pipelines. And the other feature that I love is the different cluster options that we have, which are really easy to tune for cost and performance.
What do you dislike about the product?
Debugging complex Spark jobs and understanding some of the optimization features is a time taking process in Databricks, is what I think.
What problems is the product solving and how is that benefiting you?
We basically get data into our catalogs in Databricks. The Databricks here allow us to perform many transformations and create different data products for the end user. Not just the data products, there are many other benefits like the ease of workspace, notebooks, and the cluster configuration.
