DataHub

My main use case for Data Hub is to catalog the dataset across my company and to get the lineage of data in the my company pipeline.

To give an example of how I use Data Hub in my day-to-day work, suppose the data is flowing from a source to Kafka and then to some data storages. If some cross-team wants to use the data but there is a problem at the Kafka level, we are not sure who all are consuming that data. Data Hub is very useful for us in this scenario. It can generate the lineage from source to destination, and when there is an issue at the Kafka side, we will get to know what the end results and impacted data sources are.

I would add that sometimes when we do not want to share the data or when the customer or another team wants to consume the data, we are not sure what kind of data is there. We have to look at the schema. Data Hub is useful for us as we are doing the cataloging of all the datasets across my company, allowing us to later use and see the table information and schema information so that the team can identify what data is PII or non-PII.

The best features Data Hub offers include support for cataloging and lineage very well, as we are getting all the different types of connectors to consume and use across the my company dataset pipeline. Apart from that, the GraphQL APIs provided by Data Hub are very good, allowing us to get all the information we need programmatically whenever we need it.

Regarding how the GraphQL APIs help my team in day-to-day tasks, we sometimes use custom logic to check whether the data has PII or non-PII. We have some AI model running on top of it, which requires classification. Based on the dataset URL, we are getting information about the dataset using the GraphQL APIs. GraphQL APIs are very handy, allowing us to customize properties and pass on the necessary information. For example, if we need a structured property, we can get those structured properties. If we need tags or owners, we can retrieve that as well.

Data Hub positively impacts my organization by enhancing collaboration as previously, we had to ask the team to provide the schema information. my company operates in a cross-region environment, so a person in India could wait a day to receive information about the schema from someone in the US. However, with Data Hub, we have a centralized place where we can access all the schema of the datasets, making it very helpful. Additionally, whenever there is a problem, using the lineage helps us quickly identify the impacted team or dataset.

Whenever there is an incident, we first go to Data Hub to see the downstream teams impacted and stop any jobs running on those datasets. It helps us save around eighty percent of time, as we previously had to track down information manually to find the owners, but using Data Hub, we can tag the owners of the datasets directly in the tool.

For improvements to Data Hub, I feel the security is a bit on the weaker side. We have ingestion jobs that require exact permissions for different owners, but this setup does not align with the my company grouping system. We need to create some custom grouping to manage those permissions. I would appreciate it if there were a method to consolidate all the information on a single page, which would simplify sharing permissions for running ingestion jobs.

Additionally, I do feel that the metadata test we run daily takes too long. Initially, it takes one day, which I find excessive. Ideally, we should get information within one hour. These are the two main issues that would benefit from improvement for our use case.

I have been using Data Hub for one and a half years.

Data Hub is stable in my experience. However, there are times when we attempt to upgrade it, and it may go down for a couple of minutes, but not more than that.

Data Hub handles scalability effectively, accommodating growing data and users.

I have had to reach out to Data Hub customer support multiple times. For example, when we were setting up a private link to connect to Data Hub GraphQL APIs, we required our account to be whitelisted. I have also requested some future features for our use cases. For instance, when working with a metadata test scenario, I needed to have a range date column, which was not available. I requested the Data Hub team to make it public so we could use it.

I have seen a return on investment with Data Hub. For instance, I have noticed time savings during incidents and while looking up schemas. In terms of resources, Data Hub centralizes data cataloging and classification, saving us from having to disclose PII column information to teams not utilizing it. Regarding financial metrics, I do not have specific metrics available.

Before choosing Data Hub, we looked into Unity Catalog from Databricks , but we ultimately decided to stick with Data Hub.

My advice for others looking into using Data Hub is to use it for cataloging, classification, and centralizing all your schema. Data Hub supports a variety of connectors and has excellent lineage options. Additionally, make sure to utilize the well-written documentation that can guide you in building your product solutions. I would rate this product a nine out of ten.

My main use case for Data Hub is that we use it as a library for all the data assets that we generate. It serves as an internal data mart where people can search for whatever data they need, and they can search by tags, by roles, and then add more metadata to it. This provides visibility to the data.

A specific example of how my team uses Data Hub in a real-world scenario is that we collect and manipulate a bunch of data layers. Because we have huge teams, the exposure to data that we have already manipulated can sometimes be hindered when using traditional systems. Data Hub acts as a search engine for all of the data. One example would be when the marketing team was looking for specific data around marketing. They discovered that once they searched it on Data Hub, it was easily visible. They did not have to retrieve it from the raw layer and manipulate it for their usage because another team had already built it.

Regarding how my teams interact with Data Hub, we use Data Hub with a self-hosted system. We have connectors which look into multiple data sources, manipulation engines, and orchestration layers to gather the metadata, and then that is pulled into Data Hub. This is how we get data assets in Data Hub.

The best features that Data Hub offers include primarily data discovery and data governance. Data Hub has data catalogs, which helps with the business glossary, ownership tracking, and lineage. Lineage is something that we are strongly using at this point in time. It helps us understand the impact analysis, such as what breaks if I change this column. Data Hub also provides data observability, helping us understand what data is fresh, what is not, and what has changed schema recently. Additionally, it makes our system AI and LLM ready.

The lineage feature has changed the way my team works and collaborates significantly. Because we now have data lineage through Data Hub, if we have a really huge dependent pipeline with multiple layers of upstream and downstream dependency, and something breaks in the downstream system, we can exactly pinpoint what all data assets would be affected. Having that lineage functionality helps us drill down what needs to be debugged and fixed and what exact part is breaking. It saves us time in remedying the issue.

I really like the integrations that Data Hub provides. Data Hub has a very large set of integrations that we can do with Snowflake , Databricks , BigQuery , Redshift, DBT, and Airflow .

Data Hub has positively impacted my organization as teams can now be directly dependent on one source of truth for all their data needs. The time spent finding information has become significantly smaller, which is the real productivity improvement that I have seen, impacting multiple teams throughout the organization. I estimate that we save about thirty to forty percent of the time now since we do not have to read documents or message people for specific data assets. This results in a productivity increase of around thirty to forty percent in terms of time and efficiency.

I think Data Hub can be improved by supporting the open source version better. Many features have moved to the paid version now, making it difficult for small-scale companies to operate on Data Hub because we are required to pay, even though it started as an open source project that is now essentially behind a paywall.

One needed improvement for Data Hub would be stronger AI-powered metadata discovery. I understand Data Hub has been investing in AI, but the natural language processing power on Data Hub search is not that good. The search itself is not accurate many times. Another improvement could be enhancing the DBT developer experience, such as surfacing DBT test failures directly in lineage. Additionally, when we change schema, if it could provide a risk scoring of some sort, that would also be beneficial. Lastly, automated cleanup recommendations would help because managing orphan data assets on Data Hub currently takes a lot of manual time.

I have been using Data Hub for a year.

Data Hub is pretty stable in my experience with no downtime or issues.

Data Hub's scalability has been effective, handling our organization's growth and data volume well.

I have not had to reach out to customer support.

I did not previously use a different solution before Data Hub.

My experience with pricing, setup cost, and licensing has been pleasant, and I have no complaints.

Before choosing Data Hub, we evaluated Atlan  and decided on Data Hub because it has a cleaner UI and also a decent open source community to support it.

Data Hub does most of the job it is designed to do, but there could still be improvement as the industry progresses, particularly around metadata discovery. Regarding Data Hub's AI capabilities, its governance and security do the job really well as of right now. I do not have any complaints, especially around data classification, as it allows us to have control over whatever data we are displaying, including customization for PII, sensitive, and financial data. Data Hub has met our expectations regarding its accuracy and reliability of output, and there have not been any issues.

My advice to others looking into using Data Hub is that it is a pretty nice product right now with easy integration. The pricing model could be negotiated, so it is essential to keep that in mind. I would rate Data Hub a solid eight on a scale of one to ten.

My main use case for Data Hub is to enrich the metadata to classify for PII data. As an administrator, I crawl a number of data sources and bring the metadata into a single place, then assign the ownership, such as a data owner or steward, for all the data assets. With their help, we classify the data into PII direct and indirect, sensitive, non-sensitive, and so on. We add tags and glossary terms onto the data elements. The main use case is for DSAR compliance; for GDPR DSAR compliance, we try to identify the PII data in the catalog so that we know where the PII data is in our data inventory.

The catalog helps with metadata discovery and to find the owners/ stewards of data sources. Without a data catalog, we scramble around and speak to multiple teams, which is time consuming.

The best features that Data Hub offers include the management of ownership, with standard out-of-the-box ownership such as business owner, data steward, or technical owner, which is relevant for us. It also integrates with Active Directory. In our Active Directory, we maintain certain roles based on the scrum teams related to a team member, and by integrating with Active Directory, we are able to bring the same roles and map them to the corresponding ownership roles within Data Hub. Data Hub has integrations with Slack, Snowflake , BigQuery , and so on, which we use.

Data Hub has positively impacted our organization by bringing the tribal knowledge that resides with team members into a single place where users can discover and understand the data elements before they make use of it. Users can ask questions via Slack to understand how a data element is defined and get the answers back. This definitely saves time; without a data catalog in place, users need to ask around to find out what a particular data element means and to find out the owners. Now, with the data catalog, searching and discovering data elements and the corresponding owners is easier, saving approximately thirty to forty percent of the time that would have been spent finding out the owners and definitions of the data elements.

Data Hub can be improved with more automation; there are some inbuilt automations, such as documenting definitions of data elements using AI, which is useful. I wonder if it can automate the classification exercise, possibly using AI to auto-classify PII direct and indirect items.

Just started using it.

Data Hub is stable.

Data Hub shows scalability in terms of the number of users and the number of new databases and data elements.

We have not gone that far with customer support; as far as the POC is concerned, we received good support from the team and the sales team that helped us evaluate the tool.

Neutral

We previously used OvalEdge  as our data catalog before switching to find a tool that has more AI capability and allows extension of usage to non-technical users, seeking a tool that is less clunky and more intuitive.

slightly techical. But there is enough documentation available.

No

I have not yet seen a return on investment, and I do not have that information to share.

Regarding experience with pricing, setup cost, and licensing, I think if we have a budget of one hundred thousand US dollars, we will be able to deploy a reasonable version and connect to a number of data sources.

Before choosing Data Hub, we compared it with Atlan  and Alation .

I chose seven out of ten because there are better catalogs available in the market that offer more features. The UI, especially when setting up new data sources and crawling them, is a little cumbersome, but it is a one-time activity, so it is manageable; however, the UI could be improved concerning administration.

My advice to others looking into using Data Hub, also known as Acryl, is that it is a reasonably stable product that satisfies most data catalog use cases; however, Atlan  appears to be the closest competitor, while Alation  is the market leader among the three. Data Hub has an open-source version I believe, and it may be worth considering that option as well.

I rated this review seven out of ten.