DataHub

My main use case for Data Hub is for a catalog system because we are integrating all of the data sources to Snowflake and then we want to catalog and share business glossary terms with our company employees.

A quick specific example of how I use Data Hub in my daily workflow is that we have all of the data in Snowflake and all of the employees using Snowflake did not know what kind of data is in Snowflake. They did not know all of the tables and what kind of columns and metrics, KPI definitions exist, so we are using Data Hub for searching the data in Snowflake and identifying who is using Snowflake.

My main use case is covered.

Data Hub has positively impacted my organization because there are many data analysts in each team, and the time to Q&A has significantly decreased since we started using Data Hub. This improvement is also seen in our KPI tracking.

I cannot provide specific time savings, but for example, we used to have 100 user requests for questions, which required searching Snowflake tables to determine what tables should be used, but now it is down to almost 10 questions.

In my opinion, the best features Data Hub offers are the searching function and tagging function. If I add a tag for some of the tables or columns, it is very easy to find people who need that information.

I am trying to use the tagging function for all of our data, but we are currently developing it, so we have covered almost 70% of our data.

We are using the free version of Data Hub with Docker Compose, so it is somewhat difficult to find out the lineage. If we are using Data Hub free version, then we can only figure out the tables' lineage, but we cannot search the column lineage, which is why I would like to add the columns-level lineage.

I need the lineage function for more column-level lineage and I think more example documents that are essential for our company would be very useful because there are many glossary terms and features in Data Hub, but I did not know which are more essential for us.

Additionally, I also have one more concern regarding using Docker Compose for Data Hub; the memory issues come up sometimes and consume a lot of memory resources, so I need a more efficient way to use Data Hub without these issues.

I have been using Data Hub for almost one year.

We are using private clouds in AWS, and we have deployed Data Hub on the AWS EC2 server with Docker Compose.

The cloud provider we use is AWS.

I did not purchase Data Hub through the AWS Marketplace; I am just using the EC2 server and deploying it with Docker Compose.

My advice for others looking into using Data Hub is that if there is no catalog system or data dictionary system and if there are many KPIs or metrics within their company, then I recommend Data Hub to those kinds of teams.

I give Data Hub an overall rating of 8.

My main use case for Data Hub is data lineage tracking. With Data Hub, we track multiple sources, ingestion sources, and different sources where the data resides in S3. We bring all that metadata into Data Hub to track lineage on the data ingestion patterns that we perform or transformations that we do, and how they move from different tables or assets or the data pipelines. Whatever transformations we do with Spark and S3, Snowflake, all those are being tracked via Data Hub. We have S3 buckets and Snowflake tables, and all those lineage tracking is managed through the platform.

My main use case is mostly covered as we used Data Hub for metadata tracking and lineage for whatever transformations that we do so that we can track each transformation down the line.

In my experience, the best features Data Hub offers include lineage tracking, which is mostly on the asset level, a good glossary, and good connector support.

Regarding asset level and the good glossary, we need the glossary of our products so that it is easy to track which product, what went at what time on that particular product, how many assets are related, and so on. For asset integrations, Data Hub makes it easy to ingest all that metadata of those particular assets from S3 via connectors, which is quite easy. It has good connector support, although limited in some cases.

Overall, Data Hub is a good tool. If we talk about lineage, metadata, and observability on some high level, including domain descriptions, PII classification, datasets, and keeping datasets in one place along with policies, it is good in that particular sense. We do have a plan based on project-to-project usage, but in some of the projects, we do use Data Hub as well.

I would like to add that for the connectors, there is sometimes limited support for using wildcards to get the items or assets ingested from sources like S3; it does not support very good wildcard filters. Additionally, Data Hub has a problem with column-level lineage support, especially regarding non-pro users or those without any plans. If I talk about the free features of Data Hub open source, those two I found could be improved during my use case.

Regarding improvements needed for Data Hub, I have already mentioned the limitations on the usage of wildcards in the ingestion or connectors; that can be worked upon, especially regarding the open-source part of Data Hub. The rest is that I hope the UI is quite good.

I used Data Hub for one and a half years.

My advice for others looking into using Data Hub is that it is a good tool if you want to capture all that metadata, lineage, keep track of governance, security, and observability. It just depends on how you want to use it; you can choose the open-source version or the paid version and subscription-based model. The paid versions have more features, but open-source Data Hub, which most people will try to go for, has some limitations, such as the missing column-level lineage with Spark. You need to consider those points, but overall, it is good. I would rate this product an 8 out of 10.

My main use case for Data Hub involves integrating our HR system or Active Directory, which automatically pulls in all 2,000 workers and groups them into their respective project squads and R&D teams. Each team gets its own team profile page in Data Hub, which helps solve the classic corporate headache of determining who to ask for specific information.

When a team builds a data pipeline, a Kafka topic for telecom signals, or a dashboard, it is tagged explicitly with their team profile as the owner in Data Hub. This means that if a developer in Split, working in the same company, needs to find a specific network dataset, they do not waste days spamming Slack channels; they can simply look it up in Data Hub and find the team profile that owns it along with the direct contact info or Slack channel.

Additionally, it enables us to run a data mesh model with 2,000 people, allowing one central IT team to manage everything while Data Hub facilitates splitting the company into logical domains such as electronic health, telecom networks, IoT, or smart cities.

The best features that Data Hub offers include the ability to centralize everything in one platform, such as creating profiles and organizing them into separate domains like engineering, health teams, supporting teams, and HR teams. This allows information to be shared across different domains.

Utilizing the data mesh model enables the company to maximize functionality using a single solution. Data Hub supports collaboration between different teams and departments significantly, as evidenced when we created various data mesh modules and established different domains such as E-Health, telecom networks, and IoT. This allowed us to share datasets effectively, and with authenticated users, the communication and responses were much quicker.

Among those features, I find the collaborative aspects the most valuable in my work because it has greatly improved our operations over the past year. We evaluated various licenses and methods to integrate data catalog platforms, ultimately deciding to move forward with Data Hub since it was more compatible with our company's security requirements. Compared to other tools, it received better support from the community, which is updated daily, allowing us to collaborate effectively through contact sharing.

Data Hub has positively impacted my organization by functioning as an all-in-one solution. It uses data mesh and separates domains to manage privileged access based on user validation, allowing us to share data sets across the company, which informs everyone about internal regulations. Furthermore, it significantly aids new joiners in understanding the operations and knowing who works on specific projects, while also providing updates on changes occurring within various sectors and domains.

The frequency and quality of updates or new features released for Data Hub have been impressive. This extensive community support was a key factor for us at Ericsson Nikola Tesla to choose Data Hub as our data catalog.

Regarding how Data Hub can be improved, I believe they should focus on enhancing their marketing efforts. Within our company, we were unaware of the Data Hub platform while searching for data catalog options that offered strong security and collaboration. Better marketing would help other companies learn about this effective solution.

My rating of eight rather than a nine or ten pertains to the connections with different systems. Specifically, the integration with Slack and Azure, as well as how we link our HR system to Data Hub, could be improved for better compatibility.

Integrating Data Hub with our existing tools and systems was not very easy, which is why my rating is an eight. We attempted to incorporate our HR system with Data Hub, aiming to set governance status for the 2,000 employees in our organization, but I did not complete this aspect before leaving the organization.

I have been using Data Hub for at least six months at the company called Ericsson Nikola Tesla in Zagreb, which has a massive operation with an entire ICT and R&D division of around 2,000 workers.

In terms of scalability, I believe Data Hub performs exceptionally well as more teams come on board, making it efficient for large organizations with approximately 2,000 employees. It adequately supports the scalability of data sets and the implementation of data mesh models.

During implementation, the documentation and support resources from Data Hub were very helpful. I followed the guidelines, accessed each section, and understood the platform effectively, which made the initial setup easy.

Data Hub is flexible, optimistic, and user-friendly in terms of its interface and experience. I rate Data Hub an eight on a scale of one to ten.

The learning curve for new users adopting Data Hub is addressed through their learning section that guides users on how to navigate the platform. I found it quite simple and effective to follow.

We purchased Data Hub through the AWS Marketplace.

As for specific outcomes or metrics, I currently do not possess numbers since we are still in the early stages of implementing Data Hub within our company. However, the HR department reported significant time savings in completing tasks before and after adopting Data Hub, which has resulted in faster completion and better collaboration without interrupting others.

Data Hub has worked for me personally, as I noticed that after we began ingesting Data Hub into our Ericsson Nikola Tesla company network, it proved to be incredibly helpful for easier access to information. By positioning team profiles at the center of Data Hub, it prevents the duplication of data sets, accelerates onboarding for new engineers, and fosters more connected and collaborative teams within our large employee base. Personally, it has helped me specify tasks and has contributed to the company's progress with the data catalog we chose.

My advice for others considering using Data Hub is to understand how it works and explore its integration potential within their organization. Engaging with community support can also be beneficial, as the team's collaborative approach is impressive.

My main use case for Data Hub is to build data products inside Natura; primarily, I built data products for CRM, which is customer relationship management, and also for some data products for the product field, such as analytic fields. I used it by dividing the company into domains, and each domain has its own functionality and its own structure. With that approach, I used it extensively for building domains. I also used it to build data lineage across the entire data journey, from the ingestion of the data to the use of the data in the final part, such as in a data product or in a dashboard.

A specific example of how I used Data Hub for building domains and data lineage is the domain called GenAI, which is primarily built for products based on AI, mainly generative AI. To accomplish this, I used Data Hub to track the data from the ingestion field. I used some CDP tools such as Segment and I also have data in an S3 bucket that was ingested to Databricks using Airflow. With that setup, I track this lineage from the origin system. After that, I performed a lot of transformation of the data inside Databricks to clean the data and conduct some data augmentation. After that, this data is used to train some models using Databricks LLM. With that, I ingest all this metadata into Data Hub and I can see from where the data is coming from and to where the data is going. This is primarily for LLMs to help consultants at the end of the product.

The best features that Data Hub offers include the capability to make conversational questions inside the platform, which I believe is the best thing that they built in the past year. It is also easy to connect different data sources. Since data lakes, I have connectors to some databases and also to some business analyst tools and other tools. I can connect many types of data inside Data Hub and see what is going on and how we govern the data. Data Hub is a pretty good tool for that. I also value very much the open-source version because it is free and everyone can use it.

I do not have much experience using the conversational questions feature, but I do not need to go to the asset to see from where the data is coming from and where it is going. I can simply ask, 'How can we calculate the sales in this month?' and Data Hub will identify which table will be used and from where this data is coming from and where this data is going. This is very effective.

Data Hub has impacted my organization positively by helping us build a data governance environment and share the knowledge about the data for the entire company. As we used the open-source version, we have no limitation in how many people can use the tool, which is excellent. I can conduct many tests and test as quickly as possible. It is very good for building POCs, for example. Data Hub also helps to give this understanding of the data for the entire company. Everyone in the company can see the data and know where the data is coming from and where it is going. I believe it is very effective and all the people in the organization, not only the data field personnel, can understand more about the data and also help to build better products.

I believe Data Hub could provide more functionalities in the free version. I understand that we have to pay the persons who build the platform, but the free version has some limitations. Some capabilities of the paid version being included in the free version would be beneficial. Another improvement that is needed in Data Hub is how I can get data from Data Hub to build some metrics. I know that I have the API and the GraphQL API, but I believe it could be better. If this is improved, it would be very helpful.

I used Data Hub for over two years, mostly in the open-source version.

I do not have actual metrics to provide currently; I only have some metrics. I certainly improved the data discovery part of building a data product because it is really fast to know if the data product already exists or does not exist. In the past, I had many products that were the same, and with that, I had a lot of work doing it twice or three times in different parts of the process. This is very good. I do not actually know the exact number of time saved, but I certainly saved time. I have a metric that before Data Hub, I believe 20 to 30 persons used and had knowledge about the data. Currently, I have almost 250 persons using Data Hub.

I did not use many AI features in Data Hub, as I stopped using Data Hub before it started offering these functionalities.

My advice to others looking into using Data Hub is to start by trying the tool using the free version. If it is sufficient and you already understand how to use it, you can transition to the paid version. However, you can accomplish everything in the free version. I would rate this review an eight out of ten.

My main use case for Data Hub is to catalog the dataset across my company and to get the lineage of data in the my company pipeline.

To give an example of how I use Data Hub in my day-to-day work, suppose the data is flowing from a source to Kafka and then to some data storages. If some cross-team wants to use the data but there is a problem at the Kafka level, we are not sure who all are consuming that data. Data Hub is very useful for us in this scenario. It can generate the lineage from source to destination, and when there is an issue at the Kafka side, we will get to know what the end results and impacted data sources are.

I would add that sometimes when we do not want to share the data or when the customer or another team wants to consume the data, we are not sure what kind of data is there. We have to look at the schema. Data Hub is useful for us as we are doing the cataloging of all the datasets across my company, allowing us to later use and see the table information and schema information so that the team can identify what data is PII or non-PII.

The best features Data Hub offers include support for cataloging and lineage very well, as we are getting all the different types of connectors to consume and use across the my company dataset pipeline. Apart from that, the GraphQL APIs provided by Data Hub are very good, allowing us to get all the information we need programmatically whenever we need it.

Regarding how the GraphQL APIs help my team in day-to-day tasks, we sometimes use custom logic to check whether the data has PII or non-PII. We have some AI model running on top of it, which requires classification. Based on the dataset URL, we are getting information about the dataset using the GraphQL APIs. GraphQL APIs are very handy, allowing us to customize properties and pass on the necessary information. For example, if we need a structured property, we can get those structured properties. If we need tags or owners, we can retrieve that as well.

Data Hub positively impacts my organization by enhancing collaboration as previously, we had to ask the team to provide the schema information. my company operates in a cross-region environment, so a person in India could wait a day to receive information about the schema from someone in the US. However, with Data Hub, we have a centralized place where we can access all the schema of the datasets, making it very helpful. Additionally, whenever there is a problem, using the lineage helps us quickly identify the impacted team or dataset.

Whenever there is an incident, we first go to Data Hub to see the downstream teams impacted and stop any jobs running on those datasets. It helps us save around eighty percent of time, as we previously had to track down information manually to find the owners, but using Data Hub, we can tag the owners of the datasets directly in the tool.

For improvements to Data Hub, I feel the security is a bit on the weaker side. We have ingestion jobs that require exact permissions for different owners, but this setup does not align with the my company grouping system. We need to create some custom grouping to manage those permissions. I would appreciate it if there were a method to consolidate all the information on a single page, which would simplify sharing permissions for running ingestion jobs.

Additionally, I do feel that the metadata test we run daily takes too long. Initially, it takes one day, which I find excessive. Ideally, we should get information within one hour. These are the two main issues that would benefit from improvement for our use case.

I have been using Data Hub for one and a half years.

Data Hub is stable in my experience. However, there are times when we attempt to upgrade it, and it may go down for a couple of minutes, but not more than that.

Data Hub handles scalability effectively, accommodating growing data and users.

I have had to reach out to Data Hub customer support multiple times. For example, when we were setting up a private link to connect to Data Hub GraphQL APIs, we required our account to be whitelisted. I have also requested some future features for our use cases. For instance, when working with a metadata test scenario, I needed to have a range date column, which was not available. I requested the Data Hub team to make it public so we could use it.

I have seen a return on investment with Data Hub. For instance, I have noticed time savings during incidents and while looking up schemas. In terms of resources, Data Hub centralizes data cataloging and classification, saving us from having to disclose PII column information to teams not utilizing it. Regarding financial metrics, I do not have specific metrics available.

Before choosing Data Hub, we looked into Unity Catalog from Databricks, but we ultimately decided to stick with Data Hub.

My advice for others looking into using Data Hub is to use it for cataloging, classification, and centralizing all your schema. Data Hub supports a variety of connectors and has excellent lineage options. Additionally, make sure to utilize the well-written documentation that can guide you in building your product solutions. I would rate this product a nine out of ten.

My main use case for Data Hub is that we use it as a library for all the data assets that we generate. It serves as an internal data mart where people can search for whatever data they need, and they can search by tags, by roles, and then add more metadata to it. This provides visibility to the data.

A specific example of how my team uses Data Hub in a real-world scenario is that we collect and manipulate a bunch of data layers. Because we have huge teams, the exposure to data that we have already manipulated can sometimes be hindered when using traditional systems. Data Hub acts as a search engine for all of the data. One example would be when the marketing team was looking for specific data around marketing. They discovered that once they searched it on Data Hub, it was easily visible. They did not have to retrieve it from the raw layer and manipulate it for their usage because another team had already built it.

Regarding how my teams interact with Data Hub, we use Data Hub with a self-hosted system. We have connectors which look into multiple data sources, manipulation engines, and orchestration layers to gather the metadata, and then that is pulled into Data Hub. This is how we get data assets in Data Hub.

The best features that Data Hub offers include primarily data discovery and data governance. Data Hub has data catalogs, which helps with the business glossary, ownership tracking, and lineage. Lineage is something that we are strongly using at this point in time. It helps us understand the impact analysis, such as what breaks if I change this column. Data Hub also provides data observability, helping us understand what data is fresh, what is not, and what has changed schema recently. Additionally, it makes our system AI and LLM ready.

The lineage feature has changed the way my team works and collaborates significantly. Because we now have data lineage through Data Hub, if we have a really huge dependent pipeline with multiple layers of upstream and downstream dependency, and something breaks in the downstream system, we can exactly pinpoint what all data assets would be affected. Having that lineage functionality helps us drill down what needs to be debugged and fixed and what exact part is breaking. It saves us time in remedying the issue.

I really like the integrations that Data Hub provides. Data Hub has a very large set of integrations that we can do with Snowflake, Databricks, BigQuery, Redshift, DBT, and Airflow.

Data Hub has positively impacted my organization as teams can now be directly dependent on one source of truth for all their data needs. The time spent finding information has become significantly smaller, which is the real productivity improvement that I have seen, impacting multiple teams throughout the organization. I estimate that we save about thirty to forty percent of the time now since we do not have to read documents or message people for specific data assets. This results in a productivity increase of around thirty to forty percent in terms of time and efficiency.

I think Data Hub can be improved by supporting the open source version better. Many features have moved to the paid version now, making it difficult for small-scale companies to operate on Data Hub because we are required to pay, even though it started as an open source project that is now essentially behind a paywall.

One needed improvement for Data Hub would be stronger AI-powered metadata discovery. I understand Data Hub has been investing in AI, but the natural language processing power on Data Hub search is not that good. The search itself is not accurate many times. Another improvement could be enhancing the DBT developer experience, such as surfacing DBT test failures directly in lineage. Additionally, when we change schema, if it could provide a risk scoring of some sort, that would also be beneficial. Lastly, automated cleanup recommendations would help because managing orphan data assets on Data Hub currently takes a lot of manual time.

I have been using Data Hub for a year.

Data Hub is pretty stable in my experience with no downtime or issues.

Data Hub's scalability has been effective, handling our organization's growth and data volume well.

I have not had to reach out to customer support.

I did not previously use a different solution before Data Hub.

My experience with pricing, setup cost, and licensing has been pleasant, and I have no complaints.

Before choosing Data Hub, we evaluated Atlan and decided on Data Hub because it has a cleaner UI and also a decent open source community to support it.

Data Hub does most of the job it is designed to do, but there could still be improvement as the industry progresses, particularly around metadata discovery. Regarding Data Hub's AI capabilities, its governance and security do the job really well as of right now. I do not have any complaints, especially around data classification, as it allows us to have control over whatever data we are displaying, including customization for PII, sensitive, and financial data. Data Hub has met our expectations regarding its accuracy and reliability of output, and there have not been any issues.

My advice to others looking into using Data Hub is that it is a pretty nice product right now with easy integration. The pricing model could be negotiated, so it is essential to keep that in mind. I would rate Data Hub a solid eight on a scale of one to ten.