The Alation Data Intelligence Platform

I am the administrator and server admin for Alation Data Catalog . We create custom processes to upload metadata into Alation Data Catalog . We are trying to get as much metadata into Alation Data Catalog as possible and integrate it with other tools that we are also building. The main purpose of Alation Data Catalog at Upstart is to help people looking for data find the data they need and how to use that data.

Recently, many people are trying to deprecate tables in the source databases and find what downstream impacts that would have. They use Alation Data Catalog lineage to determine if there are any obvious uses of these tables. This is the biggest use case recently. People also use it to find descriptions and service owners.

I use Alation Data Catalog's analytics piece, Alation Analytics, to create dashboards in Databricks  based on annotations or source comments on the databases themselves. This allows us to hold teams who own these tables accountable for missing annotations or possibly incorrect annotations on the source databases.

The best feature that we have used the most in Alation Data Catalog is the lineage and impact analysis piece. This is used in other services where we call Alation Data Catalog API to get the lineage and then use that as input to some LLM models and other services that another team builds for lineage services. We use the API extensively, and it is a great piece to get context. It is great to have all the context in one place, and we can use inputs to LLMs to chat with data when we create our own solutions, since Alation Data Catalog does not offer the exact answer we want for our custom LLMs yet. We are creating that as context input, and we are able to see descriptions, types of columns, and search easily, even though the search can be a little inconsistent sometimes.

Alation Data Catalog's scalability has met our needs as our data and user base has grown. We mostly need viewer access, so the licensing with the cloud-native solution allows unlimited viewer access. There are no other limits that we have even found except some of the API limits on the free version where we would have to pay if we wanted to increase our API calls.

Alation Data Catalog API integration has mostly been smooth. We have seen issues with the LLM API pieces or lineage, but there is not clarity around what the costs are, how many requests exist, and what constitutes a request to this API. One issue that we found before, which I believe is solved now, is using the same refresh token in multiple different processes that could run in parallel. The refresh token would create an API token, and the API token would then get invalidated if another process uses the same refresh token to create an API token. We are now starting to use service accounts by Alation Data Catalog and will have different service accounts for each process, so this does not happen anymore.

The current search functionality in Alation Data Catalog is not necessarily great. It does not do natural language searching as we would prefer. It mostly searches titles, and we would prefer it to search the description and some of the source comments to find answers that we need. Additionally, we would like to be able to search common queries or the queries that are used most frequently for certain aspects so people can get ideas of what queries they can use when they want to find a specific metric.

We have been able to find downstream impacts easier with Alation Data Catalog. Our machine learning team uses it quite often to find certain data. Our analytics team has yet to adopt it as much as we would prefer because it does not have the easy finding features for searching and finding queries that they can use or endorsed queries. They want to figure out how to use the tables and join them to find other metrics, which is difficult in the current state, though I believe there are things in the future to improve on. The other aspect is holding data producers accountable and being able to see who owns tables. Currently, that is a manual process, but we are creating an automated process to add owners to tables. If anybody has a question on a table that the description may not have answered, then we can find that out through the owner of the data in Alation Data Catalog table page.

We do not have as many P1 incidents anymore based on anecdotal evidence. Previously, a change may have happened and people did not know about the downstream impacts, which caused a lot of issues. Now it is easier to mitigate or just not encounter the P1 incidents in the first place.

The search feature of Alation Data Catalog could be improved. Alation Data Catalog Compose is also interesting in that we cannot search queries or see queries in the table page that are not published unless we go to the query history. We do not allow Compose on many items right now due to information security. Our security requirements do not allow Alation Data Catalog to access the underlying connections because we do not want people to pull in data. From a security standpoint, that is an issue, and we would like to have workarounds in certain cases. The other issue that we have found recently is along the same lines of security. We do not want to automatically sample tables in Alation Data Catalog because there could be some issue or we do not want that data being stored on a different server. However, if we have a schema that is enabled for sampling, then any new table automatically gets enabled for sampling. We have had to work around this by trying to figure out the correct permission and setup on the Databricks  side to not allow Alation Data Catalog to sample certain tables if we do not want it to, because it is not feasible to do that on Alation Data Catalog side.

I have been using Alation Data Catalog for over a year at this point.

Alation Data Catalog is stable for the most part. There are some issues with long load times for DBT models, but for the most part, it is stable. It has never gone down when we wanted to access it.

Alation Data Catalog's scalability has met our needs as our data and user base has grown. We mostly need viewer access, so the licensing with the cloud-native solution allows unlimited viewer access. There are no other limits that we have even found except some of the API limits on the free version where we would have to pay if we wanted to increase our API calls.

We do interact with customer support for Alation Data Catalog quite often for certain issues like missing Looker  lineage in some Redshift tables. The support time can vary. If the issue is well known, it is usually quick or easy to figure out. On the more complex issues like missing lineage, those took much longer than we would prefer because the engineering team has to get involved. Some of them have dragged on for six months, which is much longer than we would ever want some support issues to go on.

We do see a return on investment, though I cannot share exact numbers. There are fewer questions being asked in our ask data platform channel or some other channel about data itself since the implementation of Alation Data Catalog.

My advice for others looking into using Alation Data Catalog is to use the cloud-native solution. Using on-premises is much more difficult to deal with, and it is much more time-saving to use Alation Data Catalog cloud service provided by Alation so you do not have to deal with most of the things. Make  sure that the security requirements are met if you want to pull in data. Ensure that your security best practices allow that and there is no sensitive information that could potentially get leaked into Alation Data Catalog causing you to have to ask Alation Data Catalog to delete that data, which can take some time. I would rate this product a 7 overall.