WhyLabs AI Observatory: The Data and ML Monitoring Platform

My main use case for WhyLabs is its robust data observability capabilities; it provides continuous monitoring of data pipelines and ML models, allowing my team members to quickly identify issues like data drift, model degradation, and training-serving skew. The platform's privacy-preserving integration ensures that the data can be analyzed without moving or duplicating it, which is critical for maintaining security and privacy in sensitive industries like ours.

A particular time when WhyLabs helped me catch an issue early was when it detected and addressed data drift and data integrity problems, which is crucial for maintaining accurate and reliable machine learning models.

The monitoring capabilities of WhyLabs AI Observatory are the main reason why we use it, as we have very limited control of external LLM and services. WhyLabs provides alerts and insights into those LLM models, enabling us to identify and address potential issues promptly.

Setting up the monitoring has impacted my team positively, as monitoring model performance and input data quality in one place has helped us to catch issues very fast. I love it because it helps detect and address data drift and data integrity problems early, which is crucial for maintaining accurate and reliable machine learning models.

WhyLabs solves most of the machine learning model monitoring needs that models often have while being affordable.

In terms of accuracy and reliability, consistent, relevant outputs help us improve productivity, streamline our workflows, and support better decision-making.

The best features WhyLabs offers include very helpful and proactive customer support, a tool that allows for easy ingestion of a big number of features and setting up initial monitoring on them, and the ability to raise alerts to a specific group of users via specific channels such as Slack, which is very helpful. One of the features that I heavily rely on is the robust data observability capabilities.

What makes the data observability capabilities stand out compared to others I have tried is that it provides continuous monitoring of data pipelines and machine models, allowing my team to quickly identify issues such as data drift, model degradation, and training-serving skew.

I also love the platform's privacy-preserving integration, which ensures data can be analyzed without moving or duplicating it.

I love the alerts where we are able to raise them to a specific group of users via a specific channel, and we use Slack, which is very helpful.

Regarding WhyLabs's AI capabilities, I think its governance and security features are very promising; our data is handled very well with a lot of security and care.

It can be challenging to set up monitoring in the correct way when it comes to sensitivity, requiring a lot of trial and error.

A few cons include that the dashboards are in beta, and while functional, they lack polish in terms of user interface.

In the dashboards, I wish to see specific functionalities or visualizations that they are working actively on; probably a few months after this review, they may get it ready to fix it.

Additionally, defining groupings by variables must be done at ingestion time, limiting flexibility for post-ingestion analysis.

I have been using WhyLabs for two years.

WhyLabs is very stable; it focuses on monitoring data quality, detecting anomalies, and tracking model performance, which helps my team identify potential issues early and maintain a reliable AI workflow.

WhyLabs's scalability has been very promising because it offers strong scalability potential for my organization, enabling us to manage our growing AI and machine learning workloads efficiently.

I have had to reach out to customer support several times, and they are very easy to reach out to and very responsive and professional. My experience overall has been very positive, especially because of the available documentation and technical resources, which were helpful for understanding the platform, configuring monitoring, and addressing implementation questions. They were quick to analyze and offer solutions, and I appreciate responsive technical assistance and clear guidance, especially when integrating our AI monitoring into our existing AWS workflows.

A nine out of ten would be a fair rating for customer support.

We did not switch from any solution.

Deploying WhyLabs in our environment is easy and straightforward; the setup process is manageable, and the documentation helps us understand the configuration and integration requirements.

My experience with the configuration process is relatively straightforward and easy because the documentation and available configuration options help simplify integration with our existing environment.

WhyLabs integrates well and very seamlessly with our AWS-based environment, supporting our AI observability and helping us monitor data quality and model performance.

We have seen a positive return on investments mainly by reducing manual monitoring to save on time and cost, identifying data quality issues earlier, and minimizing time spent troubleshooting AI models. This has improved our productivity, reduced operational overhead, and helped our engineers focus on higher-value tasks, and this is promising to our customers because the projects are delivered on time or hitting their deadlines.

The procurement process of WhyLabs was relatively straightforward due to the available product information and licensing options that helped us understand the requirements and evaluate its suitability for our AI monitoring needs.

WhyLabs offers a promising approach to managing AI monitoring controls and costs; I value the transparent billing, clear usage metrics, and predictable pricing because they help with budgeting and resource planning.

WhyLabs offers good potential value for any organization looking to improve AI observability and model monitoring; the pricing and licensing have been manageable overall.

Before choosing WhyLabs, we considered two other AI observability tools and monitoring solutions, which included Evidently AI and Fiddler AI.

I would rate WhyLabs a four out of five.

The reason why I chose a four is that WhyLabs delivers all the standard features metrics accurately; tracking data and model drift is very straightforward using monitors. The platform also supports custom metrics creation during or after ingestion.

My advice for organizations looking to improve AI observability, data quality, and machine learning model monitoring is to go for WhyLabs.

Amazon Web Services (AWS)

My main use case for WhyLabs was for LLM monitoring and observability. At that time, I had an AI application that I deployed on Vertex AI, and I used WhyLabs for the observability, logging, and monitoring of that application and the model.

I can provide a specific example of how I used WhyLabs for monitoring my LLM application. It was a multi-agent system with around four agents involved, and each agent had around seven or eight tools that it could use or invoke. Whenever a user sent a query to the main agent, its responsibility was to delegate the request among the other sub-agents. Each sub-agent could communicate with each other using the A2A protocol and call their tools. I monitored how the request progressed through the system. For instance, if a user sent a request to one agent, which then transferred it to a third agent, the third agent used a tool, and then it went to the seventh agent. I could easily monitor all this communication between the agents, the logging time, the request, the response, any errors, and any guardrails I wanted in my application in WhyLabs.

This was my only use case, and then WhyLabs got discontinued. WhyLabs was acquired by Apple in January or February 2025. The company then open-sourced their software so that anyone can use it. It is now open-source software available on GitHub where you can set it up yourself and use it.

WhyLabs's best features are real-time guardrails, PII personal information data detection, hallucination mitigation, and monitoring. It has a centralized dashboard so I can create a project and see an overall summary of the dashboards, and I can check the health metric on specific dates or specific times for WhyLabs or for the application. Additionally, it provides an alerting system. If there is an error or the system is down, it generates an alert via email.

Out of all those features, I find the PII detection and the monitoring most valuable in my day-to-day work because it is very hard to monitor an LLM application. As I mentioned earlier, it was a multi-agent system and a query can go from one agent to another agent very easily, which created problems in debugging how the request was progressing and how the data flow was happening. The monitoring and the PII detection of the guardrails are the three features most useful to me. Regarding the guardrails or the PII detection, if I do not want my PII data given to the agents or any LLM, this feature is particularly useful in that scenario.

WhyLabs has positively impacted my organization by reducing the error time and debugging time. It has increased and enhanced the user experience. When the application is down, I receive alerts, which has reduced a significant amount of time for my team.

Regarding how WhyLabs can be improved, since it is not available in the market as of now, improvements cannot be made to the product itself. However, there is an open-source version that anyone can set up on their machine and try to accomplish the same things.

I do not think there is anything else needed for improvement.

I was using it in 2024 for around 1.5 years.

WhyLabs has saved my team time by 30 to 40%.

Regarding WhyLabs's AI capabilities, I believe its governance and security are totally secured. It was deployed in our on-premises infrastructure, so all the data remains in our infrastructure only. The guardrails and the PII detection work perfectly. I have not seen any scenario where it has not generated an alert for PII data or the guardrails have not worked, so it performed very well.

In terms of WhyLabs's AI capabilities, I believe it is totally accurate. I used it for around 1.5 years, and it was the best software available, but it was discontinued. However, it was a very good software.

My advice to others considering WhyLabs is that as of now it is open-source, and you can set it up on your own machine for free and use it. It has very good features. I would rate this product a 10 out of 10.

On-premises

Amazon Web Services (AWS)

What do you like best about the product?

The UI is very user friendly and the support team is extremely responsive and helpful.

What do you dislike about the product?

Some features such as deleting a profile or changing the data type of a feature can only be done through the API.

What problems is the product solving and how is that benefiting you?

Data and concept drift detection
Model performance monitoring

What do you like best about the product?

* The customer support is very helpful and proactive
* Tool allows for easy ingestion of big number of features and setting up initial monitoring on them
* We can use it to monitor both: input quality and model performance
* The alerts can be raised to specific group of users via specific channels (email/slack), which is helpful

What do you dislike about the product?

* It can be challenging to setup the monitoring in the correct way when it comes to sensitivity - it requires a lot of trial and error
* Some actions are not possible via UI and require specific API calls
* Documentation can be hard to navigate

What problems is the product solving and how is that benefiting you?

Monitoring model performance and input data quality in one place.

What do you like best about the product?

Great to collaborate with; very responsive; really appreciate their OHs to help out with issues that pop up; many out-of-the-box solutions for different kinds of ML models which really helped us out given the wide variety of ML models we run at the company.

What do you dislike about the product?

Nothing major to mention! We got everything resolved and the team was very helpful.

What problems is the product solving and how is that benefiting you?

Data Drift and ML Monitoring

What do you like best about the product?

One of the standout features of WhyLabs is its robust data observability capabilities. It provides continuous monitoring of data pipelines and ML models, allowing teams to quickly identify issues like data drift, model degradation, and training-serving skew. The platform's privacy-preserving integration ensures that data can be analyzed without moving or duplicating it, which is critical for maintaining security and privacy in sensitive industries like healthcare and finance​

What do you dislike about the product?

One potential drawback of WhyLabs is its relatively limited user reviews and feedback due to its newness in the market, making it harder for potential users to gauge its real-world performance at scale. This lack of detailed reviews can raise concerns about its maturity and support infrastructure​.Additionally, since it’s a newer platform, some advanced features might still be in development, and there could be steep learning curves for teams unfamiliar with observability tools in machine learning​.

What problems is the product solving and how is that benefiting you?

Data quality issues: It helps detect and address data drift and data integrity problems early, which is crucial for maintaining accurate and reliable ML models​

What do you like best about the product?

I like the privacy preserving solutions for scaling AI models. I like that WhyLabs offer responsive support and detailed documentation.

What do you dislike about the product?

I dislike that the platform might be overly technical for users who are not well-versed in AI or data science

What problems is the product solving and how is that benefiting you?

WhyLabs helps me solve issues like data drift and performance degradation in my AI models. This is crucial because I am working with sensitive medical data.

What do you like best about the product?

WhyLabs is the second observability platform I have ever used, and I can say the core features I like about the platform is that it is easy to set up and implement the features, the checks and metrics were already pre-loaded so I did not need to do much in configuring the application, and monitoring was not difficult to get started with. It also integrates well with the serving and data libraries we used for the production tutorial setup.

What do you dislike about the product?

Nothing so far, I only experienced a stability issue once (sometime in 2022), but support was able to help me quickly fix it.

What problems is the product solving and how is that benefiting you?

Since 2022, I have sparsely used WhyLabs to monitor the quality of datasets for one client and 2 customers (because it was not their core requirement but a nice piece of their stack to have). whylogs seemed like the perfect choice for a consultant that clients did not want to entirely release their data to; I found that it only captures the profile and stats info instead of the raw data here. Recently, I started testing out LLM security features with LangKit and I cannot believe how quick it is to use. I followed a workshop a few months ago that showed me how to detect jailbreak attempts and toxicity in LLM inputs and outputs using LangKit. Took that learning and now with a client's project, we have tested out logging the telemetry data from the evaluation to WhyLabs. Looks good so far, so once I upgrade the pricing limit for this client, we plan to scale our usage here. Excited about this one.

What do you like best about the product?

I've used WhyLabs for a few weeks and I was extremely pleased with it!
I will evaluate some dimensions of the tool that summarize my experience with it.

Easy Data Ingestion:
The ingestion API is straightforward to use and supports multiple connectors such as BigQuery, Databricks, and Spark, making data importation easy. Whylabs' use of Data Profiling ensures fast and secure data processing, eliminating the need to upload entire datasets, and making all the process very secure, since your data doesn't leave your servers.

Reliable Data Features:
Whylabs delivers all standard feature metrics accurately. Tracking data and model drift is very straightforward using Monitors.
Also, the platform supports custom metrics creation during or after ingestion.
Grouping by variables (segments) works well but must be defined during ingestion. Then you can analyze dataset features and track model performance per segment.

Flexible Monitors:
The monitoring system in Whylabs is highly adaptable and user-friendly, covering multiple variables with ease.
Monitors are easy to set up via the UI or JSON import, with summarized notifications for each monitor, keeping users informed without overwhelming them.
Additionally, monitors are JSON serializable, which is very helpful since you can track them with version control.

User-Friendly Usability:
Whylabs have a clean and intuitive UI, simplifying navigation for users.
While some advanced features may require programming knowledge, most tasks can be accomplished within the UI.
Thanks to data profiling, Whylabs delivers speedy performance without compromising on accuracy.

Solid Documentation:
The documentation provided by Whylabs is comprehensive and easy to understand, enabling users to make the most of the platform.

Pricing:
It's simply cheaper than its competition while having top notch features.

Customer Support:
They are always very helpful, answering all our questions and having several calls showcasing us different uses cases directly on the platform.

Overall, Whylabs offers a straightforward, efficient and affordable solution for monitoring Machine Learning models, with easy data ingestion, reliable feature analysis, and flexible monitoring options.

What do you dislike about the product?

There are a few cons:
- Dashboards are in beta, and while functional, they lack polish in terms of user interface. They are working actively on this, so probably a few months after this review this may be already fixed.
- Defining groupings by variables must be done at ingestion time, limiting flexibility for post-ingestion analysis.

That being said, they are very open to feedback and they may change or add features based on your needs.
In our case, dashboards were important and they are working on them.

What problems is the product solving and how is that benefiting you?

It solves most of the ML model monitoring needs that ML models often have while being affordable.

What do you like best about the product?

WhyLabs is an exceptional observability tool for applications built with large language models. Its ease of use enables us to integrate langkit into the existing architecture. Additionally, they have an excellent customer support community.

What do you dislike about the product?

Building custom applications with large language models is a challenging experience. Better documentation about how to effectively use Whylabs in monitoring applications with LLMs would be helpful.

What problems is the product solving and how is that benefiting you?

WhyLabs is very helpful in monitoring applications built with large language models by providing a highly simplified observability tool. This has overcome the challenges we were facing with our productionized application.