
Arize AI
Improved AI workload visibility has reduced manual planning and supports reliable monitoring
What is our primary use case?
My main use case for Arize AI is observability of AI workloads, specifically around agent TKI and agent monitoring, as well as tracking token usage.
Some of our conventional observability monitoring tools, such as DynaTrace and DataDog, were not sufficient to monitor specific agent TKI workloads from an end-to-end observability perspective. We tested Arize AI in our separate POC environment, which gave us a better picture, especially of AI workloads and more particularly agent TKI workloads.
Arize AI is getting better at giving us a clearer picture from an end-to-end observability and monitoring perspective around monitoring some of our API exposure, while also identifying where some of our gaps are in overall observability from an end-to-end perspective.
How has it helped my organization?
Arize AI has positively impacted my organization from a visibility perspective, and we also use it for our planning purposes, where we used to second-guess.
Arize AI has helped with planning by reducing errors and avoiding some manual work that was usually based on assumptions. We now have data from Arize AI that has helped us scale, especially as we introduce new environments, allowing us to baseline our forecast.
What is most valuable?
The best features Arize AI offers include visibility, tracking of very specific metrics, and fine-grained details, which have been really helpful.
The most helpful fine-grained metrics are very specific to availability, especially for EC2 workloads and containers. There is also usage tracking, especially of token spend, and recommendations on the overall spend from a planning perspective.
I have found Arize AI really helpful compared to other conventional observability and monitoring tools.
I think Arize AI's governance and security are very strong.
I find the accuracy and reliability of Arize AI's output to be very reliable. I have found its outputs to be accurate and reliable, providing consistent and trustworthy results for my monitoring and tracking needs.
What needs improvement?
I believe Arize AI can be improved with integration with some of the other providers. It works really well with some of the major frontier AI models, but I think the API should be exposed as much as possible to non-frontier models as we go along.
I think integration can be improved. It is not a major challenge, but it requires some TKI understanding along with an understanding of Arize AI from an API perspective, so it could be improved with more detailed or better user guides.
Except for exposing some of the APIs to some non-major AI frontier models, I would say we are pretty much satisfied with Arize AI.
For how long have I used the solution?
I have been using Arize AI for about six to seven months.
What do I think about the stability of the solution?
I find Arize AI to be stable.
What do I think about the scalability of the solution?
Arize AI's scalability obviously depends on the cloud providers' backend scalability, but I find it to be very scalable.
How are customer service and support?
The customer support was top-notch. We were really supported and also had training from Arize AI for our internal team for Level 1 support, so far so good.
I would rate the customer support a nine on a scale of 1 to 10.
Which solution did I use previously and why did I switch?
We haven't used any solution related to TKI, but previously, we had DynaTrace as our observability and monitoring solution, which was mainly for application performance monitoring and typically infrastructure monitoring.
What was our ROI?
We don't have any specific metrics to share regarding return on investment, but Arize AI has really avoided manual work and improved our outcomes in terms of visibility, while also providing continuous monitoring.
What's my experience with pricing, setup cost, and licensing?
I think it's reasonably priced and I don't have any major concerns.
Which other solutions did I evaluate?
Before choosing Arize AI, we did not evaluate other options. We reviewed some Gartner recommendations, and our team also had training on Maven.com, where they had access to some of the AI lab as part of the training. That sparked our interest and provided information.
What other advice do I have?
On a scale of 1 to 10, I would rate Arize AI an eight.
I choose eight out of 10 because it satisfies most of our requirements and it's reliable. The data are consistent, and it will definitely meet our future needs as well.
My advice to others looking into using Arize AI is to test it in a POC environment and spend some time to ensure it meets your specific use case. Just don't go by public reviews, as it's a very solid product and solution that's scalable and very secure.
Which deployment model are you using for this solution?
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Arize AX Streamlined Tracing, Evals, and LLM Cost Insights for AI Agents
Beyond that, its managed cost agent breaks down LLM spend and helped us reduce our overall model-routing cost by 15%. I also find the custom evaluation methods much more advanced. In particular, I use the decision judge to organize evaluations for our agentic users’ queries. There are more than enough templates to get started with evals, instead of having to build everything from scratch.
The experiment features are what I like most, especially the playground. We can test everything there—user prompts, models, and agent output—before pushing to production, which helps us avoid low CSAT. Over time, the datasets created through experiments also help us compare the current version of guardrails and other agent configurations with previous ones, so we can see the progression visually as things evolve, increasing our Agent improvement performance drastically.
Iterative observability has transformed agent debugging and now cuts test cycles in half
What is our primary use case?
I'm using Arize AI for AI observability and evaluation for AI agents development.
I have it deployed in development and I need to check for specific questions what the agent was doing, what MCP tools it used, what thinking process it went through, what steps it chose to do in LangGraph, what nodes it went through, and if it had an error, what was the cause of the error. This way we can make the application better and better with iterations on the product. Arize AI, specifically the Phoenix tool, helps with developing this application.
Arize AI has an MCP server which is built-in. I can hook that up to Claude or GitHub Copilot and it's really helpful to extract sessions and conversations. I have key users who give me feedback about how the agent performed, and I mostly care about the negative feedback. I review with the MCP server and with the help of our coding agents what the agent deployed on development did. This way I can make the experience of the users better with each iteration. The tool also helps me with testing and evaluating the AI. I have a set of questions and I use LLM as a judge inside this tool to make at each release some assessment and tests with our agents. For each iteration I do these LLM as judge tests on a set of question datasets where I have YAML files and I have specified what the agent should do for those specific scenarios. I also have a golden dataset and I'm evaluating with that one as well using this tool.
How has it helped my organization?
I have improvements on the speed of development because with the help of Arize AI, I was able to cut on development speeds. Mostly on feedback from the client side and also from my point of view. When I as a developer deploy the agent, I have a set of questions which I verify and I can automate this process. A lot of testing and continuous improvement processes are automated using their tool.
Since using Arize AI, development speed improved by 50% to 60%. I saved approximately two days on each release on the testing and evaluation.
What is most valuable?
I think the most useful feature of Arize AI is their built-in MCP server which is providing a lot of data for our coding agents and our coding assistants in Claude or GitHub Copilot.
I can process data in bulk with Arize AI. I have approximately ten conversations with negative feedback and then it would take manually to review them two to five hours. With a coding agent and the MCP server hooked up, I can do the same review in approximately ten to twenty minutes. This helps me a lot with time management and efficiency. Also the AI agents and the coding assistant AI agents are able to come up with steps to fix those issues which I had. They receive a lot of data through that MCP server which would be impossible to give them manually. I think it's the most useful tool they have, and that's why I chose them.
What needs improvement?
I think Arize AI's UI could be better because it's pretty hard to understand. Before using their tool, I was using LangFuse and the UI from LangFuse was easier to comprehend. Also, their UI does not allow filtering for specific users or filtering on feedback. I think these minor UI changes would help a lot for users and also for non-technical users because business analysts are using the tool on my side. A lot of time they want to filter by a specific user and they cannot do that. I can do it with the MCP server, but filtering in the UI would be a great addition.
Another improvement would be some guidance on how to update your tool. If you are deploying the latest version of the tool right now, but in a month they have a new major release and then there are a lot of breaking changes, it's not easy to update it when it's deployed already. Descriptions or a guide for an AI agent or a guide for humans to deploy it would be really helpful.
For how long have I used the solution?
I have been using Arize AI for six months.
What do I think about the stability of the solution?
I think Arize AI has quite good reliability. Some things could be improved, but based on my experience, as a developer I have to do some things so it works fine. I think the overall accuracy and reliability is quite good, and they are refining it with each release. I'm thankful and fulfilled with that.
Arize AI is very stable.
What do I think about the scalability of the solution?
I think Arize AI is quite easy to scale. I did not have problems with it. It was straightforward. After deploying it, I could do replicas and everything quite easily. It's straightforward and easy.
How are customer service and support?
I don't have experience with their customer support, but I have had a call with them. I think they are quite helpful and they provide useful insights.
Which solution did I use previously and why did I switch?
I used LangFuse before.
I switched from LangFuse because it was a dependency hell. I had a lot of dependencies on LangFuse and a big percentage of them was not compatible anymore after some releases and after updates of some libraries. It was not compatible with my project. Also it could not be deployed because the deployment system was also not compatible with the new versions of LangFuse. I had to switch and I started with the team to look for a new tool and that's how I found Arize AI.
What was our ROI?
The return on investment is a lot of time saved. As I said before, I'm saving approximately two to three days per release, and it's a 50% to 60% reduction in time of testing after releases. It obviously has money implications because employees have to invest less time in the process of evaluation and observability. Also fewer employees are needed with the help of the MCP server because the coding agent can do what an employee should do manually. This way the employee has time to do other important things.
What's my experience with pricing, setup cost, and licensing?
I don't know the costs because the company did this part. Setup was not straightforward, so costs were a little high time-wise. It took a lot of time to set it up.
Which other solutions did I evaluate?
I evaluated the most mainstream options. Mostly LangSmith was the other candidate.
What other advice do I have?
I would rate Arize AI nine out of ten.
Their tool is really helpful and the things I mentioned about improvement are just minor things. It's not a big issue and would be nice to have, but overall their tools are really helpful for me.
They have quite good AI capabilities included in Arize AI. I appreciate the fact that I can choose any model I want and any LLM. That part is great. About security, I can't say much. I think they are doing a good job overall. It's straightforward to use any model I want. Overall, I think they are doing a good job.
I use private cloud for Arize AI.
As I know, I use AWS and also I use Azure from Microsoft.
I did not purchase Arize AI through the AWS Marketplace.
I had to see and read the setup steps or find some tutorial to set it up. If I didn't, it's not so easy to do it. I had to check on the database part. It has some environment variables which decide what database it uses. It took me a little time to find it. A good part of it is its built-in MCP server, which is a great addition to the tool.
Which deployment model are you using for this solution?
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Quick, Accurate Answers That Match the Prompt
Unmatched AI Monitoring with Valuable Insights
Arize Ax Makes Debugging AI Agents Fast with End-to-End Visibility
Arize AX Boosts AI Visibility and Team Collaboration
Arize AX also helps with debugging. When an AI response is not correct, it can be hard to pinpoint the exact reason. The platform provides better visibility into the AI process, which makes troubleshooting and improvement more straightforward.