Tracing ai agents has boosted debugging, cut latency, and optimizes our support chatbot
What is our primary use case?
My main use case for
Comet is building AI agents, optimizing them, getting traces, and experimentation.
A specific example of how I use Comet for optimizing AI agents and tracing is with our chatbot called Oscar, which is a RAG-based, LLM-based chatbot where users can manage their chats, ask questions regarding the products, and it also acts as a support agent. Comet helps us log the traces, capture them, organize the LLM calls, and optimize the costing accordingly.
Comet has an inbuilt AI that helps us get conclusions for the cost of LLM calls and many other things.
What is most valuable?
The best features Comet offers in my opinion are the datasets and the prompt library, along with the evaluation. The traces are very deep, covering the tool calls and everything, so we have flow diagrams.
The flow diagrams help my team by aiding the log traces to capture and organize application calls, which helps in tracing from the first to last call if an API gets hit using LLM. We analyze how it works, how it travels, and each log trace using the flow diagram, which provides a visual representation.
Testing with multiple test datasets has helped us significantly.
Comet has positively impacted my organization as we have seen a drastic change. The product has improved since we started using Comet, and debugging has become much easier.
Debugging became easier for us because the tracing helps us in production to trace an LLM call, save the cost, reduce the tool calling, optimize the AI responses, increase the speed, and reduce the latency.
What needs improvement?
Comet can be improved by adding the MCP server to integrate with Chat GPT and other applications. If they provide skills to integrate while building applications to seamlessly incorporate Comet in AI agents, it would help significantly.
For how long have I used the solution?
I have been using Comet for within one year.
What do I think about the stability of the solution?
Comet is stable, and I have not faced any bugs as of now; it is going very well.
What do I think about the scalability of the solution?
Comet handles scalability well, accommodating growth and increased workloads.
How are customer service and support?
We have not reached out to customer support yet because we have not needed support as all things are going great.
Which solution did I use previously and why did I switch?
We previously used
Grafana, which was not meant for our purpose, so we felt Comet is better in terms of LLM Ops and tracing AI queries.
What was our ROI?
I have seen a return on investment since time has obviously been saved and productivity has improved. With the automation brought by Comet, the work we were doing with other tools has changed, so time has been saved and while we have not calculated the cost saved yet, it is likely there.
What's my experience with pricing, setup cost, and licensing?
My experience with pricing, setup cost, and licensing has been straightforward and easy to understand, offering unlimited team members and some metrics without needing customization for our plan yet.
Which other solutions did I evaluate?
Before choosing Comet, I evaluated
Sentry, but it is not more focused on AI; it is a combination of various logging and tracing software.
What other advice do I have?
My advice for others looking into using Comet is that if they are utilizing LLM bots at the enterprise level, tracing is essential, and Comet offers the best solution to trace logs and optimize the overall solution. I would rate this product a 9 out of 10.
Experiment and asset tracking enhance model development and ease of on-prem maintenance
What is our primary use case?
I use Comet for experiment and asset tracking during model development, as well as to support model reproducibility and transparency. I also appreciate the ability to perform an on-prem installation without the need to maintain the installation.
How has it helped my organization?
Previously, we had an on-prem installation that required frequent re-deployment due to internal security standards, which could cause down-time during model development. Using Comet within SageMaker streamlined the deployment process to require zero maintenance and also simplified billing.
What is most valuable?
Model metric tracking and comparison has been extremely beneficial. Comet's customer service has also been excellent. Any issue we've had, they have been able to help us resolve.
What needs improvement?
SageMaker itself has a cumbersome interface, which makes launching Comet somewhat of a hassle.
For how long have I used the solution?
I have used the solution for 3 months.
Which deployment model are you using for this solution?
On-premises
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Comet.ml: Streamlining Machine Learning and Collaborative Experiment Tracking Platform
What do you like best about the product?
Comet.ml provides an easy-to-use interface for tracking experiments, comparing results, and reproducing past results. This helps data scientists and machine learning engineers to keep track of their progress and make informed decisions based on their experiments. Comet.ml integrates with popular version control systems like Git, allowing users to track changes in their code and experiments over time.
What do you dislike about the product?
Comet.ml may not be suitable for large-scale machine learning projects, as it has limited scalability compared to other solutions. Some users may find the platform's user interface and features to be limited, as it may not provide the level of customization they need for their projects.
What problems is the product solving and how is that benefiting you?
Machine learning projects can involve a large number of experiments and it can be difficult to keep track of all the results and make decisions based on them. Comet.ml provides a platform for tracking experiments, comparing results, and reproducing past results, making it easier to manage machine learning projects.
Solid platform overall but there's competition
What do you like best about the product?
Simplicity to integrate into my project. Nice UI and UX overall
What do you dislike about the product?
Expensive and not so customizable overall. There are platforms that compete with this one and have better offerings, which is why i switched.
What problems is the product solving and how is that benefiting you?
Helps me speed up building my neural networks and ML tests...
Easy to Use !! Great UI
What do you like best about the product?
User interface
Easy to use
Support different View and Easy to search Text
What do you dislike about the product?
Price.
time take to pull data
small notification view
What problems is the product solving and how is that benefiting you?
Code Debug
Application monitoring
User point of view
What do you like best about the product?
It's provide the best ai interface in handling things. The user interface is excellent and all AI tools can be used abruptly with diffrebt build functions. It competes with the different launched ml in market.
What do you dislike about the product?
The comet.ml runs causly with the interruption with softwares , therefore sometimes slows down but it has ggod backup notion with carry forward improvements in consoles.
What problems is the product solving and how is that benefiting you?
AI related machine components are useful in solving the things both financially and end-ser type. The level of productivity increases with a diffrent level of business satisfication.
Build and customize better ML models
What do you like best about the product?
Comet.ml is one of the best tools to develop, customize and combine the data in the format you want, which makes it more productive. Comet.ml offers various amount features from monitoring to tracking of the experiments.
What do you dislike about the product?
There is no dislike using comet.ml. Goals that need intense demands can be achieved effortlessly.
What problems is the product solving and how is that benefiting you?
Models can be optimized and managed in their ML lifecycle. The data can be combined and represented in the format needed by the user. Provides extensive support in whichever cloud it is running.
Easy to use and integrate
What do you like best about the product?
User-friendly interface for model training, easy to use, the dashboard makes visualizations convenient
What do you dislike about the product?
Somewhat limited model tracking abilities. Documentation could be improved.
What problems is the product solving and how is that benefiting you?
Used to manage end-to-end ML life cycle. Has helped streamline the process significantly and improved transparency within the team
Good Platform to share ML data
What do you like best about the product?
It provides a single platform to share ML experiments and models. Helps to compare data and insights from different people in an efficient way. This helps save a lot of time.
What do you dislike about the product?
It takes some time to ramp up on this. I wish that the documentation was more crisp or polished. A better adoption on the academic side would help students be familiar with this tool earlier.
What problems is the product solving and how is that benefiting you?
We use it to visualize our training data and it helps us understand the data and models better. Additionally, we don't have to spend time in talking to specific people since all data is available on the dashboard
Review ML program Comet.ML
What do you like best about the product?
The easy function to use and the platform is easy to access
What do you dislike about the product?
The complicated of the program and slowness
What problems is the product solving and how is that benefiting you?
The collaboration and reduce the time to work on the project
