Sold by: Patronus AI
The Patronus AI Platform is the leading automated AI evaluation and security product. The Platform enables enterprise development teams to score LLM performance, generate adversarial test cases, benchmark LLMs, and more. Customers use Patronus AI to detect LLM mistakes at scale and deploy AI products confidently.
Overview
The Patronus AI Platform enables engineering teams to test, score, and benchmark LLM performance on real world scenarios, generate adversarial test cases at scale, monitor hallucinations and other unexpected and unsafe behavior, and more.
Customers use the Patronus AI Platform as soon as they have any kind of an LLM or LLM system in their hand. The platform is primarily used in 2 key parts of the user journey: AI product pre-deployment and AI product post-deployment. The product is typically used with not just LLMs, but also retrieval-based LLM systems, agents, routing architectures, and more. There are also 2 types of key product offerings: 1) cloud-hosted solution, and 2) on-prem self-hosted offering.
For pre-deployment: Customers use several features in the web platform for offline LLM evaluation and experimentation, all in one place. In the Evaluation Run workflow, customers can select or define parameters like the LLM and its associated settings, evaluation dataset, and criteria.
For post-deployment: Customers use the Patronus API and the LLM Failure Monitoring dashboard for LLM testing and evaluation in CI and production. The API solution allows customers to validate, log, and address LLM failures in real-time. To accompany the API and manage the alerts, there is also an LLM Failure Monitoring dashboard in the web platform to visualize, filter, and aggregate statistics on LLM failures.
Highlights
- Retrieval-Augmented Generation (RAG) Testing: Verify that your LLM-based retrieval systems consistently deliver reliable information using our retrieval evaluation API.
- Evaluation Runs: Leverage our managed service for evaluations to auto-generate test suites, score model performance on real world scenarios, benchmark LLMs, and more.
- LLM Failure Monitoring: Continuously evaluate, track, and visualize LLM system performance for your AI product in production.
Details
New
Introducing multi-product solutions
You can now purchase comprehensive solutions tailored to use cases and industries.
Features and programs
Financing for AWS Marketplace purchases
AWS Marketplace now accepts line of credit payments through the PNC Vendor Finance program. This program is available to select AWS customers in the US, excluding NV, NC, ND, TN, & VT.
Pricing
Pricing is based on the duration and terms of your contract with the vendor. This entitles you to a specified quantity of use for the contract duration. If you choose not to renew or replace your contract before it ends, access to these entitlements will expire.
Additional AWS infrastructure costs may apply. Use the AWS Pricing Calculator to estimate your infrastructure costs.
Dimension | Description | Cost/12 months |
|---|---|---|
Evaluation Samples | Total number of samples evaluated using Patronus AI Platform. | $1,000,000.00 |
Vendor refund policy
If you cancel your subscription within 48 hours of purchase, you can get a full refund. All other refunds would happen on a case-by-case basis. Reach out to contact@patronus.ai to request a refund.
Legal
Vendor terms and conditions
Upon subscribing to this product, you must acknowledge and agree to the terms and conditions outlined in the vendor's End User License Agreement (EULA) .
Content disclaimer
Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.
Delivery details
Software as a Service (SaaS)
SaaS delivers cloud-based software applications directly to customers over the internet. You can access these applications through a subscription model. You will pay recurring monthly usage fees through your AWS bill, while AWS handles deployment and infrastructure management, ensuring scalability, reliability, and seamless integration with other AWS services.
Support
Vendor support
We have a 24-hour response time SLA for all buyers. Please reach out to contact@patronus.ai if you are experiencing any issues.
AWS infrastructure support
AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.
Updated weekly
Sentiment is AI generated from actual customer reviews on AWS and G2
Functionality
Ease of use
Customer service
Cost effectiveness
Positive reviews
Mixed reviews
Negative reviews
AI generated from product descriptions
Retrieval-Augmented Generation Testing
Verification of LLM-based retrieval systems through retrieval evaluation API to ensure consistent delivery of reliable information.
Automated Test Suite Generation
Auto-generation of test suites and scoring of model performance on real-world scenarios with benchmark capabilities for LLMs.
LLM Failure Monitoring and Visualization
Continuous evaluation, tracking, and visualization of LLM system performance in production environments with failure aggregation and filtering capabilities.
Adversarial Test Case Generation
Generation of adversarial test cases at scale to detect hallucinations and unexpected unsafe behavior in LLM systems.
Real-time LLM Validation and Logging
Real-time validation, logging, and addressing of LLM failures through API integration with support for CI and production environments.
Distributed Tracing and Observability
Detailed tracing of all LLM calls and relevant application logic with UI for inspecting and debugging logs
Prompt Management and Experimentation
Built-in tools for managing prompts and conducting experiments to test application behavior before deployment
Automated Quality Evaluation
LLM-as-a-Judge approach for automatically scoring application quality with support for user and employee feedback collection
Multi-Framework SDK Integration
Python and Typescript SDKs with integrations for Llama Index, Langchain, OpenAI, Dify, and Litellm frameworks
Performance Analytics and Monitoring
Analytics and evaluation tools to monitor LLM performance, track metrics including cost and latency, and analyze user behavior patterns
Prompt Engineering and Comparison
Side-by-side comparisons between multiple prompts, parameters, models, and model providers across test cases for optimization.
Workflow Orchestration
Ability to prototype and deploy AI workflows that chain business logic, data, APIs, and dynamic prompts for various use cases.
Evaluation and Testing Framework
Creation of test case banks to evaluate and identify optimal prompt and model combinations across multiple scenarios.
Semantic Search and Retrieval
Document retrieval capability to extract company-specific data and use it as context in LLM calls.
Monitoring and Proxy Infrastructure
Reliable proxy layer connecting applications to model providers with request tracking for debugging and quality monitoring.
Standard contract
No
No
Customer reviews
No customer reviews yet
Be the first to review this product . We've partnered with PeerSpot to gather customer feedback. You can share your experience by writing or recording a review, or scheduling a call with a PeerSpot analyst.