Overview
You don't make an AI model deterministic. You make your confidence in it measurable. With AI, the same question can get a different answer tomorrow, so a single successful test proves very little. Models change, prompts change, providers retire the model you tested last quarter, and a wrong answer can look exactly like a right one. VERSDBench lets you choose models on evidence, change prompts safely, catch drift and know the cost.
PART OF THE VERSD AI SUITE VERSDBench is part of a trinity of products in the VERSD AI Suite from 2pi Software, which together give any organisation comprehensive support for using AI well: VERSD runs AI workflows on your own data, VERSDBench proves the quality of your AI's answers, and VERSDView (coming soon) monitors and governs AI and agent use and spend. In short: VERSD runs the AI, VERSDBench assures it and VERSDView governs it. Each product works on its own or together, and VERSDBench and VERSDView work with any AI product, not just VERSD.
HOW IT WORKS
- Write your questions: the real questions your users ask.
- Set the rules: plain-English pass/fail rules (for example "cites the guideline section" or "no hedging") and expected answers (ground truths).
- Pick what to test: a model directly, an agent, VERSD, or your own application's web API.
- Run: VERSDBench asks each question as many times as you like, and an independent AI judge scores every answer against every rule.
- Compare: pass rates, consistency, speed and estimated cost, side by side. Change one thing and run again.
100% HEADLESS, BUILT FOR AUTOMATED TESTING Everything in VERSDBench is available through a documented REST API: anything you can click, a script can do. Create questions, rules and system profiles, start an experiment with one API call, stream live progress and export the results, with no person in the loop unless you want one. That makes testing at scale practical: hundreds of questions, many models, many repetitions, run by scripts instead of people clicking through screens. VERSDBench testing integrates readily into your existing CI/CD pipeline, with scoped API keys (read or write, revocable instantly) to keep it secure: record each build, run the evaluation automatically and gate the release on your own thresholds (for example "every safety rule must pass, and the strict pass rate must not drop"). New model released? Re-run last week's experiment with one call and compare the two runs. AI agents can drive the API too: at 2pi Software, Claude sets up, runs and analyses VERSDBench evaluations through it.
SCORING YOU CAN TRUST
- An independent AI judge that explains every verdict, and doesn't use the system it's grading.
- Probes: for each rule, VERSDBench writes the opposite check. A good judge must pass the rule and fail the probe, so you get two numbers, the pass rate and the strict pass rate. A big gap means fix the judge first.
- Repetitions to measure consistency, because AI answers vary from run to run.
- Version-safe history, and human override so reviewers can correct a score.
- AI help that splits compound rules into simple, testable ones.
TESTS ANY AI SYSTEM Built-in support for Amazon Bedrock, OpenAI, Anthropic, Ollama, OpenRouter, LiteLLM and VERSD. Any other AI with a web API, in any cloud or self-hosted, can be a target: describe the request and where the answer sits in the response, including streaming responses and long-running jobs, with no code. Mix targets in one experiment to compare systems side by side.
SEE WHAT NEEDS ATTENTION The dashboard opens on runs in progress and stalled runs, score drops bigger than chance, failed runs (with one-click Re-run), quality trends and month-to-date spend. Each experiment reports latency, time to first token, tokens per second, throughput and estimated cost per run.
RUNS IN YOUR OWN AWS ACCOUNT VERSDBench deploys into your own AWS account from a single CloudFormation template. Your questions, results and credentials stay within your own infrastructure. Tutorials: https://versd.com.au/versdbench/tutorials/
Highlights
- Answer quality measured, not assumed. Test any model, agent or application with a web API against your own questions, plain-English rules and expected answers, and compare pass rates, consistency, speed and estimated cost side by side.
- Scoring you can trust: an independent AI judge explains every verdict, probes check the judge itself (pass rate and strict pass rate), repeats measure consistency, and history is version-safe with human override.
- 100% headless: anything you can click, a script can do through the REST API. Start runs with one call, stream progress, export results and gate releases in CI/CD with scoped API keys, for highly automated AI testing.
Details
Features and programs
Financing for AWS Marketplace purchases
Pricing
Dimension | Description | Cost/unit |
|---|---|---|
Standard Users | Number of active standard users in the subscription period | $0.013 |
Dimensions summary
Top-of-mind questions for buyers
Vendor refund policy
Non-refundable.
How can we make this page better?
Legal
Vendor terms and conditions
Content disclaimer
Delivery details
VERSDBenchCloudformation
CloudFormation for the versdbench server and supporting infrastructure.
Deploys a VPC, one EC2 host that runs versdbench, an RDS PostgreSQL database, an SES email identity for your domain, Secrets Manager secrets, a KMS key, and the Route53 record for your domain.
CloudFormation Template (CFT)
AWS CloudFormation templates are JSON or YAML-formatted text files that simplify provisioning and management on AWS. The templates describe the service or application architecture you want to deploy, and AWS CloudFormation uses those templates to provision and configure the required services (such as Amazon EC2 instances or Amazon RDS DB instances). The deployed application and associated resources are called a "stack."
Version release notes
1.0.0
- Organisation and Group based user management
- Admin console
- Compare mode for side-by-side experiment execution copmparison
- System wizard for ease-of-use
- Bulk import/export
Additional details
Usage instructions
Configure the offering via marketplace for your desired infrastructure options. Deploy and navigate to your configured application domain name by port forwarding to the admin console. The setup wizard will walk you through application set up.
Support
Vendor support
VERSDBench provides technical, billing, and general customer support for organisations deploying and using the platform through AWS Marketplace.
Email Support: support@versd.com.au Phone Support: 1800 961 919 Website: https://versd.com.au
The VERSDBench support team is based in New South Wales, Australia (AEST). Support requests submitted via phone or email are monitored during standard business hours, Monday to Friday. The expected response time for all enquiries is within 24 business hours.
VERSDBench provides:
- Technical support and troubleshooting
- Deployment and configuration assistance
- Billing and account enquiries
- General product enquiries
All enquiries are triaged internally and directed to the appropriate VERSDBench support personnel for responsive assistance.
Setup and deployment guidance is available via https://versd.com.au and through onboarding documentation provided upon purchase. Guidance includes enterprise integration support, authentication setup, and deployment best practices.
VERSDBench is designed with enterprise-grade security and sovereign-safe deployment principles. The platform supports SSO-enabled authentication for secure enterprise-wide access management and group- and role-based access controls aligned with organisational policies.
For AWS Marketplace (CloudFormation) deployments, security guidance covering encryption configuration, credential management, authentication setup, and credential rotation best practices is provided in the accompanying deployment documentation supplied during onboarding.
AWS infrastructure support
AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.