Listing Thumbnail

    VERSDBench: Continuous AI Evaluation for Any Model, Agent or API

     Info
    Deployed on AWS
    VERSDBench, part of the VERSD AI Suite from 2pi Software, tells you with evidence whether your AI system still gives the answers it should, and what it costs to run. Test any model, agent or application with a web API against your own questions, plain-English rules and expected answers. An independent AI judge scores and explains every answer, and probes check the judge itself. VERSDBench is fully headless: anything you can click, a script can do through its REST API, so a highly automated testing pipeline that re-runs your evaluations on every model, prompt or release change is readily achievable. It runs in your own AWS account and works with any AI product, not just VERSD.

    Overview

    You don't make an AI model deterministic. You make your confidence in it measurable. With AI, the same question can get a different answer tomorrow, so a single successful test proves very little. Models change, prompts change, providers retire the model you tested last quarter, and a wrong answer can look exactly like a right one. VERSDBench lets you choose models on evidence, change prompts safely, catch drift and know the cost.

    PART OF THE VERSD AI SUITE VERSDBench is part of a trinity of products in the VERSD AI Suite from 2pi Software, which together give any organisation comprehensive support for using AI well: VERSD runs AI workflows on your own data, VERSDBench proves the quality of your AI's answers, and VERSDView (coming soon) monitors and governs AI and agent use and spend. In short: VERSD runs the AI, VERSDBench assures it and VERSDView governs it. Each product works on its own or together, and VERSDBench and VERSDView work with any AI product, not just VERSD.

    HOW IT WORKS

    1. Write your questions: the real questions your users ask.
    2. Set the rules: plain-English pass/fail rules (for example "cites the guideline section" or "no hedging") and expected answers (ground truths).
    3. Pick what to test: a model directly, an agent, VERSD, or your own application's web API.
    4. Run: VERSDBench asks each question as many times as you like, and an independent AI judge scores every answer against every rule.
    5. Compare: pass rates, consistency, speed and estimated cost, side by side. Change one thing and run again.

    100% HEADLESS, BUILT FOR AUTOMATED TESTING Everything in VERSDBench is available through a documented REST API: anything you can click, a script can do. Create questions, rules and system profiles, start an experiment with one API call, stream live progress and export the results, with no person in the loop unless you want one. That makes testing at scale practical: hundreds of questions, many models, many repetitions, run by scripts instead of people clicking through screens. VERSDBench testing integrates readily into your existing CI/CD pipeline, with scoped API keys (read or write, revocable instantly) to keep it secure: record each build, run the evaluation automatically and gate the release on your own thresholds (for example "every safety rule must pass, and the strict pass rate must not drop"). New model released? Re-run last week's experiment with one call and compare the two runs. AI agents can drive the API too: at 2pi Software, Claude sets up, runs and analyses VERSDBench evaluations through it.

    SCORING YOU CAN TRUST

    • An independent AI judge that explains every verdict, and doesn't use the system it's grading.
    • Probes: for each rule, VERSDBench writes the opposite check. A good judge must pass the rule and fail the probe, so you get two numbers, the pass rate and the strict pass rate. A big gap means fix the judge first.
    • Repetitions to measure consistency, because AI answers vary from run to run.
    • Version-safe history, and human override so reviewers can correct a score.
    • AI help that splits compound rules into simple, testable ones.

    TESTS ANY AI SYSTEM Built-in support for Amazon Bedrock, OpenAI, Anthropic, Ollama, OpenRouter, LiteLLM and VERSD. Any other AI with a web API, in any cloud or self-hosted, can be a target: describe the request and where the answer sits in the response, including streaming responses and long-running jobs, with no code. Mix targets in one experiment to compare systems side by side.

    SEE WHAT NEEDS ATTENTION The dashboard opens on runs in progress and stalled runs, score drops bigger than chance, failed runs (with one-click Re-run), quality trends and month-to-date spend. Each experiment reports latency, time to first token, tokens per second, throughput and estimated cost per run.

    RUNS IN YOUR OWN AWS ACCOUNT VERSDBench deploys into your own AWS account from a single CloudFormation template. Your questions, results and credentials stay within your own infrastructure. Tutorials: https://versd.com.au/versdbench/tutorials/ 

    Highlights

    • Answer quality measured, not assumed. Test any model, agent or application with a web API against your own questions, plain-English rules and expected answers, and compare pass rates, consistency, speed and estimated cost side by side.
    • Scoring you can trust: an independent AI judge explains every verdict, probes check the judge itself (pass rate and strict pass rate), repeats measure consistency, and history is version-safe with human override.
    • 100% headless: anything you can click, a script can do through the REST API. Start runs with one call, stream progress, export results and gate releases in CI/CD with scoped API keys, for highly automated AI testing.

    Details

    Delivery method

    Delivery option
    VERSDBenchCloudformation

    Latest version

    Operating system
    Ubuntu 24.04

    Deployed on AWS

    Features and programs

    Financing for AWS Marketplace purchases

    AWS Marketplace now accepts line of credit payments through the PNC Vendor Finance program. This program is available to select AWS customers in the US, excluding NV, NC, ND, TN, & VT.
    Financing for AWS Marketplace purchases

    Pricing

    VERSDBench: Continuous AI Evaluation for Any Model, Agent or API

     Info
    Pricing is based on actual usage, with charges varying according to how much you consume. Subscriptions have no end date and may be canceled any time.
    Additional AWS infrastructure costs may apply. Use the AWS Pricing Calculator  to estimate your infrastructure costs.

    Usage costs (1)

     Info
    Dimension
    Description
    Cost/unit
    Standard Users
    Number of active standard users in the subscription period
    $0.013

    AI Insights

     Info

    Dimensions summary

    This listing uses one pricing dimension: Standard Users, counted as the number of active users during the subscription period. Pricing is usage-based, so your cost scales with how many people actively use the platform. Charges appear on your AWS invoice, and you can cancel at any time. The software deploys into your own AWS account, so your test data and results stay with you. Note that any AI model usage you run through the platform is billed separately from this user-based charge.

    Top-of-mind questions for buyers

    A Standard User is an active user account in the platform during the subscription period. People can belong to several organizations, and each organization's data stays separate at the database level. Your charge tracks how many users are active, not how many tests or runs you execute.
    No. The user charge depends only on the number of active users, not on test volume. You can run each test as many times as you like without changing the user count. However, the AI model calls you make during evaluations are billed separately from this user-based charge.
    No. The user-based charge covers platform access only. Any AI model calls you make through the platform are billed separately and appear outside this user-based charge. The platform shows estimated cost per test and per run from provider price tables, so you can track that spend.
    2pisoftware.com+1
    Helpful?

    Vendor refund policy

    Non-refundable.

    How can we make this page better?

    Tell us how we can improve this page, or report an issue with this product.
    Tell us how we can improve this page, or report an issue with this product.

    Legal

    Vendor terms and conditions

    Upon subscribing to this product, you must acknowledge and agree to the terms and conditions outlined in the vendor's End User License Agreement (EULA) .

    Content disclaimer

    Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.

    Usage information

     Info

    Delivery details

    VERSDBenchCloudformation

    CloudFormation for the versdbench server and supporting infrastructure.

    Deploys a VPC, one EC2 host that runs versdbench, an RDS PostgreSQL database, an SES email identity for your domain, Secrets Manager secrets, a KMS key, and the Route53 record for your domain.

    CloudFormation Template (CFT)

    AWS CloudFormation templates are JSON or YAML-formatted text files that simplify provisioning and management on AWS. The templates describe the service or application architecture you want to deploy, and AWS CloudFormation uses those templates to provision and configure the required services (such as Amazon EC2 instances or Amazon RDS DB instances). The deployed application and associated resources are called a "stack."

    Version release notes

    1.0.0

    • Organisation and Group based user management
    • Admin console
    • Compare mode for side-by-side experiment execution copmparison
    • System wizard for ease-of-use
    • Bulk import/export

    Additional details

    Usage instructions

    Configure the offering via marketplace for your desired infrastructure options. Deploy and navigate to your configured application domain name by port forwarding to the admin console. The setup wizard will walk you through application set up.

    Support

    Vendor support

    VERSDBench provides technical, billing, and general customer support for organisations deploying and using the platform through AWS Marketplace.

    Email Support:  support@versd.com.au  Phone Support: 1800 961 919 Website:  https://versd.com.au 

    The VERSDBench support team is based in New South Wales, Australia (AEST). Support requests submitted via phone or email are monitored during standard business hours, Monday to Friday. The expected response time for all enquiries is within 24 business hours.

    VERSDBench provides:

    • Technical support and troubleshooting
    • Deployment and configuration assistance
    • Billing and account enquiries
    • General product enquiries

    All enquiries are triaged internally and directed to the appropriate VERSDBench support personnel for responsive assistance.

    Setup and deployment guidance is available via  https://versd.com.au  and through onboarding documentation provided upon purchase. Guidance includes enterprise integration support, authentication setup, and deployment best practices.

    VERSDBench is designed with enterprise-grade security and sovereign-safe deployment principles. The platform supports SSO-enabled authentication for secure enterprise-wide access management and group- and role-based access controls aligned with organisational policies.

    For AWS Marketplace (CloudFormation) deployments, security guidance covering encryption configuration, credential management, authentication setup, and credential rotation best practices is provided in the accompanying deployment documentation supplied during onboarding.

    AWS infrastructure support

    AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.

    Similar products

    Customer reviews

    Ratings and reviews

     Info
    0 ratings
    5 star
    4 star
    3 star
    2 star
    1 star
    0%
    0%
    0%
    0%
    0%
    0 reviews
    No customer reviews yet
    Be the first to review this product . We've partnered with PeerSpot to gather customer feedback. You can share your experience by writing or recording a review, or scheduling a call with a PeerSpot analyst.