Listing Thumbnail

    DeepEval Evaluation Workbench - Software Sushi

     Info
    Deployed on AWS
    Free Trial
    Evaluate assistant answers with DeepEval on Ubuntu 24.04, with optional model judging and saved JSON score reports. This is paid repackaged open-source software.

    Overview

    This is paid repackaged open-source software supplied by Software Sushi. DeepEval 4.2.4 core uses Apache License 2.0; dependencies retain their own licenses and original notices. The image supplies the unmodified upstream DeepEval CLI, a pinned Python environment and local evaluation examples on Ubuntu 24.04 x86_64. Software Sushi is not affiliated with or endorsed by Confident AI. Confident AI hosted services, dashboard subscriptions and commercial platform features are not included.

    The supplied example evaluates two fixed synthetic returns-policy answers. The correct answer states a 30-day window with a receipt; the incorrect answer states 90 days without one. DeepEval JSON correctness checks run locally without a model. An optional configured OpenAI-compatible model acts as a separate GEval judge of those answers. It does not generate the assistant answers. Reports retain original scores, thresholds, reasons and good/bad results, then a separate reader verifies the saved example. Model scores can vary and need human review.

    Start with t3.medium, 4 GiB RAM and at least 24 GiB root storage with EBS encryption enabled. No GPU is required for the examples. Connect through AWS Systems Manager Session Manager with your own appropriate instance role and outbound endpoint access, or use your own SSH key and restricted network setup. The product has no web UI, HTTP API listener or application inbound port. Access is controlled by the Ubuntu operating-system account.

    Run sudo -iu ubuntu bash /opt/sushi/docs/demo.sh. To add a judge, run sushi-deepeval configure as ubuntu and enter an entitled provider endpoint, exact model ID and provider key through the hidden prompt. Then run bash /opt/sushi/docs/demo.sh --judge. Remote endpoints require HTTPS and JSON-object chat responses. No provider key, model weights, provider credits or Confident AI account is supplied. Judge requests send the synthetic question, the answer under test and the reference answer to your chosen endpoint; review provider privacy and retention terms before using sensitive data.

    Each run creates a new JSON report under /home/ubuntu/deepeval-results. Settings under /home/ubuntu/.config/deepeval/judge.json use mode 0600. Reports and settings persist through reboot on the same root volume. Back them up before replacement or termination. The on-instance README explains configuration, native CLI use, independent readback and recovery. Original source distributions, wheels and notices are retained on the instance. Additional upstream workflows can require optional packages or separate service agreements.

    Software is USD 0.03 per running instance-hour on each offered type. EC2, EBS, data transfer and model-provider usage are billed separately. English getting-started guide: https://softwaresushi-marketplace-branding-public.s3.us-east-1.amazonaws.com/docs/deepeval/9a7b75dab712492d745e/getting-started.html 

    Highlights

    • Run DeepEval local JSON correctness checks against explicit good and bad synthetic answers.
    • Optionally use your entitled OpenAI-compatible model as a separate GEval factual-correctness judge.
    • Keep original JSON reports, scores and thresholds on the instance with independent saved-result readback.

    Details

    Delivery method

    Delivery option
    64-bit (x86) Amazon Machine Image (AMI)

    Latest version

    Operating system
    Ubuntu 24.04

    Deployed on AWS

    Features and programs

    Financing for AWS Marketplace purchases

    AWS Marketplace now accepts line of credit payments through the PNC Vendor Finance program. This program is available to select AWS customers in the US, excluding NV, NC, ND, TN, & VT.
    Financing for AWS Marketplace purchases

    Pricing

    Free trial

    Try this product free for 5 days according to the free trial terms set by the vendor. Usage-based pricing is in effect for usage beyond the free trial terms. Your free trial gets automatically converted to a paid subscription when the trial ends, but may be canceled any time before that.

    DeepEval Evaluation Workbench - Software Sushi

     Info
    Pricing is based on actual usage, with charges varying according to how much you consume. Subscriptions have no end date and may be canceled any time.
    Additional AWS infrastructure costs may apply. Use the AWS Pricing Calculator  to estimate your infrastructure costs.

    Usage costs (4)

     Info
    Dimension
    Cost/hour
    t3.medium
    Recommended
    $0.03
    t3.large
    $0.03
    t3.xlarge
    $0.03
    t3.2xlarge
    $0.03

    AI Insights

     Info

    Dimensions summary

    You pay by the hour based on the EC2 instance size you pick. The four options—t3.medium, t3.large, t3.xlarge, and t3.2xlarge—represent increasing amounts of compute and memory. Each step up gives you a larger instance at a higher hourly rate. Pricing scales with the instance you choose, not with how much you evaluate. You select the size that fits your workload, then run the software in your own AWS account. AWS infrastructure, data transfer, and any outside model-provider charges are billed separately from this hourly software rate.

    Top-of-mind questions for buyers

    Each option is an Amazon EC2 instance type with a set amount of vCPU and memory. t3.medium carries the least compute, scaling up through t3.large, t3.xlarge, and t3.2xlarge. Choose the size that matches your evaluation workload. You can run the software on any of the four.
    The hourly software rate meters running time. A fully stopped instance stops accruing software charges. But stopped instances may still incur AWS storage fees for attached volumes. Those AWS infrastructure charges are billed separately from this software rate.
    The software rate does not cover AWS infrastructure such as EC2, storage, networking, and data transfer, which AWS bills separately. If you enable optional model judging, your chosen model provider may charge for its usage under your own account and keys.
    softwaresushi.com
    Helpful?

    Vendor refund policy

    We offer a full refund of software charges for the first 48 hours billing starts if the product does not perform as described. To request a refund, email info@softwaresushi.com  with your EC2 instance ID and a description of the issue. AWS infrastructure costs (EC2, EBS, data transfer) are not eligible for refund by us.

    How can we make this page better?

    Tell us how we can improve this page, or report an issue with this product.
    Tell us how we can improve this page, or report an issue with this product.

    Legal

    Vendor terms and conditions

    Upon subscribing to this product, you must acknowledge and agree to the terms and conditions outlined in the vendor's End User License Agreement (EULA) .

    Content disclaimer

    Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.

    Usage information

     Info

    Delivery details

    64-bit (x86) Amazon Machine Image (AMI)

    Amazon Machine Image (AMI)

    An AMI is a virtual image that provides the information required to launch an instance. Amazon EC2 (Elastic Compute Cloud) instances are virtual servers on which you can run your applications and workloads, offering varying combinations of CPU, memory, storage, and networking resources. You can launch as many instances from as many different AMIs as you need.

    Version release notes

    Packaging candidate 0.1.1; source 18c21574619a. Automatic EC2 boot, real use and host reboot verified on this exact AMI. Known OS/kernel High/Critical vulnerability findings remain unresolved; no security fix or security qualification is claimed.

    Additional details

    Usage instructions

    Launch t3.medium or a larger supported instance with at least 4 GiB RAM and 24 GiB encrypted gp3 root storage. Require IMDSv2 and an instance profile with AmazonSSMManagedInstanceCore. No inbound rule or key pair is needed for Session Manager. Connect with Session Manager, then run sudo -iu ubuntu. Read /opt/sushi/docs/README.md and run bash /opt/sushi/docs/demo.sh. Keep application ports closed; use the loopback access and tunnel instructions in the installed guide. Back up your data before replacing the instance. EC2, EBS and data transfer charges apply separately.

    Resources

    Support

    Vendor support

    Software Sushi provides AMI packaging and setup guidance. Contact info@softwaresushi.com  for installation questions. Support coverage requires a separate agreement; upstream project and model-provider support are not included.

    AWS infrastructure support

    AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.

    Similar products

    Customer reviews

    Ratings and reviews

     Info
    0 ratings
    5 star
    4 star
    3 star
    2 star
    1 star
    0%
    0%
    0%
    0%
    0%
    0 reviews
    No customer reviews yet
    Be the first to review this product . We've partnered with PeerSpot to gather customer feedback. You can share your experience by writing or recording a review, or scheduling a call with a PeerSpot analyst.