Sold by: MphasisÂ
Deployed on AWS
Estimating GenAI(LLM) Model Reliability Through Evaluation of Factual Consistency.
Overview
In today's era of GenAI reliance, the lack of an evaluation metric for factual consistency poses a challenge due to hallucination. Our solution addresses this by offering a unique method to verify the accuracy of LLM model-generated responses, leveraging Claude from Anthropic as the foundation model. We employ a multi-step process, including question generation, validation, and comparison against reference answers, to ensure factual consistency. Our approach is versatile; for instance, in customer service, it checks if representatives (AI chatbot/human) provide correct information from a knowledge base. It extends beyond AI responses, working in scenarios where accuracy is vital, such as educational platforms or content moderation systems. This broad applicability underscores our solution's effectiveness in maintaining factual consistency across various domains. The offering requires an AWS bedrock anthropic Claude Instant model subscription.
Highlights
- Our GenAI solution tackles the challenge of assessing factual consistency in LLM model-generated responses. It involves extracting noun chunks, generating fact-based questions, and comparing LLM responses against reference answers. This comprehensive approach ensures accurate evaluation outcomes, empowering users to make informed decisions regarding GenAI model suitability and performance.
- Our proposed LLM modal evaluation metric improves model assessment by providing a robust means of evaluating factual consistency. By filtering responses based on this metric, users can identify high-quality outputs, enhancing overall model efficacy and utility across diverse applications.
- Mphasis DeepInsights is a cloud-based cognitive computing platform that offers data extraction & predictive analytics capabilities. Need Customized Deep learning and Machine Learning Solutions? Get in Touch!
Details
Sold by
Categories
Delivery method
Latest version
Deployed on AWS
Unlock automation with AI agent solutions
Fast-track AI initiatives with agents, tools, and solutions from AWS Partners.
Features and programs
Financing for AWS Marketplace purchases
AWS Marketplace now accepts line of credit payments through the PNC Vendor Finance program. This program is available to select AWS customers in the US, excluding NV, NC, ND, TN, & VT.
Pricing
Pricing is based on actual usage, with charges varying according to how much you consume. Subscriptions have no end date and may be canceled any time.
Additional AWS infrastructure costs may apply. Use the AWS Pricing Calculator to estimate your infrastructure costs.
Dimension | Description | Cost |
|---|---|---|
ml.m5.large Inference (Batch) Recommended | Model inference on the ml.m5.large instance type, batch mode | $2.00/host/hour |
ml.m5.4xlarge Inference (Batch) | Model inference on the ml.m5.4xlarge instance type, batch mode | $2.00/host/hour |
ml.m5.2xlarge Inference (Batch) | Model inference on the ml.m5.2xlarge instance type, batch mode | $2.00/host/hour |
ml.m5.xlarge Inference (Batch) | Model inference on the ml.m5.xlarge instance type, batch mode | $2.00/host/hour |
inference.count.m.i.c Inference Pricing | inference.count.m.i.c Inference Pricing | $0.50/request |
Vendor refund policy
Currently, we do not support refunds, but you can cancel your subscription to the service at any time.
Legal
Vendor terms and conditions
Upon subscribing to this product, you must acknowledge and agree to the terms and conditions outlined in the vendor's End User License Agreement (EULA)Â .
Content disclaimer
Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.
Delivery details
Amazon SageMaker model
An Amazon SageMaker model package is a pre-trained machine learning model ready to use without additional training. Use the model package to create a model on Amazon SageMaker for real-time inference or batch processing. Amazon SageMaker is a fully managed platform for building, training, and deploying machine learning models at scale.
Deploy the model on Amazon SageMaker AI using the following options:
Version release notes
first version
Additional details
Inputs
- Summary
Usage Methodology for the algorithm:
- The input must be 'Input.zip' file.
- The zip file should contain three files 'credentials' and 'input' in .json format.
Input is a zip file(input_data.zip) containing a two json files, 1.credintials.json { "aws_access_key_id": "", "aws_secret_access_key": "", "region_name": "" } 2.input.json [{"query":"","context":"","response":""}]
- Input MIME type
- application/zip
https://github.com/Mphasis-ML-Marketplace/Factual-consistency-score/blob/master/input_data.zip
https://github.com/Mphasis-ML-Marketplace/Factual-consistency-score/blob/master/input_data.zip
Resources
Vendor resources
Support
Vendor support
For any assistance reach out to us at:
AWS infrastructure support
AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.
Similar products
The ExtraHop Cloud Platform delivers real-time visibility into all cloud traffic, backed by advanced machine learning which provides a simplified workflow for incident investigation and immediate response within small VPC deployments.
The ExtraHop Cloud Platform delivers real-time visibility into all cloud traffic, backed by advanced machine learning which provides a simplified workflow for incident investigation and immediate response within small VPC deployments.
Comprehensive human-reviewed assessment of large language model (LLM) outputs that rates key quality dimensions, including helpfulness, safety, and factual accuracy. Our rubric-driven evaluation identifies content risks, flags hallucinations, and delivers detailed quality scores and remediation insights. Perfect for model fine-tuning, RLHF workflows, and ensuring trustworthy AI deployments
The ExtraHop Cloud Platform delivers real-time visibility into all cloud traffic, backed by advanced machine learning which provides a simplified workflow for incident investigation and immediate response within small VPC deployments.
Customer reviews
No customer reviews yet
Be the first to review this product . We've partnered with PeerSpot to gather customer feedback. You can share your experience by writing or recording a review, or scheduling a call with a PeerSpot analyst.