Amazon Sagemaker
Amazon SageMaker is a fully-managed platform that enables developers and data scientists to quickly and easily build, train, and deploy machine learning models at any scale. With Amazon SageMaker, all the barriers and complexity that typically slow down developers who want to use machine learning are removed. The service includes models that can be used together or independently to build, train, and deploy your machine learning models.
Factual Consistency Metric - LLMOps
By:
Latest Version:
v1
Estimating GenAI(LLM) Model Reliability Through Evaluation of Factual Consistency.
Product Overview
In today's era of GenAI reliance, the lack of an evaluation metric for factual consistency poses a challenge due to hallucination. Our solution addresses this by offering a unique method to verify the accuracy of LLM model-generated responses, leveraging Claude from Anthropic as the foundation model. We employ a multi-step process, including question generation, validation, and comparison against reference answers, to ensure factual consistency. Our approach is versatile; for instance, in customer service, it checks if representatives (AI chatbot/human) provide correct information from a knowledge base. It extends beyond AI responses, working in scenarios where accuracy is vital, such as educational platforms or content moderation systems. This broad applicability underscores our solution's effectiveness in maintaining factual consistency across various domains. The offering requires an AWS bedrock anthropic Claude Instant model subscription.
Key Data
Version
By
Type
Model Package
Highlights
Our GenAI solution tackles the challenge of assessing factual consistency in LLM model-generated responses. It involves extracting noun chunks, generating fact-based questions, and comparing LLM responses against reference answers. This comprehensive approach ensures accurate evaluation outcomes, empowering users to make informed decisions regarding GenAI model suitability and performance.
Our proposed LLM modal evaluation metric improves model assessment by providing a robust means of evaluating factual consistency. By filtering responses based on this metric, users can identify high-quality outputs, enhancing overall model efficacy and utility across diverse applications.
Mphasis DeepInsights is a cloud-based cognitive computing platform that offers data extraction & predictive analytics capabilities. Need Customized Deep learning and Machine Learning Solutions? Get in Touch!
Not quite sure what you’re looking for? AWS Marketplace can help you find the right solution for your use case. Contact us
Pricing Information
Use this tool to estimate the software and infrastructure costs based your configuration choices. Your usage and costs might be different from this estimate. They will be reflected on your monthly AWS billing reports.
Contact us to request contract pricing for this product.
Estimating your costs
Choose your region and launch option to see the pricing details. Then, modify the estimated price by choosing different instance types.
Version
Region
Software Pricing
Model Realtime Inference$0.50/inference
running on any instance
Model Batch Transform$2.00/hr
running on ml.m5.large
Infrastructure PricingWith Amazon SageMaker, you pay only for what you use. Training and inference is billed by the second, with no minimum fees and no upfront commitments. Pricing within Amazon SageMaker is broken down by on-demand ML instances, ML storage, and fees for data processing in notebooks and inference instances.
Learn more about SageMaker pricing
With Amazon SageMaker, you pay only for what you use. Training and inference is billed by the second, with no minimum fees and no upfront commitments. Pricing within Amazon SageMaker is broken down by on-demand ML instances, ML storage, and fees for data processing in notebooks and inference instances.
Learn more about SageMaker pricing
SageMaker Realtime Inference$0.056/host/hr
running on ml.t2.medium
SageMaker Batch Transform$0.115/host/hr
running on ml.m5.large
Model Realtime Inference
For model deployment as Real-time endpoint in Amazon SageMaker, the software is priced based on the number of inferences generated by the ML Model per month. Typically, the number of inferences is the same as the number of successful calls to the real-time endpoint. For models that support multiple inputs in a request, sellers have the option to meter the number of inputs processed in a request to count generated inferences.
Additional infrastructure cost, taxes or fees may apply.
Usage Information
Model input and output details
Input
Summary
Usage Methodology for the algorithm: 1) The input must be 'Input.zip' file. 2) The zip file should contain three files 'credentials' and 'input' in .json format.
Input is a zip file(input_data.zip) containing a two json files, 1.credintials.json { "aws_access_key_id": "", "aws_secret_access_key": "", "region_name": "" } 2.input.json [{"query":"","context":"","response":""}]
Input MIME type
application/zipSample input data
Output
Summary
The output will be a zip file containing a "otuput.json" file containing an added column with the evaluation score for the provided input.
Output MIME type
application/zipSample output data
Sample notebook
Additional Resources
End User License Agreement
By subscribing to this product you agree to terms and conditions outlined in the product End user License Agreement (EULA)
Support Information
Factual Consistency Metric - LLMOps
For any assistance reach out to us at:
AWS Infrastructure
AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.
Learn MoreRefund Policy
Currently, we do not support refunds, but you can cancel your subscription to the service at any time.
Customer Reviews
There are currently no reviews for this product.
View allWrite a review
Share your thoughts about this product.
Write a customer review