Sold by: MphasisÂ
Deployed on AWS
Evaluate privacy awareness with RBAC for an LLM based chatbot such that responses to information seeking queries respect access controls.
Overview
The solution helps evaluate privacy awareness driven by RBAC of LLM based conversational AI systems. The system generates questions tailored to different user roles to test the chat system's performance for privacy awareness for these user roles. It identifies if the LLM is aware or under-aware of the access controls for each role. Under-awareness indicates that the chat system can provide such information to a user even though the role doesn't permit access to that data point. The solution takes as input: the metadata of the backend database, the description of roles and row/column level access controls for these roles. Anthropic Claude v2 is leveraged to process this information and a questionnaire with synthetic database is provided in the first step. The system to be evaluated is configured with the synthetic DB to run against the queries under different scenarios mentioned in the questionnaire. The results are fed back to the inference endpoint to get the final evaluation.
Highlights
- The solution is intended to evaluate privacy awareness of fine-tuned LLMs/any LLM chat workflow, generating role-specific questions and assessing performance for domain-specific tasks.
- The solution uses application information, table schema, and role details to generate privacy-related questions, aiding in evaluating the LLM's performance for each role separately.
- Mphasis DeepInsights is a cloud-based cognitive computing platform that offers data extraction & predictive analytics capabilities. Need Customized Deep learning and Machine Learning Solutions? Get in Touch!
Details
Sold by
Delivery method
Latest version
Deployed on AWS
Unlock automation with AI agent solutions
Fast-track AI initiatives with agents, tools, and solutions from AWS Partners.
Features and programs
Financing for AWS Marketplace purchases
AWS Marketplace now accepts line of credit payments through the PNC Vendor Finance program. This program is available to select AWS customers in the US, excluding NV, NC, ND, TN, & VT.
Pricing
Pricing is based on actual usage, with charges varying according to how much you consume. Subscriptions have no end date and may be canceled any time.
Additional AWS infrastructure costs may apply. Use the AWS Pricing Calculator to estimate your infrastructure costs.
Dimension | Description | Cost/host/hour |
|---|---|---|
ml.m5.xlarge Inference (Batch) Recommended | Model inference on the ml.m5.xlarge instance type, batch mode | $0.00 |
ml.m5.xlarge Inference (Real-Time) Recommended | Model inference on the ml.m5.xlarge instance type, real-time mode | $0.00 |
ml.m5.xlarge Training Recommended | Algorithm training on the ml.m5.xlarge instance type | $10.00 |
ml.m4.4xlarge Inference (Batch) | Model inference on the ml.m4.4xlarge instance type, batch mode | $0.00 |
ml.m5.4xlarge Inference (Batch) | Model inference on the ml.m5.4xlarge instance type, batch mode | $0.00 |
ml.m4.16xlarge Inference (Batch) | Model inference on the ml.m4.16xlarge instance type, batch mode | $0.00 |
ml.m5.2xlarge Inference (Batch) | Model inference on the ml.m5.2xlarge instance type, batch mode | $0.00 |
ml.p3.16xlarge Inference (Batch) | Model inference on the ml.p3.16xlarge instance type, batch mode | $0.00 |
ml.m4.2xlarge Inference (Batch) | Model inference on the ml.m4.2xlarge instance type, batch mode | $0.00 |
ml.c5.2xlarge Inference (Batch) | Model inference on the ml.c5.2xlarge instance type, batch mode | $0.00 |
Vendor refund policy
Currently we do not support refunds, but you can cancel your subscription to the service at any time.
Legal
Vendor terms and conditions
Upon subscribing to this product, you must acknowledge and agree to the terms and conditions outlined in the vendor's End User License Agreement (EULA)Â .
Content disclaimer
Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.
Delivery details
Amazon SageMaker algorithm
An Amazon SageMaker algorithm is a machine learning model that requires your training data to make predictions. Use the included training algorithm to generate your unique model artifact. Then deploy the model on Amazon SageMaker for real-time inference or batch processing. Amazon SageMaker is a fully managed platform for building, training, and deploying machine learning models at scale.
Deploy the model on Amazon SageMaker AI using the following options:
Version release notes
This is the first version.
Additional details
Inputs
- Summary
The training pipeline gives generated data and questions (questions.csv). Please give your chatbot these generated data (generated_data.xlsx) as context and give answers for the given questions. For inference/assessment provide the csv file similar to questions.csv with predicted_access_level column added to this. This predicted_access_level column can have two values within or outside. Within signifies that according to chatbot the role has access to data asked in the question and vice versa.
- Input MIME type
- application/zip, application/gzip, text/csv
https://github.com/Mphasis-ML-Marketplace/Privacy-Aware-LLM-Evaluation-with-RBAC/blob/main/Inference_input/test_data.csv
https://github.com/Mphasis-ML-Marketplace/Privacy-Aware-LLM-Evaluation-with-RBAC/blob/main/Inference_input/test_data.csv
Input data descriptions
The following table describes supported input data fields for real-time inference and batch transform.
Field name | Description | Constraints | Required |
|---|---|---|---|
test_data.csv | csv must contain following columns: RoleID, RoleName, Question, AccessLevel, predicted_access_level | Type: FreeText | Yes |
Resources
Vendor resources
Support
Vendor support
For any product support you can reach out to us at:
AWS infrastructure support
AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.
Similar products
At Privacient AI, we provide an engaging learning platform designed to empower your employees with the knowledge and skills to enable data privacy and embrace responsible AI practices. Our interactive training experience ensures that your team stays informed, compliant, and proactive in navigating the complexities of privacy and AI, fostering a culture of privacy and ethical responsibility throughout your organization. Whether, you want to train your workforce with engaging comic books or build privacy champions with our rigorous learning paths, we have it all.
Meet the privacy and data security requirements to grow business. Kaamel’s security and privacy veterans can assist you with 10+ years of hands-on experience.
Secureframe is the leading, all-in-one platform for security, privacy and compliance. Through our world-class governance, risk and compliance (GRC) solutions, Secureframe makes it fast, easy, and cost effective for organizations of all sizes to achieve and maintain security, privacy and compliance.
A-LIGN offers a powerful solutions to safeguard your customers' and partners' information Combining our expert privacy staff with our advanced compliance management technology, we help you achieve GDPR and Privacy compliance in a fraction of the time required by other privacy professionals. With A-LIGN, you can assure your customers and partners that their information is private.
Batch and event-based processing for sensitive data on your Kubernetes cluster with the STRM Data Plane.
Customer reviews
No customer reviews yet
Be the first to review this product . We've partnered with PeerSpot to gather customer feedback. You can share your experience by writing or recording a review, or scheduling a call with a PeerSpot analyst.