Monitoring LLM Uncertainty in Financial Services on AWS

Quantifying predictive uncertainty of large language models (LLMs) in production is a critical challenge for financial services: ground-truth labels are often delayed, expensive, or unavailable, making it impractical to monitor accuracy for tasks like compliance assessment, credit risk rating, or document classification. These classification tasks are increasingly handled by LLMs in production, where a single misclassification can have significant business or regulatory impact.

Conformal prediction addresses this gap. Instead of outputting a single class, the model returns a prediction set, a set of plausible classes with a coverage guarantee. In this context, ‘guarantee’ refers to the mathematical coverage property of conformal prediction, not a product warranty or service commitment. When the model is confident, the set contains one class. When uncertain, the set expands. The average prediction set size becomes a label-free monitoring metric tracked in real time, so you can detect rising uncertainty automatically and route uncertain predictions to human review.

The conformal prediction framework underpinning this approach provides distribution-free coverage guarantees for any black-box model, with foundational theory established [Vovk et al., 2005] and [Papadopoulos et al., 2002]. Recent work by [Ye et al., 2024] validates these methods on LLMs, demonstrating that prediction-set size reliably captures model uncertainty across diverse natural-language classification tasks, without requiring ground-truth labels.

This post demonstrates on Massive Multitask Language Understanding (MMLU) [Hendrycks et al., 2021], an open-source multiple-choice benchmark accessible via Hugging Face Hub, selecting four business-relevant subjects. AWS provides the platform for operationalizing conformal prediction at enterprise scale. It lets you detect unreliable predictions before they reach end users, reduce misclassification costs through automated human-in-the-loop routing, and support regulatory requirements for model monitoring and human oversight. Amazon SageMaker AI delivers managed real-time endpoints from which token-level probabilities and nonconformity scores can be computed with minimal infrastructure overhead. Native integration with Amazon CloudWatch enables continuous tracking of prediction-set size as a custom metric, with automated alarms when uncertainty exceeds calibrated thresholds. The underlying platform provides the elasticity, scalability, multi-AZ robustness, and security posture that financial services regulators require.

Prerequisites

This post walks through the solution’s design and the reasoning behind it; full deployment steps live in the repository README. To follow along or deploy it yourself, you need an AWS account with Amazon SageMaker Studio access. The Studio execution role needs broad permissions covering AWS CloudFormation, Amazon CloudWatch, Amazon DynamoDB, Amazon EventBridge, AWS Key Management Service (AWS KMS), AWS Lambda, Amazon Simple Notification Service (Amazon SNS), Amazon Simple Queue Service (Amazon SQS), and Amazon Simple Storage Service (Amazon S3). The Lambda execution roles, along with the DynamoDB table, S3 buckets, Lambda functions, SQS queue, SNS topic, and EventBridge schedule, are created automatically by the AWS CloudFormation stack.

Why predictive uncertainty quantification matters for LLMs in financial services

Two factors make predictive uncertainty quantification essential for production LLMs: the practical challenge of obtaining ground-truth labels and the growing regulatory expectation that AI systems demonstrate reliability and support human oversight.

The label-delay problem

In financial services deployments, ground-truth labels arrive weeks or months after prediction. Compliance reviews take time, credit outcomes unfold over loan lifecycles, and regulatory filing classification requires expert decision. Without labels, you can’t compute standard accuracy metrics in real time. Therefore, you need label-free monitoring signals that detect degradation before it impacts your business outcomes.

Regulatory drivers

The need for predictive uncertainty quantification is becoming a regulatory requirement. SR 11-7 (Federal Reserve) calls for ongoing monitoring of model performance, including the ability to detect when models operate outside their validated range. EU AI Act requires high-risk AI systems to provide transparency about their limitations, including measures of accuracy and robustness. Model monitoring and human oversight mechanisms must be in place. UK SS1/23 (PRA Supervisory Statement on Model Risk Management) requires firms to have effective model risk management frameworks, including identifying model limitations and monitoring performance with human oversight mechanisms.

Organizations that proactively implement uncertainty monitoring gain a competitive advantage by deploying AI systems faster with evidence to support compliance efforts.

The regulatory references in this post are provided for informational purposes only and don’t constitute legal advice. Consult your legal and compliance teams to determine specific requirements for your use case and jurisdiction.

Conformal prediction for classification explained

Conformal prediction provides a rigorous framework for quantifying predictive uncertainty.

Core idea

Conformal prediction transforms a classifier’s softmax (a function that converts raw scores into probabilities that sum to 1) outputs into prediction sets with guaranteed coverage: given a user-specified error rate α, such as 0.1, conformal prediction guarantees that the true label is contained in the prediction set at least (1 – α) = 90% of the time.

This statistical assurance requires only that calibration and inference data come from the same distribution. That means that the types of inputs seen during calibration are representative of those encountered in production. This is a weaker assumption than most methods, making it practical for LLM monitoring where strong distributional assumptions are hard to verify. When input data drifts, that assumption breaks and standard conformal prediction is no longer valid. Detecting this is the first step: a shift-detection method on the inputs (e.g., a distance metric), complemented by the average prediction-set size the solution tracks in CloudWatch, which rises as an output-side symptom of drift. Once drift is confirmed, you recalibrate, refitting on a fresh labeled holdout when new labels are available, or, when the new data is unlabeled, with weighted conformal methods that restore coverage by reweighting calibration scores (e.g., for covariate shift: [Tibshirani et al. 2019]). Monitoring then resumes.

Important: this coverage guarantee is marginal, it holds on average across the full input distribution, not conditionally per class or per subpopulation. In financial services settings where per-group reliability matters (for example, ensuring that high risk credit applicants receive the same coverage as low-risk ones), Mondrian conformal prediction [Vovk et al., 2003] provides group-conditional coverage guarantees.

The LAC method

This solution uses the Least Ambiguous set-valued Classifiers (LAC) method [Sadinle et al., 2019], which efficiently produces prediction sets in two steps while meeting coverage properties:

1. Calibration (offline)

• Run the LLM on a held-out calibration set with known labels.
• For each sample, compute the nonconformity score as one minus the probability of the true label.
• Compute the threshold q̂ as the quantile of calibration scores at the target coverage level, with a small upward correction that accounts for the calibration set size.

2. Inference (online, per-prediction)

• For each new input, obtain the LLM’s probability distribution over K classes p[1],…, p[K].
• Include class j in the prediction set if 1 – p[j] ≤ q̂.
• The resulting set size indicates model confidence: certainty (size of 1) or uncertainty (size > 1, route to human review).

Architecture on AWS

At a high level (see Figure 1), the system operates in three phases, each implemented in detail in the demonstration that follows. Offline calibration scores the calibration set against the live SageMaker endpoint, computes LAC scores, derives q̂ and writes it to Amazon DynamoDB. PyDeequ, AWS’s Spark-based data-quality library, validates the calibration data before scoring begins. Online prediction reads q̂ from Amazon DynamoDB and builds the prediction set for each request. It then publishes the prediction set size metric to Amazon CloudWatch and routes uncertain predictions to Amazon SQS for human review. Periodic labeled replay employs Amazon EventBridge to trigger periodic batch evaluation against labeled holdouts, publishing coverage, certain accuracy (accuracy of certain predictions) and uncertain accuracy to Amazon CloudWatch. It also writes batch JSONs to Amazon S3 for uncertainty explanation.

Figure 1: Reference architecture diagram

Complete demonstration

This walkthrough covers the full pipeline end-to-end: deploying the infrastructure and model, calibrating the conformal threshold, running online inference with human-review routing, and monitoring uncertainty through batch evaluation.

Dataset and model

The evaluation uses MMLU, selecting four subjects relevant to business and social sciences: econometrics, marketing, management, sociology. The dataset provides 652 multiple-choice questions with 4 answer choices each. The calibration set contains 306 samples (50% random split), the test set has 306 samples, and the difficult set includes 40 samples (10 per subject, based on lowest model confidence).

The model is Meta Llama 3 8B from Amazon SageMaker AI JumpStart on an ml.g5.2xlarge instance.

Step 1: Deploy and calibrate

The deployment packages Lambda code to S3, provisions the infrastructure with AWS CloudFormation (IAM roles, DynamoDB table, S3 buckets, Lambda functions, SQS queue, SNS topic, and EventBridge schedule), deploys the model to a SageMaker endpoint, prepares the MMLU data splits, validates data quality with PyDeequ, and calibrates the conformal threshold. All scripts derive resource names (S3 buckets, DynamoDB table) from the AWS account ID and region automatically.

For the full deployment commands (environment setup, Lambda packaging, CloudFormation deploy, model deployment, data preparation, and calibration), see the Deployment steps in the repository README.

Result: q̂ = 0.85 (target coverage = 90%), calibrated on 306 samples.

The calibration script extracts probabilities via prefill-only forward passes, appending each answer token and reading its log-probability, then applying softmax:

def get_answer_probs(predictor, prompt, answer_tokens=("0", "1", "2", "3")):
    logprobs = []
    for token in answer_tokens:
        payload = {"inputs": prompt + " " + token,
                   "parameters": {"max_new_tokens": 1, "details":True,
                                  "decoder_input_details": True}}
        response = predictor.predict(payload)
        logprobs.append(response["details"]["prefill"][-1]["logprob"])
    return softmax(logprobs)  # normalized probability distribution

Step 2: Online inference with routing

Once calibrated, the inference handler builds a prediction set for each incoming question. The core logic is straightforward: it includes every class whose probability exceeds the threshold:

# Build prediction set: include class j if p[j] >= 1 - q_hat
prediction_set = [j for j in range(K) if probs[j] >= 1 - q_hat]
set_size = len(prediction_set)
routing_decision = "auto" if set_size == 1 else "human_review"

Certain predictions (set size = 1) are returned directly and uncertain ones (set size > 1) are routed to an SQS queue for human review, and a SetSize metric is published to Amazon CloudWatch on every request.

Step 3: Batch evaluation with uncertainty monitoring

The solution processes the test set in three batches with 5-minute gaps between each, injecting the difficult questions into the final batch to simulate a challenging scenario.

Step 4: Cleanup

To stop incurring charges, delete the SageMaker endpoint and run aws cloudformation delete-stack. See the repository README for full cleanup instructions including version-aware bucket emptying.

Results

Table 1: Batch evaluation results

The results from Table 1 show that coverage holds at or above 90% across all batches, the conformal guarantee is maintained. Meanwhile, the average set size spikes from 1.6 to 2.3 when difficult questions hit, providing a label-free signal of rising uncertainty visible in Amazon CloudWatch without any ground-truth labels. Crucially, the accuracy gap between certain and uncertain predictions widens from 35% to 50% under stress, confirming that the conformal routing decision becomes more valuable on hard questions.

Explaining predictive uncertainty

Table 2 compares the vocabulary of questions where the model is certain (set size = 1) versus uncertain (set size > 1) using differential frequency ratios. Words with ratio close to 100% appear almost exclusively in uncertain predictions.

Table 2: Differential term rankings for certain and uncertain predictions

The uncertain vocabulary clusters in econometrics/statistics terminology (coefficient, variance, GARCH, OLS). The certain vocabulary is in marketing/management (customers, brand, service). This reveals the model systematically struggles with quantitative reasoning.

You can act on these insights immediately by routing uncertain predictions to human review or specialized models, fine-tuning on underperforming domains to reduce uncertainty, or adjusting prompts to provide additional context for difficult questions.

Financial Services applications

While this demonstration uses MMLU as a reproducible benchmark, the same architecture applies directly to regulated financial services use cases (shown in Table 3).

Table 3: Financial Services applications for LLM uncertainty monitoring.

Adapt to your use case

The following steps show how to adapt this pipeline to your own classification task:

Table 4: Steps to adapt the pipeline to your own classification task

Average latency is roughly 250 ms per question (K=4 prefill calls) on ml.g5.2xlarge. See the repository README and Amazon SageMaker AI pricing for details.

Conclusion

This post showed how to operationalize conformal prediction for LLM predictive uncertainty monitoring on AWS, from calibration through real-time routing and batch evaluation, with a ready-to-deploy reference architecture. This solution includes label-free monitoring via prediction-set size, rigorous human routing, differential word-frequency explanation, and support for regulatory frameworks (SR 11-7, EU AI Act). As LLMs move to production in financial services, conformal prediction provides the statistical rigor; AWS provides the infrastructure to operationalize it at scale.

The complete code is available on GitHub. Clone the repository, deploy via CloudFormation, and follow the demo notebook (examples/demo.ipynb) in SageMaker Studio for the full pipeline end-to-end. For more on the underlying services, see Amazon SageMaker AI JumpStart and Amazon CloudWatch custom metrics.

References

1. Vovk, V., Gammerman, A., & Shafer, G. (2005). Algorithmic learning in a random world. Boston, MA: Springer US.
2. Papadopoulos, H., Proedrou, K., Vovk, V., & Gammerman, A. (2002, August). Inductive confidence machines for regression. In European conference on machine learning (pp. 345-356). Berlin, Heidelberg: Springer Berlin Heidelberg.
3. Sadinle, M., Lei, J., & Wasserman, L. (2019). Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 114(525), 223-234.
4. Ye, F., Yang, M., Pang, J., Wang, L., Wong, D. F., Yilmaz, E., Shi, S., & Tu, Z. (2024). Benchmarking LLMs via uncertainty quantification. Advances in Neural Information Processing Systems, 37, 15356-15385.
5. Vovk, V., Lindsay, D., Nouretdinov, I., & Gammerman, A. (2003). Mondrian confidence machine. Technical Report.
6. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2021). Measuring Massive Multitask Language Understanding. International Conference on Learning Representations 2021.
7. Tibshirani, R. J., Foygel Barber, R., Candes, E., & Ramdas, A. (2019). Conformal prediction under covariate shift. Advances in neural information processing systems, 32.