GenAI Evaluation, Security and Observability on AWS by Assembly

GenAI evaluation, security and observability on AWS by Assembly

Assembly is an AWS Advanced Tier Services Partner. For generative AI in production or close to it, we measure quality, close the new AI risks and keep watch on cost and latency, so every change is a measured improvement, not a debate about vibes.

Relation to AWS services This professional service works on generative AI applications on AWS. Evaluation uses Amazon Bedrock Evaluations with golden datasets wired into CI; security uses Amazon Bedrock Guardrails, AWS Identity and Access Management (IAM), AWS PrivateLink and AWS CloudTrail; and observability uses Amazon CloudWatch and AWS X-Ray across the AI request path.

Three packages, one AWS team

• GenAI evaluation (typically 4-6 weeks): Baseline, Harness, Optimise, Handover. Measure accuracy, groundedness, safety and latency on real traffic, build golden datasets and automated scoring into your pipeline, then redesign context and model routing, proving each change against the harness.
• AI security and compliance (typically 3-5 weeks): Assess, Remediate, Evidence. Threat model prompt injection, data exfiltration, tool abuse and model supply chain risks, implement guardrails, data protection, least-privilege tool access and human review where it matters, and map controls to SOC 2, ISO 27001 and your customers' questionnaires.
• AI observability (typically 3-4 weeks): Instrument, Measure, Alert, Improve. Trace prompts, retrieval, tools and model calls, set quality, cost and latency thresholds, connect alerting to your on-call tooling and feed production signals back into evaluation.

Why it matters

• Without evaluation you can't tell an improvement from a regression.
• What you send, in what order, to which model is usually the biggest lever on cost per request.
• AI security questionnaires now gate deals; a clear control story turns a blocker into a differentiator.

What you receive

• An automated evaluation harness in CI, a quality baseline and an ongoing scorecard
• A context and routing redesign, with cost per request measured before and after
• An AI threat model and risk register, guardrails and audit-ready control mapping
• Reusable answers for security questionnaires
• A traced AI request path, quality, cost and latency dashboards, and drift and regression alerting

Proof Human eSources turned nearly three decades of talent science into AI that explains every job match: Talent Scout AI, built by Assembly and live on Amazon Bedrock behind TruTalent, its recruiting platform.

Important

• We build the controls and evidence; the audit itself, and any SOC 2 report or ISO 27001 certificate, comes from your auditor.
• Start with a GenAI/ML Readiness Assessment, funded by AWS for companies that qualify, subject to AWS approval. We check your fit with you before anything goes to AWS.
• You are responsible for any AWS infrastructure and service charges incurred in your AWS account, billed separately from your AWS Marketplace transaction.