Hyundai AutoEver: Building a multi-tenant generative AI sandbox and production AIOps on Amazon Bedrock

At Hyundai AutoEver, the mobility software arm of Hyundai Motor Group, generative AI ideas have always been abundant. What we wanted was an equally fast path from ideas to a secure, validated prototype. Rather than having every team repeat infrastructure provisioning, security configuration, and budget approval, we built the GenAI Sandbox. This pre-configured, secure, cost-governed environment on AWS lets any team start building and validating generative AI applications on day one.

This post is a technical deep dive. It explains the Sandbox’s multi-tenant isolation model along with its inherited security and cost controls. It then examines two production-grade multi-agent AIOps systems our teams built on top of it, including the LangGraph (an open source multi-agent orchestration framework) state model, Retrieval-Augmented Generation (RAG) design, OpenSearch query patterns, parallel root cause analysis (RCA) with self-falsification, and the human-in-the-loop safeguards that help make agentic recovery safe in production. Code samples are illustrative and simplified for readability.

This post shows how to design a secure, multi-tenant generative AI architecture and build trustworthy multi-agent AIOps systems on it with Amazon Bedrock, LangGraph, and Amazon OpenSearch Service.

Solution overview

The GenAI Sandbox is a managed offering, not a per-project environment. It pairs the foundation models in Amazon Bedrock with compute, storage, databases, and monitoring, and gives every onboarded team an identical secure baseline with a defined path to production. The remainder of this post is in two halves:

• The architecture (security, cost, multi-tenant permissions, networking, and the baseline package)
• Two deep dives into the multi-agent AIOps systems teams shipped on it.

The GenAI Sandbox is built around five core pillars: security, cost, permissions, networking, and development packages, as shown in the following figure.

Figure 1: Sandbox account architecture showing network, security, cost, and permission framework

Figure 2: High-level architecture of the GenAI Sandbox showing multi-tenant isolation, cost controls, security framework, and environment management.

Offering architecture: The GenAI Sandbox

The GenAI Sandbox architecture is designed to give multiple users isolated, secure environments for experimentation, all within a single AWS account. The following sections describe how this is achieved across tenant isolation, cost governance, security, and monitoring.

Multi-tenant isolation with a single account and ABAC

Hosting multiple teams safely often means multiple AWS accounts and heavy management overhead. The GenAI Sandbox uses a single-account, multi-tenant model. AWS IAM Identity Center permission sets combine with attribute-based access control (ABAC) so that access decisions are driven by tags rather than per-resource policies: a principal tagged with a project attribute can act only on resources carrying the matching tag. The following pattern illustrates the tag-on-tag condition at the heart of the isolation model.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:PutObject"],
    "Resource": "arn:aws:s3:::sandbox-*/*",
    "Condition": {
      "StringEquals": {
        "aws:ResourceTag/project": "${aws:PrincipalTag/project}"
      }
    }
  }]
}

Because the condition compares the principal’s project tag to the resource’s project tag, one policy scales to every tenant without writing per-project policies; adding a team is a tagging operation, not a policy-authoring one. This is the key trade-off behind the single-account choice: we accept tag-governance discipline (enforced through permission sets and AWS IAM Access Analyzer) in exchange for avoiding account sprawl and cross-account management overhead.

Inherited security controls

A security framework tuned for generative AI workloads is provisioned before any project exists, so onboarding inherits it automatically. Detection and audit run on AWS Config (configuration drift and compliance rules), Amazon Macie (sensitive-data discovery in Amazon Simple Storage Service (Amazon S3)), Amazon GuardDuty (threat detection), Amazon Inspector (workload vulnerability scanning), and IAM Access Analyzer (external-access findings), with all findings aggregated and triaged centrally in AWS Security Hub. Builders never assemble these controls themselves; the controls travel with the environment, which is what keeps a fast-moving sandbox continuously compliant with corporate standards.

Cost governance with automated guardrails

Per-project cost is aggregated and monitored continuously through cost allocation tags that mirror the ABAC project attribute. Idle resources are detected automatically, and when a project crosses a spend threshold, automated controls (up to and including removing permissions) are applied so experimentation never becomes runaway cost. Administrators get a real-time, per-project cost view; developers are insulated from day-to-day budget mechanics.

Network integration and the baseline development package

The Sandbox connects to the corporate internet data center (IDC) network, so prototypes interface with real internal systems and are validated against production-like data rather than in isolation. On onboarding, a developer receives a baseline serverless package already wired together: scoped AWS Identity and Access Management (IAM) permissions, Amazon API Gateway, AWS Lambda, Amazon Bedrock Knowledge Bases, Amazon OpenSearch Serverless, and Amazon S3. With no infrastructure to assemble, teams reach working prototypes in days, which is how 14 hackathon teams shipped in 4 weeks.

“We automated a manual process that used to take two to three hours, cutting processing time to 10 seconds while also improving the quality of incident analysis.”
Customer Incident Report Service Automation team

We validated the offering with a company-wide, 4-week hackathon: 14 teams and 150 employees participated, and adoption has since grown to 30 teams across 25 divisions. The following two systems had the most technically instructive outcomes.

Deep dive 1: ErrorWatcher: A sequential multi-agent incident analyzer

The Vehicle Control Service Development team runs connected car services (digital key, vehicle security) all day, worldwide, where diagnosing an incident means correlating signals across application, database, Kubernetes, and load-balancer layers. Because this process requires hours of manual work, the team developed a solution called ErrorWatcher, which won the hackathon’s Spotlight Award. ErrorWatcher is a LangGraph multi-agent pipeline of four role-specialized agents under an orchestrator: Monitor, Detective, Solver, and Reporter, as shown in the following figure.

Figure 3: ErrorWatcher sequential multi-agent pipeline

Why LangGraph instead of a managed agent runtime

The team evaluated a fully managed agent service, but needed deterministic, Python-level control over how state passes between stages and over each agent’s tool scope. They modeled the pipeline as a LangGraph StateGraph with an explicit, typed state object, gaining precise control over execution order and inter-agent contracts at every stage.

class AgentState(TypedDict):
    original_query: str          # Triggering alert / query
    symptom_summary: Dict        # Monitor output
    root_cause_analysis: Dict    # Detective output
    solution_architecture: Dict  # Solver output
    final_report: str            # Reporter output

workflow = StateGraph(AgentState)
workflow.add_node("monitor",   call_monitor_agent)
workflow.add_node("detective", call_detective_agent)
workflow.add_node("solver",    call_solver_agent)
workflow.add_node("reporter",  call_reporter_agent)

workflow.set_entry_point("monitor")
workflow.add_edge("monitor",   "detective")
workflow.add_edge("detective", "solver")
workflow.add_edge("solver",    "reporter")
app = workflow.compile()

RAG with per-agent metadata filtering

All knowledge-grounded agents query the same Amazon Bedrock Knowledge Bases index (backed by Amazon OpenSearch Serverless), but each constrains retrieval to the corpus relevant to its job through metadata filters. This raises precision and cuts token cost by excluding irrelevant documents before they ever reach the model.

The Detective agent passes a retrieval filter alongside the query so that the vector search is scoped at request time:

retrieve_config = {
    "vectorSearchConfiguration": {
        "filter": {
            "in": {"key": "source_folder",
                   "value": ["past-incidents", "architecture"]}
        }
    }
}

Confidence scoring and a self-improving knowledge loop

Every agent attaches a confidence label (HIGH, MEDIUM, or LOW) to its output, computed from retrieval relevance and the model’s own assessment. The Reporter writes the final analysis to Amazon S3, and any report scored HIGH is promoted into the learned-cases/ prefix indexed by the knowledge base. This turns the resolution of a novel incident today into retrievable context for tomorrow. The following walkthrough shows the data contract as it flows through the pipeline for a Kubernetes CrashLoopBackOff caused by an OOMKilled pod.

# Monitor -> symptom_summary
{
  "detected_domain": "INFRASTRUCTURE",
  "key_metrics": [{"metric": "k8s_pod_restarts_total", "value": 15}],
  "key_logs": [{"pattern": "CrashLoopBackOff detected", "count": 15}]
}

# Detective -> root_cause_analysis
{
  "root_cause": "OOMKilled: pod memory limit below working set",
  "confidence_level": "HIGH",
  "evidence": ["CrashLoopBackOff x15", "matches 2025-08-14 incident"]
}

# Solver -> solution_architecture
{
  "short_term": ["kubectl patch deployment ... limits.memory=1Gi",
                 "kubectl rollout restart deployment/vehicle-service"],
  "long_term": ["Add HPA on CPU/memory utilization"],
  "solution_confidence": "HIGH"
}

Design trade-off: the pipeline is deliberately sequential, not parallel. Because each stage depends on the previous stage’s structured output, a linear StateGraph keeps the contract straightforward and debuggable; the cost is latency, which is acceptable here because the end-to-end run still completes well under the five-minute target.

The result: mean time to resolution (MTTR) dropped from hours to 5 minutes, with identical quality regardless of which engineer is on call or when the incident occurs.

Deep dive 2: A fault-tolerant big data incident response agent

The Data Platform Technology team runs Hadoop-based clusters at all times, all year, where one alert triggers a long chain of manual checks and response quality varies by on-call engineer. To streamline this process, the team built a system that earned the Excellence Award at the hackathon. Their system is a 14-node LangGraph workflow (9 active nodes in production) that automates the path from alert to RCA. It splits into a VDI Agent (an Outlook monitor, a Teams interface, and a read-only SSH runner) and a Main Agent Server (a FastAPI gateway, the LangGraph orchestrator, Amazon Relational Database Service (Amazon RDS) for PostgreSQL for state, Amazon OpenSearch Service for logs, and Amazon Bedrock for the models). Nodes communicate with the VDI runner over a WebSocket bridge. The following figure shows the interaction between the local VDI agent and AWS core services for automated incident detection, orchestration, and resolution.

Figure 4: Overall system architecture showing the interaction between the elements

State schema: Delta returns and reducer merge for safe parallelism

Because several nodes run in parallel and write to shared state, naive full-state writes would race. The team has each node return only its delta and registers a reducer per field using Annotated, so LangGraph merges concurrent writes deterministically instead of overwriting.

# imports omitted
def _merge_dict(a, b):
    out = dict(a or {}); out.update(b or {}); return out

def _add_list(a, b):
    return (a or []) + (b or [])

def _last_value(a, b):
    return b if b is not None else a

class GState(TypedDict, total=False):
    incident_id: str
    status:    Annotated[str, _last_value]
    analysis:  Annotated[Dict[str, Any], _merge_dict]
    timeline:  Annotated[List[Dict[str, Any]], _add_list]
    pre_snapshot: Annotated[Dict[str, Any], _merge_dict]

With the _merge_dict reducer on analysis, the parallel RCA-A and RCA-B nodes can each write analysis.rca_a and analysis.rca_b respectively without interfering with one another. State is persisted with PostgresSaver checkpointing on Amazon RDS for PostgreSQL at every node boundary, so a long-running incident workflow resumes from its last checkpoint after any interruption—an essential capability for runs that pause for human approval.

Cost-aware log triage on Amazon OpenSearch Service

The Log Investigator first runs an aggregation to find which components are failing, rather than streaming raw logs into a model. This Query DSL aggregation buckets events by component and severity over the incident window:

POST logs-*/_search
{
  "size": 0,
  "query": {"bool": {"filter": [
    {"term":  {"cluster_id": "PROD01"}},
    {"range": {"@timestamp": {"gte": "...T13:00:00Z", "lte": "...T13:20:00Z"}}}
  ]}},
  "aggs": {"by_component": {
    "terms": {"field": "component.keyword", "size": 20},
    "aggs":  {"sev": {"terms": {"field": "level.keyword"}}}
  }}
}

From the aggregation, the agent generates up to three structured SearchIntent objects, then applies a two-stage large language model (LLM) pattern: a low-cost triage model decides whether retrieved evidence is sufficient, escalating to a high-performance model only on a DEEPER_DIVE verdict.

Parallel RCA with self-falsification and a Reflector

A single model doing RCA is prone to confirmation bias. The team runs two distinct Amazon Bedrock reasoning models, RCA-A and RCA-B, over the same EvidencePack independently. Each must generate falsifications of its own conclusion; the harder its conclusion is to refute, the higher its robustness_score. A join gate waits for both, then a reflector cross-validates the two analyses and adjudicates between them. Because both rca_a and rca_b have edges into rca_join, LangGraph runs the join node only after both branches complete; the router then forwards to the Reflector, and the wait branch is a defensive mechanism.

class RCAOutput(BaseModel):
    root_cause: str
    confidence: float          # 0.0 - 1.0
    robustness_score: float    # from self-falsification
    final_confidence: float    # confidence + robustness
    key_evidence: list[str]
    falsifications: list[str]  # model attacks on its own answer

def _rca_join_router(state: GState) -> str:
    analysis = state.get("analysis", {}) or {}
    a = state["analysis"].get("rca_a", {}).get("root_cause")
    b = state["analysis"].get("rca_b", {}).get("root_cause")
    return "go" if (a and b) else "wait"

g.add_conditional_edges("rca_join", _rca_join_router,
                        {"go": "run_rca_reflector", "wait": END})

If both models independently converge, confidence rises sharply; if they diverge, the Reflector weighs evidence strength and falsification robustness to decide which prevails, leaving an auditable rationale.

Model routing for cost and quality

Model selection is centralized in an LLMRegistry and can be overridden for each node through an environment variable. Lightweight models absorb roughly 60% of calls, high-performance models are reserved for reasoning-heavy steps.

Operational safety: Human-in-the-loop and read-only diagnostics

Diagnosis is fully automated, but state-changing recovery is not. An approval node pauses the graph and presents the plan in Microsoft Teams; only after operator approval does an executor run the plan, after which a verify step re-checks state up to three times. During diagnosis, the SSH runner is restricted to read-only commands (for example uptime, ss -lntp, jps -l, systemctl status, df -h), and a SHA-256 hash of each (hostname, command) pair deduplicates repeated probes across a loop of up to 50 iterations.

The outcome: The time to the first status report fell from more than 3 hours to a few minutes, quality became consistent across shifts, and every incident’s snapshot, timeline, and trace accumulate as reusable organizational knowledge. The team is evaluating Amazon Bedrock AgentCore to further reduce operational overhead.

Architectural lessons across both systems

The two systems solve different problems but expose a consistent set of design decisions worth generalizing:

• Explicit orchestration over magic: Both systems use LangGraph’s StateGraph for deterministic control of state and tool scope. ErrorWatcher stays sequential for a straightforward, debuggable contract; the big data agent runs in parallel, which requires reducer-based merging and checkpointing. This matches the topology to the dependency structure of the work.
• Push cost control upstream: RAG with metadata filtering and the big data team’s aggregation-first log triage both reduce tokens before the model runs. The two-stage triage to deep-model pattern is the clearest cost lever, avoiding the expensive model whenever inexpensive evidence suffices.
• Engineer against the model’s failure modes: Single-model RCA invites confirmation bias; independent RCA-A and RCA-B with self-falsification and a Reflector turns model disagreement into a signal rather than a silent failure.
• Make safety structural: Read-only diagnostics, human-in-the-loop approval for state changes, and checkpointed, resumable workflows help operations teams trust agentic automation in production.

Conclusion

The GenAI Sandbox changed who can build generative AI at Hyundai AutoEver by making security, cost, multi-tenant access, networking, and tooling part of the environment itself rather than per-project work. On that foundation, two teams built production-grade multi-agent AIOps systems: one a sequential analyzer that cut MTTR from hours to minutes, the other a fault-tolerant, parallel RCA engine with self-falsification and human-in-the-loop recovery.

The patterns here, including ABAC multi-tenancy, inherited security, typed LangGraph state with reducer merges, filtered RAG, aggregation-first log triage, and adjudicated parallel reasoning on Amazon Bedrock, are reusable building blocks. To go deeper:

1. Explore Amazon Bedrock for foundation model access and Amazon Bedrock Knowledge Bases for managed RAG
2. Learn about Amazon OpenSearch Service for log analytics and vector search
3. Contact your AWS account team to discuss how a secure, self-service sandbox could unlock broad AI adoption in your organization