How BMW builds toward Industry AI on Amazon SageMaker AI

BMW logos

This post was co-authored with Saltuk Kezer, Johannes Klepsch, and Marvin Erdmann from BMW Group.

The authors would like to thank Sharlina Keshava, Thomas Welle, Christelle Xu, Nicole Meyers, Brian Jensen, Mark Norton, Alexander Opfolter (AWS), and Andre Luckow, Christoph Grote, and Martin Tietze (BMW Group) for their contributions in making this project possible.

BMW Group is building toward Industry AI on Amazon SageMaker AI by fine-tuning task-specific LoRA adapters on engineering process data and serving multiple domain capabilities from a single endpoint. Large language models (LLMs) produce convincing output on general tasks, but company-internal engineering processes, proprietary taxonomies, and domain-specific quality standards aren’t represented in public training corpora. Prompting with rules and examples doesn’t reliably close this gap when a task imposes many hard constraints simultaneously because the model has no training signal telling it which constraints are inviolable.

BMW Group’s Industry AI strategy addresses this gap through domain-specific models and agents that combine industrial knowledge with BMW data, tools, and workflows. Rather than being centered on a single model, it relies on a portfolio of capabilities that can evolve over time. During a 12-week collaboration with the AWS Forward Deployed Engineering (FDE) team, we took a concrete step toward it: proving the method on real use cases, building the repeatable infrastructure, and establishing where fine-tuning delivers value that prompting alone can’t.

In this post, we describe the architecture, walk through the results, and present a repeatable framework you can apply to your own domain.

Choosing the right model customization strategy

Enterprises exploring domain-specific models typically consider two paths first. Their suitability depends on the capability gap, data availability, and target use case.

Continued pre-training (CPT) on proprietary data is the intuitive way to build a domain model from the ground up, but it’s a significant investment, and models that undergo CPT still need supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT) afterward, with data that reflect downstream tasks. Beginning with task-specific SFT can be a pragmatic first step before CPT. It builds the evaluation infrastructure, data-quality discipline, and fine-tuning methodology that later customization can reuse, and it can deliver measurable value earlier with a smaller initial investment.

A larger model with a detailed prompt is a reasonable baseline. But in our tests, prompted models treated some hard constraints (preserve every identifier, never hallucinate, match the language, follow a strict output structure) as soft preferences, and more detailed instructions didn’t fix this. The results section quantifies this failure mode.

A key enabler for developing Industry AI capabilities is reliable, automated evaluation for each downstream use case. Without it, you can’t tell what the common failure modes are, whether fine-tuning helped, or whether more data and compute are warranted.

We scoped this engagement to stage one of three: task-specific adaptation, then broader multi-domain adaptation and, eventually, foundation model development.

Two use cases in the BMW incident management workflow

We chose the BMW engineering incident management workflow as the proving ground. It surfaces design improvement opportunities during research and development, speeds up the resolution of vehicle issues, and spans the whole development process, so what we learn here transfers broadly. The data carries two natural supervision signals: each ticket’s edit history and root-cause analysis outcomes (which incidents were linked to the same underlying issue). It’s bilingual, roughly half German and half English.

We scoped two use cases:

Ticket quality assurance – Help engineers produce high-quality incident tickets faster, so issues route and resolve sooner. It decomposes into three generative tasks: rewrite the title into a concise, triage-ready form; rewrite the description into a standard structure; and generate clarifying questions that help surface missing information.
Incident matching – Given a new incident and a set of 1–8 candidate historical incidents, reason about which (if any) share the same underlying root cause. This is a discriminative matching task that helps engineers find related issues and avoid duplicate investigations.

The two tasks are deliberately different, one generative and one discriminative, to show that the same infrastructure works for both.

The method: Benchmark, curate, train, evaluate, compose

The solution is one repeatable workflow: two Amazon SageMaker Pipelines, one that curates data and one that trains and evaluates an adapter, plus a single real-time endpoint that serves every adapter from one shared base model. Amazon Bedrock in European Union (EU) Regions runs the generator and judge models. SageMaker training jobs train model adapters on ephemeral GPU instances, Amazon Simple Storage Service (Amazon S3) stores the artifacts, and Amazon SageMaker AI with MLflow tracks every experiment. Every step is parameterized, cached, versioned, and reproducible. The following figure shows the architecture. Solid green lines show data flow to and from Amazon S3, dashed teal lines show Amazon Bedrock inference, and dashed purple lines show MLflow tracking.

Figure 1: End-to-end architecture

Choosing the models

No single leaderboard captures the required bilingual domain reasoning, so we benchmarked 16 foundation models (FMs) (2B–235B parameters) on a bilingual suite of engineering reasoning, math, extractive question answering, and translation tasks.

That study picked three roles:

A target model to fine-tune – Gemma 4 E4B, a small open-weight model with roughly 4B active and about 8B total parameters that’s cheap to iterate on and to serve.
A generator model for data curation – Qwen3-235B for ticket quality and Gemma 4 31B for incident matching, chosen for bilingual generation and reasoning quality.
A prompted baseline to beat – Gemma 4 31B, about four times the target model’s size and given the full task instructions and rubrics.

Curating training data

Each task’s curation runs as a SageMaker pipeline of three parameterized processing steps, with step-level caching so reruns skip unchanged steps:

Generate – A generator model on Amazon Bedrock uses batch inference to produce tens of thousands of candidate training pairs per run. We randomly strip metadata and descriptions from some samples so the model learns to handle the sparse inputs of a newly created incident.
Filter – A separate judge model on Amazon Bedrock drops any pair that fails a task-specific gate (no hallucination, all identifiers preserved, valid structure, matching language).
Export – We split training and evaluation temporally, evaluating on incidents created after the training cutoff, to prevent leakage and mirror production.

Generation and filtering run on Amazon Bedrock in EU Regions, which keeps the data in Region and frees GPU capacity for training.

Training

We train every adapter with QLoRA (4-bit NF4) on the shared base model, using SageMaker training jobs with Unsloth and Hugging Face Transformers Reinforcement Learning (TRL). Amazon SageMaker AI with MLflow records full lineage from training data through hyperparameters to evaluation metrics.

Evaluating

For the generative ticket quality tasks, we combine two layers of automated evaluation using Amazon Bedrock.

First, compliance gates are pass-or-fail checks that every output must satisfy. For rewrite tasks, these are:

Hallucination – Nothing invented beyond the source
Content preservation and information loss – Nothing dropped
Structure – Valid output format
Language – Response matches the ticket’s language

For clarifying questions, these are:

Grounding – Targets a real gap
Redundancy – Doesn’t ask for something already present in the ticket
Language – Matches the ticket’s language

Second, an LLM judge scores overall ticket quality on a 1–5 rubric: format precision and triage value for titles, and clarity, completeness, structure, and terminology for descriptions.

The judge (Qwen3-235B) is a different model from the generator, to reduce confirmation bias. Incident matching uses deterministic metrics (F1, exact match). Because evaluation runs as an Amazon Bedrock batch job inside the training pipeline, every experiment produces comparable metrics automatically.

One endpoint, many capabilities

A LoRA adapter is tens of megabytes against a base model of tens of gigabytes. One base model hosts many task adapters behind a single endpoint: the caller names a capability per request, and every adapter reuses the base model’s GPU memory. The following diagram shows stage 3 of the architecture in more detail.

Figure 2: The multi-adapter inference stage

We combine Amazon SageMaker efficient multi-adapter inference and multi-LoRA serving with vLLM in one bring your own container (BYOC) vLLM endpoint backed by the DJL-LMI serving container. The base model and each adapter run on a single GPU, and each adapter loads from GPU, CPU, or disk in milliseconds.

A single joint adapter can serve a whole task family. For ticket quality, one joint adapter served all three tasks near-best, although per-task adapters still give the best per-task performance.

Results

All results are single-seed, single-hyperparameter runs on held-out evaluation sets, and incident matching uses fixed candidate sets rather than end-to-end retrieval. Treat them as strong directional evidence rather than a tuned ceiling.

Ticket quality assurance

Given many simultaneous constraints (preserve every routing code, never invent a component, match the ticket’s language, follow a six-section structure), the prompted 31B baseline treated some as soft preferences. It hallucinated routing codes in about 19 percent of title rewrites and missed language matching in about 13 percent of clarifying questions. Fine-tuning on curated examples that pass every gate fixed this. The fine-tuned model gained 8 points of compliance on title rewrite and 13 points on clarifying questions and came within a point on descriptions. The following graph shows compliance by task.

Figure 3: Ticket quality compliance by task

On the 1–5 rubric, every model lifts quality well above the human baseline (the original tickets). The fine-tuned model raised title quality from 3.74 to 4.71 and description quality from 2.84 to 3.87, matching the larger prompted model on titles (4.75) and exceeding it on descriptions (3.77), while winning on compliance. The main remaining gap is description completeness (2.78 out of 5): by design, the model doesn’t invent information the reporter didn’t provide, and the clarifying questions task closes that gap by asking for it at the start. The following graph shows rubric quality by task.

Figure 4: Rubric quality (1–5) by task

Incident matching

The prompted models showed a bias toward finding connections: they identified matches even when no candidate shared the root cause. The fine-tuned joint adapter beat the prompted baseline, which is about four times larger, at every candidate-set size, by about 9.6 to 14.7 points of F1. Most of the gain comes from recognizing when nothing matches and returning an empty list. With two candidates, the larger prompted model scored 47 percent no-match F1 and the same-size prompted model 25 percent, and the fine-tuned adapter reached 70 percent. Unusually, its no-match F1 rose as the candidate set grew, as the following graph shows.

Figure 5: No-match F1 by candidate-set size

A reusable framework

The engagement produced a repeatable, four-step framework for building toward Industry AI capabilities:

Map use cases – Map the use cases across your domain against available data and current model capability.
For each use case – Define an evaluation and an off-the-shelf baseline, curate high-quality training data (filter aggressively: data that violates your compliance gates teaches the model to violate them), and train a small task-specific adapter.
Compose – Identify tasks that share enough structure to benefit from joint training and serve them from one endpoint.
Let the residual gaps define real investment – Where SFT and RFT can’t close the gap, you’ve found, with evidence, the case for pre-training or architectural work.

The reusable assets are a configurable evaluation harness, data-curation templates, the training pipelines, an adapter registry, and multi-adapter serving. BMW retains control over the use case definitions, evaluation criteria, curated domain data, and adapter artifacts. These are strategic control points for protecting BMW knowledge and comparing future model offerings. Each later use case with similar data reuses this infrastructure, so it’s far cheaper than the first.

Conclusion

BMW Group’s Industry AI strategy is a multi-year journey across domain-specific models and agents. The fastest way to move toward it isn’t to start by pre-training a giant model, which is costly, slow, and still requires the same fine-tuning and evaluation infrastructure afterward.

“We trained a small, task-specific model with 8 billion parameters that broadly matched the quality of a prompted model with 31 billion parameters. It achieved more than 25 percent higher quality scores in our evaluation. With Amazon SageMaker AI Training Jobs managing the training infrastructure, we could focus on AI engineering while keeping our data secure within our AWS environment.” — Andre Luckow, Head of Innovation and Emerging Technologies at BMW Group IT

In this post, we showed how to identify the gaps in off-the-shelf models, fine-tune small adapters on curated domain examples, compose them behind one endpoint, and measure rigorously. In our experiments, fine-tuning on high-quality examples that embody BMW business processes resolved failures of a prompted model four times larger. Each use case you work through this way either ships value directly or shows, with evidence, where the next investment belongs.

AWS for Industries