How Config Scales Robot Training Data Without Scaling Data Collection

Config Intelligence Inc. (Config) is building the data infrastructure and technology necessary to realize General-Purpose Robot Foundation Models. They operate an end-to-end pipeline that spans from large-scale data collection and preprocessing to model training and real-world deployment, enabling robots to perform bimanual manipulation tasks reliably across diverse environments. To date, they have accumulated more than 200,000 hours of action data and continue to collect data at a rate of roughly 20,000 hours per month.

Figure 1: About Config

A key strength of Config is its proprietary pipeline that estimates robot-aligned actions from video footage of humans performing tasks. This approach allows its customers to efficiently secure vast, diverse training datasets while significantly reducing the costs and operational burdens associated with robotic data collection. The resulting Robot Foundation Models (RFMs) can be fine-tuned to master customer-specific target tasks and deployed within just 48 hours, requiring only a small amount of task-specific data.

In this post, the Config and AWS Generative AI Innovation Center (GenAIIC) teams introduce how Config extended its real-world data acquisition pipeline with a model-assisted data diversification approach, designed to broaden dataset coverage and improve the generalization performance of the Robot Foundation Models.

With this initiative, Config can amplify the visual diversity of demonstrations already collected by human operators, multiplying the value of every hour of real-world data capture without sacrificing the physical realism essential for effective robotic training.

The Hidden Bottleneck in Physical AI

Training a robust robot foundation model (RFM) requires more than simply collecting a large amount of data. The dataset itself must be high-quality, not only in terms of physical fidelity and action accuracy, but also in terms of diversity. For robotics, diversity is one of the most important and often overlooked dimensions of data quality.

A model only generalizes as far as the variety in its training data reaches: the more backgrounds, lighting conditions, object types, surfaces, and tasks it has seen, the more capable it becomes when faced with new real-world environments.

Config pursues diversity the rigorous way: by collecting real demonstrations. Human data collectors perform tasks across diverse configurations — varying objects, lighting, surfaces, backgrounds, and task settings — captured by synchronized overhead and wrist-mounted cameras. The recordings are then converted into training data, with each demonstration paired with actions represented in the target robot-aligned action space. The result is high-quality, physically grounded data that reflects real contact, real geometry, and real-world variation.

The problem is that this kind of diversity scales linearly with human labor. Each new combination of surface, lighting, and object is another set of episodes a person has to record. To double the diversity would mean roughly doubling the operator hours, and those hours represent irreplaceable work capturing real physics. For a foundation model that needs broad environmental coverage, the question becomes how to get more value from every demonstration already collected.

As part of the Physical AI Fellowship powered by AWS, Config partnered with the AWS Generative AI Innovation Center (GenAIIC) to diversify its data efficiently at scale, multiplying the variety in its dataset without a proportional increase in human effort, and crucially, without sacrificing the physical realism that makes the real demonstrations worth training on in the first place.

“Before this collaboration, improving our robots’ ability to handle new environments often required collecting additional real-world demonstrations across diverse conditions. We had built a real-world data acquisition pipeline with diversity protocols, but scaling that diversity still required significant operational effort. Working with the AWS Generative AI Innovation Center, we developed a complementary pipeline that helps us broaden dataset diversity more efficiently. This has contributed to stronger generalization in our models and gives us a more scalable way to improve robot foundation models for real-world deployment.” – Hyungmok Son, CTO

The Challenge: Diversify while Preserving the Physics

The goal of this project sounds simple — re-render existing episodes under new surfaces and lighting — but three hard constraints made it anything but:

• Action labels must stay valid. The data carries precise labels for what the robot does, where it grasps, where it makes contact, and the path it follows. Any change to the scene must leave the objects’ shapes, positions, and physics untouched, so those labels stay accurate.
• The gripper must stay unchanged. The shape of the grippers, and especially their distinctive green fingertips, is a visual cue used later in Config’s pipeline. The augmentation can repaint backgrounds and surfaces, but it must never distort the gripper itself.
• Everything must stay consistent over time and across all three camera views. The new scene has to stay stable across an entire video (no unphysical surface changes or shifting lighting) and appear consistently across all three cameras, with no object appearing in one view but missing from another.

Meeting all three at once is what turned a straightforward image-editing idea into a genuine research problem.

The Solution: A Custom Multi-View Cosmos-Transfer Model on AWS

Figure 2: The generative augmentation pipeline on AWS — episodes flow from Amazon Simple Storage Service (Amazon S3) through Amazon Bedrock captioning and the post-trained Cosmos-Transfer2.5 model running on Amazon SageMaker HyperPod / Amazon Elastic Compute Cloud (Amazon EC2), back to Amazon S3.

The heart of the engagement is a custom, post-trained version of NVIDIA’s Cosmos-Transfer2.5 , a generative video model for data augmentation, adapted to Config‘s exact camera setup and run on AWS infrastructure.

The pipeline takes one existing episode and produces a diversified copy in five stages:

1. Input — three synchronized camera views of Config’s data are loaded from Amazon S3.
2. Captioning — a vision-language model (running on Amazon Bedrock) describes what each camera sees and saves as text captions.
3. Diversifying — captions are edited to specify a new look, such as a different surface or lighting (chosen from 88 augmentation styles).
4. Edge extraction— the original scene’s edge outlines are extracted so the new version keeps every object in exactly the same place.
5. Re-render — the model regenerates all three camera views together in the new style, producing a fresh, fully-labeled episode that stays true to the original motion.

The entire pipeline runs on AWS, using Amazon SageMaker HyperPod or Amazon EC2 for GPU compute, Amazon FSx for Lustre for fast data access during training, and Amazon S3 for inference input and output storage.

Why “Multi-View” Was the Breakthrough

The project began with the public, single-view version of the model, which processes each camera independently. It worked, but it had three problems that no amount of tuning could fix: colors drifted between the separately generated views, the model created “ghost” objects that were never in the scene, and keeping the gripper visually consistent required heavy, costly extra processing on every frame.

Figure 3: A failure mode of the single-view approach — Gripper color was not well preserved / Color of the table and inside of the cup are not consistent across views.

Figure 4: A failure mode of the single-view approach — Inconsistent ghost objects and glares

The key insight was to train a single model that generates all three camera views together, so they stay consistent with one another, and to fine-tune it directly on Config’s own demonstrations. Because it learned from Config’s real data, it reproduces the visuals of the gripper (including the distinctive green fingertips) on its own, with consistent colors across views and sharper detail, and without the costly extra processing the single-view approach required.

How We Built It

Delivering this pipeline meant making progress in three areas: teaching the model to faithfully reproduce Config’s specific setup, scaling the training data efficiently, and making generation fast enough to be practical at production volumes. Here is how each came together.

A Multi-View Model, Trained by Config’s Data

We started from NVIDIA’s Cosmos-Transfer2.5, a powerful general-purpose video generation model, and made two changes that, together, addressed the hardest requirements of the project.

First, we adopted the model’s multi-view design, which generates all three camera views in a single, coordinated pass rather than one at a time. The mechanism that makes this work is cross-view attention: as the model generates each view, it continuously “looks at” the corresponding points in the other two views, so the three are produced in agreement with one another. The result is that the same surfaces, lighting, and objects appear consistently across every camera, directly addressing the cross-view consistency problem that defeated the single-view approach.
Second, we post-trained the model on Config’s own demonstrations, fine-tuning it rather than rebuilding it from scratch, so it learned Config’s particular setup: how the two arms and grippers look, including the distinctive green fingertips.

Scaling the Training Data

To train the multi-view Cosmos-Transfer2.5 model, we progressively scaled up the training dataset. We deliberately started small, using a few hundred episodes to get the training recipe right and confirm the model was learning the right behavior. Only once that recipe was stable did we scale up in stages: first to a few thousand episodes, and finally to 15,000 episodes, totaling 250,000 short video segments for training. Scaling in stages meant we caught issues early, when experiments were cheap, rather than after committing weeks of GPU time.

Captioning Strategy for Addressing the “Ghost Object” Problem

One stubborn issue was “ghost objects”, items the model painted into a scene that were never actually there. Counterintuitively, the fix had nothing to do with the model itself; it came down to how we described each scene to it. Early on, we summarized an entire video clip with a single description, but a robot’s wrist-camera view changes quickly as the arm moves, so that one description soon fell out of step with what was on screen. The solution was to describe the video in short, frequent segments, so the description consistently matched the current frame. In a blind human review, this change was preferred 69% of the time versus 31% for the previous approach, and the ghost objects largely disappeared.

Making It Fast Enough for Production

Scale also depends on speed. Because Config aims to scale augmented video generation to large datasets, per-clip generation time becomes one of the dominant costs, so we made inference speed a first-class goal. Three complementary optimizations, stacked together, achieved a 4.17× speedup with no meaningful loss in quality (less than a 2% drop in Peak Signal-to-Noise Ratio (PSNR), a standard measure of image fidelity):

• DiT caching lets us skip redundant computation. The model refines each clip over a series of small steps, but its output barely changes from one step to the next. By reusing cached results, we skip roughly 70% of denoising steps.
• FP8 precision takes advantage of NVIDIA’s Transformer Engine to run the model’s heaviest calculations at lower numerical precision (8-bit), which modern GPUs process much faster.
• A compiled tokenizer speeds up the component that converts video to and from the model’s internal format, so it runs more efficiently.

In our evaluations, none of the three noticeably degrades output quality. Together, they turn a workload that would otherwise be prohibitively expensive into one that is economically practical at production scale.

Results: A More Robust Robot Policy

The trained multi-view Cosmos-Transfer2.5 model produces diversified episodes with no ghost objects, strong cross-view consistency, and intact gripper fidelity, across 88 surface and lighting strategies.

Figure 5: Before: a failure mode of the single-view approach — “ghost” objects hallucinated by the model.

Figure 6: After: a successful multi-view augmentation from the multi-view trained model — consistent surfaces and lighting across all three views, with the gripper preserved.

These weren’t subjective impressions. A blind, side-by-side human evaluation — 5 evaluators, 15 samples spanning chromakey, static, and moving scenes, and 75 votes in total — pitted the multi-view models against the single-view approaches head-to-head. The multi-view model captured 68% of the votes, versus 32% for the single-view model, and multi-view led in every scene category.

The Real Test: Does It Make the Robot Policy Better?

Better-looking video is only worth it if it makes the robot better. To find out, Config ran the experiments: training their robot policy model[HS1]  with various mixtures of real and augmented datasets and mixing augmented data at various stages of training to analyze the effect of augmentation. They then evaluated the model in out-of-distribution conditions — unfamiliar table covers and lighting the model had never seen during training.

Figure 7: (a) Target task: moving the canned goods on the left to the top box and the donut on the right to the bottom plate. (b) Out-of-distribution conditions for evaluation

The results were striking. While the baseline policy achieved nearly 100% success on in-distribution evaluations, its performance dropped dramatically to just 8.3% on these challenging out-of-distribution trials. Adding Config’s Cosmos-augmented data lifted that to as high as 75.0%, roughly a 9× improvement in robustness to distribution shift. They also found that mixing augmented data with the original real data was more effective than using augmented data alone, and introducing augmented data at pretraining stage produced the strongest results.

What This Demonstrates

In this post, we showed how the AWS Generative AI Innovation Center collaborated with Config to turn a known generalization bottleneck into a scalable lever for data diversity: a custom, multi-view generative model that re-renders existing episodes under new surfaces and lighting, multiplying dataset variety without scaling data collection. In an upcoming companion post, we cover a second, complementary path, a physics-grounded simulator that generates entirely new demonstrations on demand.

The broader takeaway is that diversifying robot data this way is an effective, efficient, and reusable solution, one that robot-foundation-model builders can apply to make their model more robust and achieve better real-world performance. By post-training a generative model on a customer’s own demonstrations and running it on cloud-scale GPU infrastructure, the same pipeline amplifies the diversity of an existing dataset, complementing, not replacing, the real-world demonstrations that human operators provide.

Looking Ahead

The immediate path forward is to put the delivered pipeline to work at production scale. Further opportunities include object-level augmentation, hybrid generative-plus-simulator data generation, and extended model training with the latest Cosmos 3 model.

Get Started

If you’re exploring how generative AI can drive autonomous systems in your organization, whether in security, industrial inspection, or beyond, the AWS Generative AI Innovation Center can help. The program has supported customers including Formula 1, Nasdaq, Ryanair, and S&P Global, with more than 73% of recent projects reaching production. Connect with the GenAIIC team to start your journey!

Data Attribution: Quantitative metrics cited in this post, including training data volume, generation throughput, deployment timelines, and production success rates, are based on Config’s internal benchmarks and operational records as of July 2026.