Bringing a Frontier World Model to the Convenience Store: Inside Telexistence’s DreamZero Experiment on AWS

Introduction

Imagine walking into a convenience store in Tokyo and watching a robot scan your items, bag your groceries, and restock the shelves — all without missing a beat. That’s not a vision of the future, it’s what happens when a world model running on AWS gives a robot the ability to imagine before it acts. It’s what Telexistence is doing right now.

Telexistence builds the robots, and the AI that drives them, to take on real work in physical retail environments. The company owns the full stack — from robot hardware to AI model development — and has deployed over 300 robots with major Japanese convenience-store chains. The company is also developing humanoids that handle cashier scanning, bag packing, and shelf picking in live stores, every day. The mission is deceptively simple: one robot application that reliably executes diverse, contact-rich manipulation tasks across every location.

But here’s the hard part. Teaching a robot to grab a rice ball is nothing like teaching it to bag a bottle of tea. Getting there means staying at the absolute frontier of robotics AI — and the latest direction in this field pairs an action policy with a video generation “world model,” essentially giving the robot imagination. Before it moves, the robot can anticipate how the scene will change, planning its actions the way a human might visualize a task before executing it.

This post explores how Telexistence, as part of the Physical AI Fellowship powered by AWS, partnered with the AWS Generative AI Innovation Center (Innovation Center) to push these ideas into production — rigorously developing on NVIDIA’s DreamZero World Action Model for real retail manipulation tasks, and building an end-to-end fine-tuning, curation, and evaluation pipeline entirely on AWS.

Figure 1: A Telexistence robot performing a real retail manipulation task — checking out a bottle of drink at the convenience-store cashier.

Why World Models Matter for Retail Robots

Telexistence is a leading Japanese robotics startup that owns the full stack, from robot hardware to AI model development. Today, its existing robots run on Vision-Language-Action (VLA) models, fine-tuned to handle multiple tasks at the convenience-store cashier. A key strength of the company’s approach is teleoperation: human operators drive the robot through real tasks, capturing high-fidelity demonstrations grounded in real contact and real physics. This data is what makes the models work.

But there’s a fundamental ceiling. A robot policy only generalizes as far as it can anticipate the consequences of its own actions. VLA models — like the ones Telexistence deploys today — map observations directly to motor commands. They have no explicit understanding of what happens next. They react, but they don’t imagine.

Recent robotics foundation models break through this ceiling by adding a video generation model — a World Action Model (WAM) — that predicts future camera views alongside actions. The robot’s left-hand camera, right-hand camera, and head-mounted camera each get a predicted “next frame,” giving the model richer context about how the scene will evolve as it moves. The result: steadier, more capable manipulation.

NVIDIA’s DreamZero is one of the most ambitious models in this direction — a 14-billion-parameter WAM that jointly generates action trajectories and multi-camera future video. The promise is significant. But so are the unknowns. Adapting a model this large to a specific robot embodiment, with specific cameras and a specific set of retail tasks, was uncharted territory.

Which brought Telexistence to a concrete question: would a world-model-based policy like DreamZero actually outperform the VLA models already working in production? This proof of concept (PoC), conducted through the Physical AI Fellowship powered by AWS, set out to find the answer.

“Evaluating a 14-billion-parameter world action model on our own retail tasks is exactly the kind of frontier question we can’t answer from a paper alone — we needed to fine-tune it, run it, and measure it on real and simulated data. Working with the AWS Generative AI Innovation Center under the Physical AI Fellowship, we stood up a complete pipeline on AWS — from data-format conversion and GenAI-based curation to distributed fine-tuning, simulator evaluation, and inference benchmarking — in a matter of weeks. We came away with a clear, evidence-based read on where DreamZero helps, where the real-to-sim gap bites, and exactly what to invest in next. That clarity is worth far more than a benchmark number.” — Genki Sano, CTO of Telexistence

The Challenge: Fine-Tune and Evaluate a Frontier World Model on a New Embodiment

The goal sounds simple — fine-tune DreamZero on our tasks and measure how well it performs — but several hard constraints turned it into a genuine engineering and research effort:

• Cross-embodiment adaptation. Our target robot is Telexistence’s custom robot, TX-G2. The most suitable starting checkpoint was pre-trained on the AgiBot G1 embodiment, so every experiment also carried the burden of generalizing across embodiments
• Incompatible data formats. DreamZero consumes data in NVIDIA’s GEAR format, while our simulation data is LeRobot v2.1 and our real teleoperation data is LeRobot v3.0. Bridging these reliably — at matching camera resolutions — was a prerequisite to any training.
• Evaluation without an edge deployment. With a 14B model that must run in the cloud and no simulator for some real-world tasks, we had to design a multi-pronged offline-and-simulator evaluation protocol that could still produce reliable signals about closed-loop performance.

Meeting all of these at once, under a schedule timed to a live demo at the Robotics Summit & Expo in Boston May 2026, is what made this a real PoC rather than a checkbox.

The Solution: An End-to-End DreamZero Pipeline on AWS

The heart of the engagement is a complete fine-tuning and evaluation pipeline for DreamZero, built and run entirely on AWS infrastructure, and adapted to Telexistence’s exact robot configuration.

The pipeline spans four stages:

1. Data conversion — Simulation (LeRobot v2.1) and real teleoperation (LeRobot v3.0) datasets are converted into the GEAR format DreamZero expects, with all three camera streams rescaled to a consistent resolution.
2. Curation — Recorded demonstrations are filtered for quality, using both manual review and an automated GenAI (vision-language model) multi-agent pipeline that flags noisy episodes.
3. Distributed fine-tuning — DreamZero is fine-tuned from the DROID-AgiBot checkpoint on a single-node Amazon Elastic Compute Cloud (Amazon EC2) p6-b200.48xlarge instance (8× NVIDIA B200 GPUs), with action-head full-parameter training.
4. Evaluation and benchmarking — Fine-tuned models are scored through a multi-pronged protocol: training-loss inspection, generated-video review, per-joint action MSE against ground truth, closed-loop rollouts in the Genie Sim 3.0 simulator, and an automated vision-language judge — plus an inference-latency benchmark across GPU instance types. Figure 2 shows an example of the generated three views for evaluation.

Figure 2: The three synchronized camera streams (head + two wrist views) that form each observation.

The entire workload runs on AWS: Amazon EC2 p6-b200.48xlarge for distributed fine-tuning, Amazon EC2 g7e instances (NVIDIA RTX PRO 6000 Blackwell) for simulation and inference serving, Amazon EC2 instance store and Amazon Simple Storage Service (Amazon S3) for dataset and checkpoint management, and Capacity Blocks for ML to reserve the B200 capacity across the engagement. Figure 3 shows the overall pipeline of the process.

Figure 3: The end-to-end DreamZero fine-tuning and evaluation pipeline on AWS — data conversion, GenAI curation, distributed fine-tuning on B200 GPUs, and multi-pronged evaluation.

Why a GenAI Curation Pipeline Was the Breakthrough

Teleoperation data is high-fidelity, but it is also noisy: a real dataset contains episodes where the robot was idle, picked the wrong-colored object, or where a human hand entered the frame. At scale, inspecting every episode by hand is impractical — and noisy training data measurably degrades what the model learns.

So, we built a multi-agent pipeline, powered by video understanding models, that automatically curates robot demonstration data in two phases:

• Phase 1 — Movement & environment check. Each video is split into 30-second segments. A Movement Agent checks for robot activity while an Environment Agent detects anomalies, such as a human hand appearing in the scene. Demonstrations with more than 60 seconds of inactivity, or with environmental anomalies, are flagged as low-quality.
• Phase 2 — Final conclusion. The pipeline emits a pass/fail verdict per demonstration, combining inactivity detection and task completion status.

To validate it, we ran the pipeline against manual human review on a real pick_block_color dataset of 183 episodes. The automated pipeline flagged 72/183 episodes (39.3%) as noisy; human reviewers found 91/183 (49.7%). The two are roughly consistent, validating the pipeline as a scalable first-pass filter — and the ~10% gap was instructive, traced mainly to the model’s difficulty distinguishing close colors (blue vs. purple) and to dataset anomalies only a careful human caught. Crucially, the same architecture extends well beyond curation: it can filter augmented data, screen both sim and real sources, and — by judging task success automatically — open the door to online reinforcement learning.

How We Built It

Delivering this PoC meant making progress on several fronts: bridging incompatible data formats across embodiments and designing an evaluation that produces trustworthy signals without an edge deployment. Here is how each came together.

Fine-Tuning and Evaluating in Simulation

On clean Genie Sim 3.0 pick_block_color data, DreamZero fine-tuned successfully from the DROID-AgiBot checkpoint, with smoothly converging action and video-generation losses. We then evaluated four model configurations in closed-loop simulation, varying training steps, joint-control scope, and data composition. The results were revealing:

Figure 4: Training on clean Genie Sim 3.0 data: total loss and the action / dynamics components converge smoothly over 2,000 steps.

The best model reached a 45% success rate — and, surprisingly, it was trained exclusively on place trajectories, suggesting place motions carry richer manipulation primitives that transfer to picking.

• A model trained for 2000 steps on task-aligned pick data reached 35%, confirming that modest, well-targeted training is enough for meaningful closed-loop performance.
• We reinforced these findings with offline per-joint MSE analysis, which surfaced an important lesson: offline MSE does not monotonically predict closed-loop success.

A model with low average prediction error can still fail in the loop, while a model with higher MSE can succeed by having learned transferable primitives. Closed-loop evaluation remained the definitive measure. An example of training loss decrement is shown in the previous Figure 4.

Confronting the Real-to-Sim Gap

When we fine-tuned on Telexistence’s real pick_block_color data and evaluated in simulation, success dropped to 0% — and the failure was diagnostic, not disappointing. Analyzing joint-angle distributions across 21 real episodes against simulation revealed a fundamental domain gap: mismatched joint distributions, different state scales (in radians), a real dataset covering only right-arm manipulation, and a missing joint mapping between the physical robot and the simulation model. Combined with residual noise in the teleoperation data, this fully explained the gap — and pointed directly at what closing it would require.

We also fine-tuned on a real cashier dataset (692 episodes, 1,395 bottle-pick sub-task segments), focusing on the bottle-pick sub-task and offline action MSE since no simulator exists for this task. Increasing training from 1000 to 2000 steps improved mean MSE by 55.6%. Figure 5 shows a representative episode from this evaluation.

Figure 5: Per-joint action MSE for cashier task Episode 442 after 1,000 and 2,000 training steps. Most active joints improved after longer training, particularly J14 and J15, while J7 and J10 regressed.

In this representative episode, additional training reduced error on most active joints, but did not improve every control dimension uniformly. This distinction matters because offline MSE measures trajectory agreement, not task completion. For tasks with a simulator, closed-loop success remains the definitive measure; for cashier, the MSE breakdown provides a practical offline signal and identifies joints that may need targeted data or training adjustments.

Making Inference Practical

Finally, we benchmarked inference latency on two AWS GPU instances. The Amazon EC2 p6 (B200) instance ran roughly 8× faster than EC2 g7e (RTX PRO 6000 Blackwell), holding a stable ~583 ms per step — near the range needed for real-time control. We reached this through a stack of optimizations, the largest being DiT caching (reusing computed velocities to skip redundant forward passes), plus reducing denoising steps from 16 to 4, decoupling video noise, and applying torch.compile with CUDA Graphs.

Results That Reframed the Question

This PoC produced exactly what an evaluation should: clear evidence about where a frontier model helps and where it doesn’t, on our own tasks and infrastructure.

What worked

DreamZero fine-tunes successfully on the TX-G2 embodiment from a G1 checkpoint — a real cross-embodiment generalization result. On clean simulation data, longer task-aligned training produced visibly better execution and video generation, losses converged smoothly, and the automated evaluation stack (Genie Sim 3.0 + a DreamZero inference server + a vision-language judge) produced coherent, automated success verdicts.

All success rates, MSE figures, latency measurements, and other quantitative results reported in this post were obtained during internal testing conducted as part of this engagement.

What surfaced

Real-world data is materially harder than sim — fine-tuning on real demonstrations showed mid-training loss spikes that point to a need for stronger curation. And offline evaluation has limited diagnostic power for online behavior: because the offline setup feeds the model ground-truth video frames, generated actions deviate only slightly from ground truth by design, so offline rollouts understate the model’s true closed-loop influence.

These were not subjective impressions but measured outcomes — success rates, per-joint MSE breakdowns, distribution analyses, and latency benchmarks — that together give Telexistence an evidence-based footing for its next decisions.

What This Demonstrates

In this post, we showed how the AWS Generative AI Innovation Center collaborated with Telexistence to turn an open frontier question — is a world action model right for our retail robots? — into measured answers. By building a complete, cloud-native pipeline for data conversion, GenAI-based curation, distributed fine-tuning, and multi-pronged evaluation, we were able to fine-tune a 14B model across embodiments, quantify its closed-loop performance, diagnose the real-to-sim gap precisely, and benchmark production inference — all in weeks.

The broader takeaway is that rigorous, cloud-scale evaluation is itself a reusable capability. The same pipeline — and especially the GenAI multi-agent curation and judging components — applies to other robot-foundation-model builders weighing a new architecture, on other embodiments, against their own data.

Get Started

If you’re exploring how generative and physical AI can drive autonomous systems in your organization — whether in robotics, manufacturing, retail, or beyond — the AWS Generative AI Innovation Center can help. The program has supported customers including Formula 1, Nasdaq, Ryanair, and S&P Global, with more than 73% of recent projects reaching production. Connect with the Innovation Center team to start your journey, or explore Amazon SageMaker AI to begin building today!