Physical AI
Simulation
Every vendor quotes a different simulation ratio and they span two orders of magnitude. What the one published sweep found, and where the performance cliff sits.
18 September 2026
Author: Sneha Tanna, Content Writer, FutureBeeAI
Reviewed by: Jesal Thakkar, Founder & CEO, FutureBeeAI
Last updated: 18/09/2026
A team can raise its simulation success rate from 90 to 95 percent and lose real-world performance doing it, without finding out for months. In the only published sweep of the sim-to-real training mixture, policy success held at 0.95 from 90 percent simulated data per batch through 99 percent, then fell to 0.60 at 99.5 percent and 0.40 at 99.9. A sim-to-real data strategy is the decision about how much of a robot policy's training signal comes from a simulator and how much comes from the physical world, and it behaves like a mixture you measure rather than a sequence of phases you plan.
What makes that result urgent is where it lives. It sits in one preprint barely a year old and in none of the nineteen commercial guides reviewed for this piece, each of which offers a ratio instead, spanning two orders of magnitude.
This piece is published by a company that sells real-world data collection, which is why the sections below concede what simulation does better, with figures. A recommendation that always favors whatever the author sells is pricing information.
Both, in a mixture weighted heavily toward simulation by volume and entirely dependent on real data for its anchor and its evaluation. The defensible starting point from published evidence is roughly 90 percent simulated data per training batch against a real anchor set of 20 to 50 demonstrations per task, swept upward toward 99 percent and stopped the moment real-world performance stops improving.
That answer carries two conditions, and without them it is useless. The mixture works only if the simulated and real data describe the same task with aligned camera geometry and identical success criteria, and only if you can measure the result on real hardware rather than inside the simulator that produced the training data. Everything below is those conditions in detail, starting with why the market cannot agree on the number.
Survey the published guidance and a pattern emerges that should make any buyer cautious. One camp recommends roughly 90 percent synthetic data, another recommends 60 to 80 percent real data for dexterous manipulation, and a third quotes 1,000 to 10,000 real trajectories with no scaling rule attached.
They are not in conflict, because they are not measuring the same thing. One counts images in a perception dataset, another splits a procurement budget, a third counts samples in a single training batch. A ratio without a denominator carries no information, which is how this market sustains four incompatible recommendations at once. So when a quote arrives with a mixture attached, the first question is not whether the number is right, it is what the number is a fraction of, which raises whether anybody has measured it.
One study set out to find the number empirically. Training a diffusion policy on a weighted mixture of simulated and real demonstrations, 20 real demonstrations and 1,000 simulated ones, the researchers swept the simulated share per batch and recorded real-world success: 0.60 at 10 percent, 0.75 at 30 percent, 0.85 at 50 percent, 0.95 at 90 percent, 0.95 at 99 percent, then 0.60 at 99.5 percent and 0.40 at 99.9 percent. Across six tasks, the mixture lifted average success from 45.3 percent for real-only training to 83.2 percent, and held even when the real dataset grew to 400 demonstrations per task.
A separate 2026 analysis reached the same region from the opposite direction, sweeping the real-data share and finding best performance consistently between 1.6 and 30 percent real. Two groups, two conventions, one answer: simulation-dominant mixtures win, and the real component stays small in count while remaining non-optional in function.
The same analysis explains the mechanism, and the mechanism matters more than the ratio. It attributed roughly half the variation in training loss to how well the simulated and real representations aligned, and only about a fifth to the mixing ratio, with alignment distance correlating with success at 0.6 to 0.8. Tuning the ratio while your domains stay misaligned optimizes the smaller term.
The far end of that sweep belongs in any planning meeting, because more simulation is not monotonically better. Past some point the extra simulated data drowns the small real signal anchoring the policy to the robot, and because it still lifts simulation scores, the dashboard rises while the hardware degrades.
That failure is only visible if the instrument can see it, and this is where programs quietly lose money. A study running more than 1,500 paired simulated and real evaluations across two robot embodiments and eight task families compared how well evaluation methods predicted real-world performance. Careful real-to-sim evaluation predicted it at 0.929 and 0.890, while the validation-loss metric most teams use to select checkpoints predicted it at 0.308.
A 2025 follow-up reports that on a standard benchmark, nearly all tested policies scored between 90 and 95 percent while spanning the full spectrum from strong to weak in the real world. So a team can rank its checkpoints, ship the strongest, and learn later that the ranking was noise, which is why teams reach for architecture changes when the root cause sits in the data mixture and the instrument judging it.
The mixture is a hyperparameter, hyperparameters get tuned against a metric, and the default metric here is close to uninformative. Before any argument about 10 percent or 90 percent is worth having, somebody has to build an evaluation capable of noticing it is wrong, which means a small real holdout on hardware, at a site the training data never saw, scored on task success.
The collection brief failure we see most often is that the real evaluation set gets carved out of the training set after collection, from the same sessions, the same site, often the same operator. It is then used to justify a simulation ratio, and it cannot, because it shares every condition the training data shares. At FutureBeeAI the holdout site is now specified before the co-training run on sites and kept out of every training run. It costs one extra mobilization, and it is the only reason anybody can say afterward whether the mixture helped.
If the holdout is still an open line item, settle it before anything else gets priced. It is one of two collections specified around a deployment rather than a task and the second is the anchor.
The anchor has a floor that does not scale down as the simulated set scales up and missing that is the most expensive misunderstanding in this category. Teams reason that if simulation is cheap, the real requirement shrinks as simulated volume grows and the sweep says the opposite: past 99 percent simulated, the policy loses its connection to the system it is meant to control. You cannot generate your way out of collection and the working range sits at 20 to 50 aligned demonstrations per task.
The word doing the work there is aligned and it is a specification rather than an adjective. The real episodes share task boundaries with the simulated ones, sit within a stated tolerance of the same camera pose and are scored against an identical success criterion. An anchor set that misses those three is not a weak anchor, it is a second unrelated dataset featuring the same robot.
Clients discover this late, usually after the first training run disappoints. An anchor set collected to a general brief, with the camera wherever the rig allowed and success recorded as a free-text note, cannot be paired with a simulated set without reworking both ends. FutureBeeAI now treats the simulated task definition as the collection specification: the same camera pose within a stated tolerance, the same episode boundaries and the same success criterion as a numeric threshold rather than a sentence. It is the difference between an anchor and a coincidence.
Both are specified around a deployment rather than a task description, which is why they sit ahead of the ratio below.
The ratio is the last decision and running the sequence in this order separates a measured mixture from an inherited one.
1. Classify every gap: Mark each discrepancy between simulator and deployment as a quantity the simulator represents but gets wrong or a phenomenon it does not represent at all. This decides whether money goes to measurement or to collection and nothing downstream can be seized until it exists.
2. Spend on identification before randomization: Measure the parameters you can, then randomize only the residual uncertainty around those tolerances. A small number of identification rollouts closes most of a parametric gap and broad randomization substitutes poorly for measurement.
3. Build the real holdout before the training set: Reserve a site and a task set the training data will never touch, scored on hardware. Without it, every later decision about the mixture is unfalsifiable.
4. Size the real anchor set, not the simulated set: Plan for 20 to 50 aligned demonstrations per task, matched to the simulated task on camera geometry, episode boundaries and success criteria. Alignment explains more of the outcome than the mixture does.
5. Set the batch ratio last: Start at 90 percent simulated, sweep upward, stop when the real holdout stops improving and never assume higher is safer. The published optimum sits between 90 and 99 percent, with a cliff immediately past it.
The step that gets skipped is the third, and skipping it produces the pattern that brings teams here, 90 percent in simulation and 40 percent on hardware. Decisions three and four are both collections specified around a deployment rather than around a task description, which is a different brief from the one most programs write.
A ratio with no denominator: quoted without stating what the number is a fraction of.
Success reported only in simulation: or only as validation loss, which predicts real performance at 0.308.
No real holdout in the plan: or a holdout carved from the same sessions, sites, and operators as the training data.
A real holdout on a separate site: specified up front and excluded from every training run.
The anchor specified to the simulated task definition: camera tolerance and a numeric success threshold included.
Evaluation on hardware by task success: not by loss, and not inside the simulator that made the data.
The mixture is named as a starting point: with a sweep procedure and a stopping rule, not a constant.
Every green flag constrains the supplier and most reduces billable simulation volume, which is what makes a supplier willing to write them into a statement of work worth talking to.
The simulation vs real-world data question is usually framed as two sources competing for one budget. It is better understood as an argument about what you already know. Simulation converts existing knowledge about your deployment into training signals at enormous volume, which is why it dominates any good mixture by count, and real collection is how that knowledge gets in and how you learn whether it was right.
FutureBeeAI works on the side of that equation nobody can generate: multi-sensor collection in the target environment, specified to the definition the simulator already uses, with the holdout scoped as a deliverable. If you are sizing a hybrid program, talk to the FutureBeeAI team about running the gap classification and anchor specification before the first episode is recorded. If you are earlier than that, browse the library of 2000+ ethically sourced datasets to see how collection specs are written.
A. Measure on hardware, at a site your training data never saw, and score task success rather than loss. In a study of more than 1,500 paired evaluations, careful real-to-sim evaluation predicted real-world performance at around 0.93, while the validation-loss metric most teams use for checkpoint selection managed 0.308. If your only signal is simulation success or loss, you cannot tell a working mixture from one overfitting to the simulator.
A. Published results work with anchor sets of roughly 20 to 50 real demonstrations per task against a thousand or more simulated ones, and the benefit persists even when the real set grows to 400. The count matters less than the alignment: the real episodes need the same task boundaries, comparable camera geometry, and an identical success criterion.
A. It works as a coverage amplifier with modest returns, and never as an evaluator. In one published pipeline, expanding 88 hours of real teleoperation into 827 hours of generated video at roughly 105,000 GPU-hours produced average gains of 8.8 percent in simulation and 5.8 percent on real tasks. Treat it as a third tier under simulation, not a substitute for either input.
Acquiring high-quality AI datasets has never been easier!!!
Get in touch with our AI data expert now!
Share this article on
