Physical AI: The Complete Guide to Data, Models and Deployment in the Real World

Author: Sneha Tanna · Reviewed by: Jesal Thakkar · Last updated: 09/09/2026

Physical AI describes systems that perceive an environment, decide what to do in it and take physical action: the output is movement rather than information. It is also the first category of AI whose training data does not already exist and that single fact governs almost everything else about building one.

Here is what that looks like in practice. A picking policy clears its acceptance threshold in the pilot cell, signed off by everyone in the room. It ships to a second site inside the same customer's network and success falls by twenty points. Nothing was retrained. No code changed. No hardware failed. The shelving is a little taller, the floor is sealed concrete instead of epoxy, the lighting sits at a different angle, and the operators phrase the same instruction differently. That is the normal outcome of a first deployment, not an unlucky one, and no amount of model work will fix it.

This matters more this year than last because the category stopped being speculative while the practice underneath it stayed improvised. Gartner published a dedicated Hype Cycle for Physical AI in 2026, having not mentioned physical AI anywhere in its 2025 AI Hype Cycle. Robotics startups raised $18.8 billion by late June 2026, against $15 billion across all of 2025. The International Federation of Robotics counts 4.66 million industrial robots in operation. Capital and hardware arrived first. The data practice is being invented in public, one deployment at a time.

What follows is an order of operations rather than a tour: what physical AI is, what works today, how these systems learn, where their training data comes from, why they fail after they ship, how that data is sourced responsibly, and what changes the day the model leaves the lab.

What Physical AI Is and What Separates It From Every AI Before It

You almost certainly arrived with a definition. What this section can give you instead is the distinction underneath it, because that distinction is what makes the rest of the problem behave the way it does.

Physical AI is a system that senses the physical world, reasons about what it senses, and acts on it. Cameras, depth sensors, force sensors and microphones on the input side; motors, grippers, wheels and actuators on the output side; a learned policy in between deciding what to do. NVIDIA, which did more than anyone to popularize the term, frames it as autonomous systems that perceive, understand, reason and then perform complex actions in the physical world.

Physical AI compared with generative AI. The difference people reach for is modality, images and text against sensors and motors. The difference that matters is reversibility. A wrong generated paragraph costs you a regeneration. A wrong action has already happened by the time you notice, and the object is already on the floor.

Physical AI compared with traditional robotics. Industrial robots have moved things accurately for fifty years by executing knowledge a person wrote down in advance. Physical AI systems hold knowledge learned from data, which is why they degrade gracefully in situations nobody scripted instead of stopping. That shift from programmed to learned knowledge is the whole argument for why this generation of robotics differs from the last three that promised generality.

Physical AI compared with embodied AI. Used interchangeably. Embodied AI is the older research term; physical AI is the commercial one and usually includes autonomous vehicles nobody would call a robot.

The distinction that earns its place here is what happens inside the loop. Perception, decision and action are not three capabilities you optimize separately and bolt together. A perception error becomes a decision made on a false premise, and that decision becomes a physical action with a consequence that persists. In a language model the failure is contained and the user asks again. In a physical system it propagates forward and lands in the world. Anyone designing training data for one of these systems is designing for the whole chain, not three independent buckets.

Irreversibility also explains something odd about where physical AI works today. The systems succeeding at scale are not the general ones.

Where Physical AI Already Works

Three categories are past the demo stage, and the pattern hiding inside their numbers is the most useful thing on this page.

Autonomous ride-hailing is the clearest case: Waymo has driven more than 170 million fully autonomous miles at roughly four million a week, reporting 92% fewer crashes involving serious or fatal injury than the human benchmark on the same road types, in a comparison that has been through peer review. Warehouse robotics is the largest by unit count, with Amazon running over a million robots across three hundred-plus fulfilment facilities and a 2025 fleet coordination foundation model, trained on its own inventory-movement data, that cut robot travel time by 10%. Industrial robotics has the deepest installed base: 542,000 new installations in 2024 and 4.66 million units operational. Humanoids are the exception that proves the rule, with impressive demonstrations, falling hardware costs, and analyst projections for 2030 volumes spanning from under one million units to more than six million, which is another way of saying nobody knows.

Now the pattern. Every system in the first three categories operates inside a bounded environment. Mapped roads with known signage conventions. A warehouse whose layout the operator controls. A cell with fixed geometry. The boundary is not a failure of ambition, it is what keeps the conditions the system trained on close enough to the conditions it runs in for learned behavior to hold. And the warehouse fleet model gives the game away: the system that improved was trained on data generated in the exact environment it operates in.

If the boundary is a data property rather than a hardware one, the next question is what is doing the learning inside it.

The Models: How a Physical AI System Learns

Two families of the models do the work and what matters is not their architecture in detail but why neither is where your advantage will come from.

Vision-language-action models

A vision-language-action model, or VLA, takes a vision-language backbone and adds action as an output modality. The model inherits semantic knowledge from web-scale image and text pretraining, then learns to emit motor commands conditioned on what it sees and what it was told to do. Google DeepMind's RT-2 named the category in 2023 by showing that web knowledge transfers into robotic control.

The number that matters most for anyone thinking about data is OpenVLA. At seven billion parameters, trained on 970,000 real-world robot demonstrations, it beat a 55-billion-parameter predecessor by 16.5 percentage points of absolute success across 29 tasks, with seven times fewer parameters. Model scale was not the variable. Data composition was.

The standard shape now is two systems running at different speeds: a slower vision-language model reasoning about the goal, paired with a small fast network producing motor commands in real time. Figure's Helix runs a seven-billion-parameter reasoning model at seven to nine hertz over an eighty-million-parameter action model at two hundred hertz.

World models and simulation

A world foundation model simulates how an environment evolves and how an agent's actions change it, which lets policies be trained and evaluated without spending real-world attempts. DeepMind's framing of Genie 3 is worth borrowing: systems that use their understanding of the world to simulate aspects of it, so agents can predict how an environment will evolve and how their actions will affect it. NVIDIA's Cosmos offers the same idea as infrastructure, with a 2026 release handling language, image, video, audio and action in one architecture.

Simulation becomes useful through domain randomization, established in 2017: randomize textures, lighting, object placement and physics so widely that reality looks like one more variation. It works well enough that the original paper trained a real-world detector accurate to 1.5 centimeters using only non-realistic random textures. It does not close the gap. The 2026 peer-reviewed review of the reality gap is direct about why: simulations are abstractions and approximations, so discrepancies are inherent rather than a bug awaiting a fix, and best practice combines randomization, real-to-sim transfer and sim-real co-training rather than relying on any one of them.

Here is the part that should change how you plan. The strongest models in this space are increasingly open and reproducible. Weights are downloadable, architectures are in papers, and two teams can start this quarter from the same policy and the same simulator. What they cannot share is data collected in their own deployment environments, which puts the whole weight of the problem on the one input nobody can download and in physical AI that input is almost always multi-sensor data collected in the target environment.

The Data Problem Nobody Can Buy Their Way Out Of

Every piece of content on physical AI training data tells you data is the bottleneck. That is true, and it is where almost all of them stop, which is unfortunate, because the interesting part is what happens when you reject the unit everyone measures the bottleneck in.

Start with the scale honestly. Ken Goldberg's 2025 assessment in Science Robotics puts the internet-scale text and image data behind contemporary vision-language models on the order of 100,000 years, and the largest reported robot dataset on the order of one year. Internet video adds roughly 35,000 years, but what robots need is video paired with the motion commands that produced it, and that combination does not exist on the internet in any quantity. Epoch AI confirms it from the other direction: the largest manipulation models train with roughly 1% of the compute frontier models used elsewhere, a gap driven by data availability rather than silicon.

Where robot training data actually comes from

There is no single source, and the sources trade off along one axis: the more abundant a source is, the less directly usable its signal.

Internet video sits at the base, effectively unlimited and carrying no depth, force or action labels. Egocentric human video comes next, first-person capture of people performing real tasks, far more plentiful than robot data and carrying genuine human strategy: Ego-Exo4D holds 1,286 hours from 740 participants across 13 cities, pairing first-person and third-person views with gaze, audio, point clouds and IMU. Above that sit capture methods where humans manipulate the actual robot end-effector, producing action labels without a robot in the loop. Then simulation, unlimited and imperfect. Then teleoperation, highest in fidelity and hardest to scale. Then deployment data, the most valuable of all and the only one that arrives after you ship.

The public robot datasets tell you where free data runs out. Open X-Embodiment pooled sixty datasets from thirty-four labs into over a million trajectories across 22 embodiments and 527 skills; DROID added 76,000 trajectories and 350 hours of interaction from fifty collectors across three continents. Neither will train your product.

Why "how much data do I need" has no answer

Three results retire the question. Figure trained Helix on roughly five hundred hours of high-quality supervised data, under 5% of the size of previously collected VLA datasets. Meta's V-JEPA 2, pretrained on internet video, reached 80% zero-shot success on pick-and-place with 62 additional hours of robot data. And Real2Render2Real produced policies matching training on 150 human teleoperation demonstrations, starting from a single one.

Volume behaves badly as a unit because video is byte-dense and signal-sparse. Thirty frames of a gripper approaching a cup carry less usable learning signal than thirty tokens of text. Hours are a purchasing unit, not a learning unit, and a vendor quoting you hours is quoting the thing that is easiest to count.

What replaces the question is composition: which mixture of sources reaches your reliability threshold most cheaply, given your task, your embodiment and your environment. That answer is specific to you, and it moves when any of the three change. One constraint makes it concrete. Skills learned by one robot do not readily transfer to another, because a policy is coupled to the body that produced its data. Cross-embodiment datasets exist as an active research effort precisely because transfer is hard, not because it is solved. Data collected for one body, in one place, under one set of conditions, is not a general asset.

The most common planning error we see is a team deciding its data volume before deciding its data mixture. A number gets committed in a budget cycle, usually expressed in hours, and then the collection design is reverse-engineered to fit it. What that produces is a large single-source dataset, almost always teleoperation in one facility, when the same budget split across egocentric capture in the target environments plus a smaller high-fidelity teleoperation set would have covered more of the deployment distribution. The number is the last decision, not the first.

If composition is specific to an environment, then the environment deserves a much closer look than most program give it.

Why the Same System Fails in a Different Building

There are two gaps between a working model and a working deployment, and conflating them is the most expensive mistake in this field.

The first is the sim-to-real gap, which is well documented, widely discussed and the subject of active research. The second is the collection-to-deployment gap: the difference between the real conditions your data was collected under and the real conditions the system will run in. Almost nobody writes about the second one, and it is the one that breaks pilots.

The misdiagnosis follows a predictable script. Performance drops after deployment, so the team retrains, then tunes the architecture, then buys more data of the same kind from the same collection setup and gets a marginal improvement that does not close the gap. None of it works, because the model did not learn poorly. The deployment environment contains conditions the training data never contained, and more samples from inside your existing coverage cannot fill a hole outside it.

Condition coverage is the frame that makes this auditable. A dataset with a thousand demonstrators performing a task in one clean, well-lit facility with one floor surface is heavily biased, and the bias is environmental rather than demographic. Teams arriving from language and vision AI check representation, find it balanced, sign off, and miss the physical dimension entirely, because their instincts were formed on data that has no lighting condition or floor surface.

Then there is the dimension almost nothing in this category addresses: what happens when a collection spec crosses a border. The task changes, not just its labels. Different objects on the shelf and different packaging geometry to grip. Different surface heights and floor materials. Different hand behavior and grip conventions for the same physical action. Different language and phrasing for the same spoken instruction. A team that treats locale as an annotation setting rather than a task variable will pass its own coverage review and still fail on deployment.

The regulation already agrees. Article 10(4) of the EU AI Act requires that training, validation and testing datasets for high-risk systems account for characteristics particular to the specific geographical, contextual, behavioral or functional setting in which the system will be used. The legal obligation to fit the deployment setting now exists in a form most methodologies have not caught up with.

In our own locale-varied collection work, the moment that consistently surprises clients is not the language difference, it is the object difference. A spec written for a European retail environment lists container types that either do not exist or exist in different proportions on an Indian shelf and the grip strategy a contributor uses is different as a result. The same is true of surface height, packaging rigidity and how densely goods are stacked. We collect speech across in-car, call center and noise-varied conditions in more than a hundred locales, and condition coverage is a practiced discipline here rather than a principle, which is why the object-level divergence is something we design for at spec stage rather than discover in review.

If you are currently scoping collection for a specific deployment, the highest-value hour you can spend is mapping the conditions of your target sites against the conditions of your planned collection setup, before a single episode is recorded.

Once you accept that the data has to be produced in the environments where the system will run, a question arrives that most teams have not been asked to answer: who produced it, and under what terms.

How Physical AI Data Is Sourced Responsibly

This is normally the section a reader skims, because in most content it is a compliance paragraph. Read this one. It decides whether the most expensive asset in your program is still usable when your product ships.

In text and image AI, sourcing questions arrive after the fact, as a licensing dispute over data collected years earlier. Physical AI cannot work that way. This data is produced deliberately, in real places, by identifiable people, often inside homes, factories and workplaces, and frequently while other people are present. Collecting first and resolving consent later is not an option here.

The specific failure is consent routing. Consent from a site owner or an employer is not consent from the person wearing the camera, and neither is consent from the coworker, customer or family member who walks into frame. The Egocentric collection captures bystanders by design, not by accident. A program holding one signature and believing it holds three has a defect it cannot detect by inspecting its data, and it will not surface until someone asks.

Deletion policies do not close this, because inferences learned from data cannot be unlearned: a promise to delete footage within forty-eight hours says nothing about what a model derived from it meanwhile. Nor can you anonymized your way out, since video cannot be stripped of identifying detail without reducing the training value that made it worth collecting.

The regulatory floor is worth stating as dates rather than dread. European guidance on video devices is explicit that consent works as a lawful basis for video only in exceptional cases, because you would need it from every person filmed rather than only your intended subjects, which pushes most collection onto a legitimate-interest assessment with a real necessity test, and applying biometric identification moves that footage into special-category processing requiring explicit consent. India's Digital Personal Data Protection Rules were notified on 14 November 2025, with full data fiduciary obligations landing around May 2027. The EU AI Act's high-risk obligations for AI embedded in regulated products, where machinery and robots sit, now apply from 2 August 2028 after the 2026 postponement. If you are working from the older dates, they have moved.

Now the part that matters commercially. Consented, documented collection is a quality mechanism, not a tax on one. A contributor who understands what is being recorded, why, and where it goes performs the task deliberately and consistently, and the difference shows up in the data. A documented provenance chain, contributor by contributor and session by session, is what lets a dataset survive a legal review two years after collection instead of being quarantined at exactly the moment the product needs it. This is why FutureBeeAI runs contributor obligations through a published Crowd Code of Ethics with enforcement attached rather than a values statement: for physical AI data, provenance is what makes the asset durable.

Durability matters because deployment is not the end of a data program. It is where the most valuable part of it begins.

Deployment: What Changes When the Model Leaves the Lab

Two things make deployment harder than the pilot suggests, and neither of them is the model.

The first is that evaluation is itself a bottleneck. Comparable evaluation of generalist policies does not scale, because existing approaches depend on heavy standardization, either fixing tasks and environments or centralizing everything into a challenge; Rob Arena's answer is distributed real-robot evaluation across seven institutions, which tells you how unresolved the problem is. And the difficulty is easy to underrate: on a long-horizon everyday-activity benchmark, end-to-end visuomotor control scored zero percent on representative tasks, and the best policies transferred to the real world at roughly 22%.

The second is the reliability tail. The unsolved commercial problem in physical AI is not reaching capability, it is moving from around 80% task success to the 99.9% a customer will accept from a system running unattended. Every point in that tail is a distribution shift problem, which is why runtime failure prediction is now a research area of its own: generative policies fail unpredictably in environments they were not trained on.

Safety standards are moving unevenly. ISO 10218-1 and 10218-2 were both revised in 2025, absorbing collaborative-application requirements and adding cybersecurity requirements for the first time, and the EU Machinery Regulation applies from January 2027 with explicit provisions for autonomous mobile machinery and AI systems. Worth knowing honestly: the industrial robot safety standards do not address learned policies. They govern the machine, not the fact that its behavior came from data. Plan around that gap rather than assuming a certification covers it.

All of which reframes the first collection plan. Deployment data is the highest-value source in the mixture and the only one that cannot be acquired in advance, so the initial dataset is not a purchase that ends. It is a decision about how quickly the loop starts turning.

Who Builds What in Physical AI

The market sorts into five layers. Compute and platform supplies accelerators, simulators, world foundation models and reference data pipelines, with NVIDIA at the center of gravity and the most quotable line in the category: compute is data. Robot builders make the hardware and increasingly the policies running on it. Foundation model labs build generalist policies, much of which is published and open. Deployment operators run fleets at commercial scale and are quietly accumulating the most valuable data anyone in this field holds, because it comes from the environment the system runs in.

The data layer covers collection, annotation and evaluation, and it is the hardest layer for a buyer to read. The market norm is to present hours delivered, annotator headcount, locale counts and certification badges as proof of quality, which makes two providers with very different capability look nearly identical on a slide. The question that separates them is narrower: can they design collections around your specific deployment environment, in the locales your system will run in, with provenance documented well enough to survive review. That is the layer FutureBeeAI works in, and it is the question worth asking whoever you talk to.

The Question Worth Asking Instead

The field has spent two years asking what physical AI can do. Every answer to that question carries a hidden qualifier, and the qualifier is where.

Computing is purchasable. Model architectures are published and converging fast enough that any advantage in them has a short life. Simulation is available to everyone on the same terms. Data collected under the conditions of a specific deployment, in the environments a system will actually operate in, with provenance that holds up two years later, is the only part of a physical AI program that cannot be acquired, copied or inherited. Which makes it the part to plan first rather than the part to procure last.

So the more useful question is not what these systems can do. It is where yours will work, and whether you can name the conditions that answer depends on.

If you are scoping a physical AI program now, talk to the FutureBeeAI team about mapping a collection plan against your deployment environments and target locales.

If you are still working out what your requirements are, subscribe to our newsletter for continuing coverage of the physical AI data layer, including the collection methods and sourcing questions this guide only introduces.

Frequently Asked Questions

Q. What sensors and data types does physical AI training data include?

A. Most physical AI datasets combine RGB video, depth or stereo imaging, LiDAR or radar point clouds, inertial measurement data, force and torque readings, joint or actuator states, audio, and the action commands themselves, all time-synchronized to shared timestamps with calibration metadata. Time alignment matters more than most teams expect: if camera and LiDAR streams drift by even a few milliseconds, the system misestimates object position and velocity. Hand tracking, gaze and GPS appear in egocentric and mobile collections.

Q. Can synthetic data alone train a physical AI system?

A. No, though it can carry more of the load than most teams expect. Simulation and world models generate unlimited variation cheaply and are effective for coverage of rare events, and domain randomization transfers surprisingly well. What synthetic data cannot reproduce is sensor artefacts, material behavior and the specific physical quirks of a real deployment site. Current best practice combines synthetic generation with real-world anchoring data from the target environment rather than choosing between them.

Q. How many demonstrations does it take to fine-tune an existing robot policy?

A. Far fewer than to train one from scratch, and the number depends on task complexity more than on any published benchmark. Open generalist policies are designed to be adapted, with published examples fine tuning to new robots in hours on consumer GPUs. One production humanoid policy was trained on roughly five hundred hours of supervised data in total. The practical planning approach is to fix the mixture of data sources first and treat volume as the output of that decision.

Q. Who is in frame when egocentric training data is collected, and whose consent is required?

A. Everyone visible is in frame, which is why consent has to be handled at more than one level. Consent from a site owner or employer covers site access, not the person wearing the camera, and it does not cover coworkers, customers or family members who enter the recording. Under European video-processing guidance, consent is workable as a lawful basis only in exceptional cases for video, so most collection requires a documented necessity assessment plus explicit consent from identifiable participants.

Q. Can an off-the-shelf robotics dataset be used for a commercial deployment?

A. Rarely as the primary training source. Public datasets like Open X-Embodiment and DROID are valuable for pretraining and benchmarking, but their scale is measured in hundreds of hours against internet-scale corpora, and their episodes were collected for other robots in other environments. Skills do not transfer cleanly across embodiments. The realistic use is pretraining or evaluation, with deployment-specific collection supplying the data that determines whether the system works on site.