Sim-to-Real Transfer: The Hardest Problem in Robotics That Nobody Outside the Field Talks About

There is a moment in almost every robotics demo that the people running it dread: when the robot, which performed perfectly in the laboratory, is put in the real world and fails immediately.

The robot that navigated a clean, flat, well-lit simulated environment stumbles on an actual floor. The arm that picked up thousands of simulated objects drops the real one. The humanoid that walked perfectly in simulation walks for three steps and falls over.

This is the sim-to-real transfer problem. It is the central unsolved challenge in practical robotics deployment, and it is the reason that NVIDIA, Google DeepMind, and Disney Research jointly built an open-source physics engine called Newton to address it.

What the problem actually is

Robots today are primarily trained using reinforcement learning in simulation. The process works like this: you build a physics simulation of the robot and its environment. You define a reward function that gives the robot points for doing what you want (moving forward, picking up an object, maintaining balance) and penalties for failing (falling, dropping things, colliding). You run millions of training iterations inside the simulation. The robot learns a policy — a mapping from states to actions — that maximises its reward.

This approach has produced remarkable results. The Unitree robots that performed autonomous kung fu at China's Spring Festival Gala in February 2026 were trained this way. The Boston Dynamics Atlas that runs, jumps, and does backflips was trained this way. Industrial robot arms that can pick and place thousands of different objects were trained this way.

The problem is the simulation. Simulations are abstractions of reality. They model physics mathematically, and all mathematical models of physical systems are approximations. The approximations are good — good enough that robots trained in simulation can do impressive things. But they are not perfect, and the imperfections matter.

The specific ways simulation fails to match reality

Friction: Every surface has friction, and friction depends on material properties, surface texture, temperature, and lubrication in ways that are extremely difficult to model accurately. A robot trained on simulated friction that is 10% too low will apply slightly too much force when grasping objects. That might not cause visible failure in most cases, but it causes systematic errors that accumulate.

Contact dynamics: When a robot's foot hits the ground, there is a complex interaction involving deformation of both surfaces, energy absorption, and micro-vibrations. Simulations model this as a simplified contact force. The gap between the simplified model and the real physics is small in absolute terms and enormous in practical impact — it is why bipedal robots trained in simulation often fall over immediately when deployed on real ground.

Motor dynamics: Simulated motors respond exactly as commanded. Real motors have backlash (mechanical play in gears), electrical noise, heating effects, and imprecision in sensors. A policy trained with perfect simulated motors will not work well with real imperfect motors.

Sensor noise: Real cameras, IMUs, and force sensors have noise — random variation in measurements. Simulations often use noiseless or simple noise models. When a robot trained on clean simulated sensor data encounters real sensor noise, its decision-making breaks down.

Dynamics gaps: Some physical phenomena are simply not modelled in most simulations because they are too computationally expensive or too difficult to parameterise. Flexible cables, deformable objects, liquids, powder, and granular materials all exhibit complex physical behaviour that standard rigid-body simulations ignore. Real robots encounter all of these.

The techniques researchers use to bridge the gap

The field has developed several approaches to the sim-to-real problem. None of them fully solves it, but together they have enabled the dramatic progress in real-world robot deployment of the last three years.

Domain randomisation is the most widely used technique. Instead of training the robot in one specific simulation, you randomise the parameters — friction coefficients, motor response curves, sensor noise levels, lighting conditions, object masses — across a wide range during training. The robot learns a policy that works across all of these variations, which means it is more likely to work in the real world (which is just another point in the distribution of possible physics parameters). Domain randomisation is why simulated training works at all for many real-world tasks, and it is computationally expensive — you need to run many more training iterations across the parameter space.

System identification is the process of carefully measuring the real robot's physical parameters and building a simulation that matches them as closely as possible. This reduces the sim-to-real gap by making the simulation more accurate. The limitation is that system identification requires significant engineering effort for each robot, does not generalise easily, and still cannot capture all of the real-world phenomena that matter.

Sim-to-real fine-tuning starts with a policy trained in simulation and then continues training with a small amount of real-world data. The idea is that the simulation gets the robot most of the way there, and a small amount of real experience closes the remaining gap. This works well when real data collection is relatively safe and inexpensive, which is increasingly true as robots become more robust.

Better simulations are the approach that Newton takes. Instead of working around an imprecise simulation, build a more precise one. Differentiable physics — where gradients propagate through the physics calculations — enables more sophisticated learning algorithms. Better contact models reduce the friction and impact errors. GPU acceleration makes it feasible to run high-fidelity simulations fast enough to use in training.

What Newton brings that existing simulators did not

MuJoCo has been the gold standard physics simulator for robotics research for over a decade. It is fast, relatively accurate for rigid body dynamics, and has a large ecosystem of research built on it. But it runs primarily on CPUs, which limits training throughput.

Newton, co-developed by NVIDIA, Google DeepMind, and Disney Research, changes this in several ways:

The GPU acceleration via NVIDIA Warp makes simulation dramatically faster — 70x compared to MuJoCo's existing GPU-accelerated version (MJX). Faster simulation means more training iterations in the same time, which means better-learned policies.

Differentiable physics means gradients flow through the simulation. This enables gradient-based optimisation directly through the physics, rather than only through the neural network that generates robot actions. The result is more sample-efficient learning — the robot needs fewer training iterations to learn good behaviour.

Better multiphysics support enables simulation of flexible objects, cables, and soft bodies — exactly the kinds of physical phenomena that existing simulators handle poorly and that cause real-world failures.

Why Disney's contribution matters

Disney Research's involvement in Newton is not cosmetic. They are solving the hardest version of the real-world deployment problem: a robot that must operate in a crowd of humans, in real time, in an environment it has never seen before, while appearing to be emotionally engaging.

The BDX droids — the Star Wars-inspired rolling robots being deployed at Disney theme parks — operate in conditions that a factory robot or a warehouse robot never encounters. Unpredictable human movement. Children running. Uneven surfaces. Variable lighting. The requirement not just to function but to appear to have personality and intent.

Training a robot for those conditions in simulation requires a simulator that can model crowd dynamics, human movement patterns, surface variations, and the physical interactions between a small wheeled robot and an unpredictable environment. Disney Research's contribution to Newton is the experience of building those simulations and the requirements that emerged from deploying robots in real theme parks.

The open-source decision

Newton's donation to the Linux Foundation means any developer or researcher can use it. This is significant because high-quality physics simulators have historically been either expensive commercial products or academic tools with limited support.

For robotics developers, Newton's open-source availability at this quality level is comparable to when PyTorch became open source for AI research — it removes a significant barrier and makes frontier-quality tools available to everyone.

The sim-to-real problem is not going to be fully solved by Newton. It is too deep a problem for any single tool to resolve completely. But Newton represents a meaningful advance in the quality of simulation available to the entire field, and advances in simulation quality historically translate quickly into advances in real-world robot capability.

What this means going forward

The GTC 2026 conference on March 16 will include updates on Newton and NVIDIA's Isaac GR00T robotics stack. The physical AI track is one of the most anticipated parts of the event.

What to watch for: which companies are using Newton in production, what improvements have been made since the GTC 2025 announcement, and whether the sim-to-real gap for specific tasks (manipulation, bipedal locomotion, human-robot interaction) has been measurably closed.

The robots performing at the Spring Festival Gala were trained in simulation. The BDX droids being deployed at Disney parks were trained in simulation. The industrial robots being deployed by Unitree, Agility, and Figure are trained in simulation. The quality of that simulation is the current limiting factor for what those robots can do.

Newton is an attempt to raise that ceiling. How high it raises it will become clear over the next 18 months.