Machine learning eats attempts. A language model can read the internet; a robot learning by trial in the physical world gets one attempt per attempt, each costing seconds, supervision and occasionally a broken wrist joint. Simulation removes the ceiling: thousands of virtual robots failing in parallel, faster than real time, at the price of electricity. Most of modern robot learning happens there - which makes the journey back to reality the field’s central engineering problem.
The gap, precisely
A simulator is a physics engine, and physics engines are honest about some things and lying conveniences about others. Rigid bodies, gravity and collisions are cheap and faithful. The places robots actually live are not: contact - the friction, compliance and micro-slips of a gripper meeting an object - is notoriously mis-modelled; deformables like cloth, cables and food barely simulate at all at useful speed; sensors return clean geometry where real cameras return glare, motion blur and a hand in front of the lens; and every motor in simulation responds identically, while every motor in a warehouse has its own wear, backlash and mood. A policy trained purely in the clean world learns to exploit its conveniences - and exploits are exactly what fail to transfer.
The tricks that cross it
The field’s standard answer is deliberately counterintuitive: make the simulator worse, everywhere, at random. Domain randomisation varies friction, masses, lighting, textures, latencies and camera positions across millions of training runs, so the policy never meets the same world twice and cannot overfit to any of them - reality then arrives as just one more variation. The canonical demonstration is OpenAI’s 2019 dexterous-hand work - a five-fingered robot hand trained to manipulate objects entirely in randomised simulation, then run on physical hardware - and the technique has been standard equipment across the field since. Around it sit the supporting cast: system identification, which tunes the simulator to measurements of the specific robot; real-world fine-tuning on top of sim-trained policies; and residual learning, where simulation provides the competent baseline and a small learned correction absorbs what physics engines cannot say. None closes the gap; together they make it jumpable for a growing class of tasks.
What this does to evidence
For a verification-minded reader, sim-to-real creates a specific reporting hazard: simulation results photograph beautifully and prove little. A montage of a thousand virtual robots is genuinely informative about training scale and nothing else; success rates quoted without the word “real” attached should be assumed simulated; and even honest real-hardware numbers deserve the follow-ups this desk applies to any demo - how many trials, on which objects, with what variation from the training distribution. The strongest results in the field state all of it plainly: trained in simulation at such-and-such scale, transferred, evaluated over N real-world trials across M object categories, success rate X with confidence bounds. That sentence pattern exists in the literature. Its absence in a press release is a choice.
The honest summary
Simulation is not a shortcut around reality; it is a lever on it - the only known way to buy robot experience at internet prices. The gap is real, the bridging techniques are real, and the distance between them is measured, task by task, in exactly one currency: performance on physical hardware, counted in public. Everything else is rendering.
Simulation results photograph beautifully and prove little. The only currency that spends is performance on hardware, counted in public.The Robotics Desk
FIELD NOTES - READING A SIM-TRAINED RESULT
- Find the word “real” - success rates without it should be assumed simulated.
- Count the trials - N, object categories, and distance from the training distribution.
- Ask what was randomised - the domain-randomisation list is the honest disclosure of what the team feared.
- Look for the sentence pattern - trained at scale X, transferred, N real trials, success Y with bounds. It exists in the literature; its absence in a press release is a choice.
