The cube clears.
The arm collides.
Four gate trials ended when link5 touched a post during lowering.
15.03–44.00 Nmeasured force / 0.05 N limit
OPEN EXPERIMENTS / MUJOCO × JEV
From state
to motion.
One robot. Five physical tasks.
A closer look at what language-driven control can do — and where it fails.
Measured simulator state → intent → action
SIMULATION OBSERVATORYFRANKA PANDA
CONTACT. CONTROL. CORRECTION.Actual evaluated scene · gate pick & place
05physical tasks
100evaluated episodes
10seeds per task & policy
Two independent policies.
Every completed failure counted.
01 / EXPLORE THE LAB
Watch a decision
become an action.
Pick a task, then watch the original demonstration. Every movement comes from JEV decisions and physical contact.
No natural failure recorded.
Simulation-time playback; API waits are omitted. New task videos are captured from the formal evaluation and end with a labeled two-second still. Original task videos use separate demonstration seeds.
02 / MEASURED, NOT ASSUMED
The results.
Including the misses.
Ten fixed seeds for each task and policy. Shared physics and success checks. Independent rule baseline. Two frozen campaigns.
Successful trials
JEVRule
Loading results…
Full results & confidence intervals 5 tasks / 100 episodes
| Task | JEV | Rule | JEV Wilson 95% |
|---|
Loading separate spatial-input campaign status…
Ten trials are a small sample. Even 10/10 does not guarantee future success. These results describe the evaluated scenes, not general-purpose manipulation.
03 / UNDERSTANDING FAILURE
A failure is a result.
Seven failed trials in the new 40-episode campaign. Their original outcomes, measurements and boundaries are preserved.
The cube clears.
A wrong turn,
repeated.
Two JEV trials requested +Y while the measured grasp direction was −Y. The object was never grasped.
302 / 305rejected actions / 350-decision budget
An inconsistent
model choice.
One JEV response selected an option below the maximum probability. That action was not executed.
0.49 < 0.50selected probability / maximum
Inspect all seven failed trials JEV + rule baseline
| Task / policy | Seed | Decision | Evidence & boundary |
|---|
04 / SPATIAL INPUT COMPARISON
Two gates.
Two views of the scene.
The original input remains the default. Full geometry adds simulator-only robot and obstacle measurements as a diagnostic comparison.
| Task | Input | Success / 10 | Task failure | Response validation | Unresolved API |
|---|---|---|---|---|---|
| Single gate | Original (default) | 3 | 4 | 3 | 0 |
| Single gate | Full geometry | 2 | 4 | 4 | 0 |
| Staggered double gate | Original (default) | 0 | 10 | 0 | 0 |
| Staggered double gate | Full geometry | 0 | 7 | 3 | 0 |
No success-rate improvement observed. September 27 campaign: 40 fixed trials, separate from the five-task 100-episode evaluation. Single gate: 3/10 original vs 2/10 full geometry; double gate: 0/10 each. Ten paired seeds per task; exact McNemar two-sided p=1.00 for both, with insufficient evidence of equivalence or harm.
40/40 traces verified offline with zero network requests. All 113 transport errors recovered with physics paused; 38 pre-recharge HTTP 402 account blocks retained separately. Task failures comprise 23 collisions and 2 exhausted decision budgets. The 10 response-validation failures remain failures. Six original-trajectory recordings follow; API waits are omitted.
Failure · seed 4
Failure · seed 0
No successful trial in this group’s ten fixed seeds.
No successful trial in this group’s ten fixed seeds.
05 / INSIDE THE LOOP
State. Intent. Motion.
JEV receives structured simulator measurements. Physical success is checked independently of what the model says.
Read the state.
Object poses, tool position, finger contacts and geometric relationships.
Choose the next step.
JEV selects a task-specific intent: approach, grasp, carry, lower or release.
Choose the motion.
A second JEV decision selects X, Y, Z directions and a gripper command.
Let physics respond.
Cartesian control, real contact, measured feedback and independent evaluation.
Structured state, no camera input. No rule fallback for JEV. Evaluated in MuJoCo; real-robot generalization remains untested.
06 / OPEN & REPRODUCIBLE
Your turn at the controls.
Python 3.11 · MuJoCo · Franka Panda
Run the rule baseline locally, or bring a TypeSafe key for JEV.
START AN EXPERIMENT
# Loose-fit insertion
robojev --task peg_insert --policy rule
# Language-driven obstacle transport
robojev --task obstacle_pick_place --policy jev