finds.dev

simpler-env/SimplerEnv

Evaluating and reproducing real-world robot manipulation policies (e.g., RT-1, RT-1-X, Octo) in simulation under common setups (e.g., Google Robot, WidowX+Bridge) (CoRL 2024)

View on GitHubHomepage ↗

★ 1,169Jupyter NotebookMITupdated Dec 2025

// the take

SimplerEnv is a simulation benchmark for evaluating generalist robot manipulation policies (RT-1, RT-1-X, Octo) against real Google Robot and WidowX+Bridge setups, so you can rank checkpoints without burning real robot time. It's built for robot learning researchers who need reproducible, cheap policy evaluation as a proxy for real-world rollouts.

It ships two distinct evaluation methodologies (visual matching via real-image overlay, and variant aggregation across backgrounds/lighting/distractors) plus published metrics (MMRV, Pearson correlation) that quantify how well the sim results track real robot performance, not just a black-box simulator. It comes with real-world performance numbers for RT-1, RT-1-X, and Octo baked in, so you can directly compare a new evaluation approach against ground truth using the provided REAL_PERF data and metrics.py. The API is a one-liner (simpler_env.make(...)) with prepackaged environments and a Colab notebook, so you get usable pixels and rewards before touching SAPIEN or ManiSkill internals. The ManiSkill3 branch adds GPU parallelization for the Bridge environments at a claimed 10-15x speedup over the CPU-based ManiSkill2 path.

The full install is a multi-hour ordeal: CUDA pinned to 11.8-13, numpy<2.0 to avoid IK breakage in pinocchio, a specific git submodule commit for Octo, and separate tensorflow/jax stacks depending on whether you're running RT-1 or Octo. SAPIEN's Vulkan rendering is flaky enough to warrant its own troubleshooting section, and non-RTX GPUs (1080Ti, A100) are explicitly called out as slow for ray-traced environments. Coverage is narrow out of the box — 4 robot embodiments, 10 base tasks — and adding a new robot or environment means system identification and asset creation work spelled out in a separate README, not a same-day task. The GPU-parallelized ManiSkill3 path lives on a different branch entirely, so the main branch you land on is already partially legacy and the docs split attention between two simulator generations.

We read the README, the directory tree and a key source file in September 2026. Nobody pays to be listed. How we judge →

// at a glance

Stars
1,169
Forks
201
Language
Jupyter Notebook
License
MIT
Last commit
20 Dec 2025

// nearby

If SimplerEnv isn't quite it.

All repos →