// the find
Unity-Technologies/Robotics-Object-Pose-Estimation
A complete end-to-end demonstration in which we collect training data in Unity and use that data to train a deep neural network to predict the pose of a cube. This model is then deployed in a simulated robotic pick-and-place task.
A Unity-based tutorial showing the full synthetic-data pipeline for 6-DoF object pose estimation: generate labeled training images in Unity, train a deep neural network, then deploy it to control a UR3 arm via ROS and MoveIt. Aimed at robotics researchers and ML engineers who want a concrete end-to-end example of sim-to-real transfer — but in simulation only, not actual hardware.
The pipeline is genuinely end-to-end: Unity Perception handles domain randomization and label export, the model trains on that synthetic data, and the inference result feeds directly back into robot motion planning — no hand-waving between steps. The domain randomization setup (light, pose, texture randomizers) is well-structured and each randomizer is its own composable component, which is the right abstraction. The 89% pick-and-place success rate is a real number from 100 actual trials, not a cherry-picked demo. Kubeflow pipeline support is included for teams that need to scale training beyond a laptop.
Last commit was April 2022 — this is effectively archived. It targets Unity 2020.2, ROS Noetic (EOL January 2025), and Python 3.8-era dependencies; getting any of this to run today requires non-trivial environment archaeology. The task is also pathologically simple: one cube, one texture configuration, one robot, one pick-and-place motion — the 54% success rate under occlusion shows the model barely generalizes even within simulation. There is no path shown for transferring to real hardware (no sim-to-real gap discussion, no calibration, no real robot results). The GCS-dependent storage module is baked in and not optional, so local-only users hit dead imports.