// the find
sahilrajpurkar03/nlp-pnp-robotic-arm
SPARC is an intelligent ROS2 Humble-based pick-and-place robotic system integrating MoveIt, YOLOv8-OBB, and Isaac Sim. It enables real-time object detection, motion planning for Franka Panda and UR5 arms, and natural language control via an Ollama-powered chatbot—making robotic manipulation smart, interactive, and human-friendly.
SPARC is a ROS2 Humble project that wires together YOLOv8-OBB object detection, MoveIt motion planning for UR5/Franka Panda, and an Ollama-backed chatbot so you can issue plain-English pick-and-place commands to a simulated robot arm in Isaac Sim. It's aimed at robotics students and researchers who want a starting point for perception-to-manipulation pipelines with an NLP control layer.
- The stack integration is genuinely interesting: custom ROS2 message types (yolov8_obb_msgs) bridge the YOLO detection output to MoveIt goal poses, which is non-trivial plumbing that saves real setup time.
- Training data and OBB labels are included in-repo, so you can actually reproduce or fine-tune the detection model without hunting for the dataset separately.
- Both a Python moveit_py path and a C++ MoveIt path (ur5_pick_place_cpp_r/s) are provided, giving readers a concrete comparison of the two MoveIt APIs.
- The chatbot architecture is sensibly split: llm_mapper.py handles intent extraction, cmd_bridge.py handles ROS2 publishing, and target_publisher.py is its own node—reasonable separation of concerns for a student project.
- The YOLO model trains on only 40 images of what appear to be Isaac Sim renders of a single object class. Generalisation to anything other than that exact simulated scene is basically zero, and there's no pretrained weights file or download link in the repo.
- Setup is a maze of manual steps across three different environment contexts (system ROS2, conda env, Isaac Sim Python), with hardcoded paths like ~/Team7_SPARC scattered in launch scripts—the repo directory and that path don't even match, which will silently break the chatbot launch for anyone who clones it.
- There is a stray file literally named 'h origin master --force' committed to the root of the repo, which is a pushed git command. That alone signals the repo hasn't been reviewed for production quality.
- No information is provided about which Ollama model to pull, what prompt template llm_mapper.py uses, or how robust the NL→command mapping is to anything other than happy-path inputs. The chatbot is effectively a black box from the README.