Overview
Go-Fetch computes risk-aware trajectories by combining reachability analysis with a Normalized 3D Gaussian Splat.
This combination allows Go-Fetch to constrain the probability of collision between the robot’s reachable set and the scene.
System Pipeline
The pipeline takes a scene and a target object as input and produces a safe, executable trajectory on hardware. It runs end to end through five stages:
1. Scene Understanding A SAGA GUI lets the user select a target object in the scene. SAM v2 handles segmentation, and the environment is represented as a Normalized Gaussian Splat (NGS), a compact, differentiable representation that plays nicely with downstream optimization.
2. Grasp Generation AnyGrasp generates candidate grasp poses for the selected object. These feed directly into the IK stage as target end-effector configurations.
3. Inverse Kinematics with Collision Avoidance IPOPT solves for joint configurations that reach the target grasp pose while respecting NGS-based collision constraints. The robot is approximated as a set of spheres, and the NGS occupancy field provides differentiable collision gradients that IPOPT can use directly.
4. MPC Trajectory Optimization SPLANNING's MPC optimizes a trajectory over an 8-step receding horizon. At each replanning step, the planner looks ahead 8 configurations and optimizes jointly for reaching the goal while staying collision-free.
The Green Color Boxes represent the Intermediary Goal Way Points
The Purple Color Sphere packing is the Predicted 11DOF Joint Reachability Set created for the next 8 Steps(Horizon)
5. Execution The planned trajectory is executed in simulation (Habitat) and on physical Fetch hardware. Both pipelines share the same planner, validating sim-to-real consistency.
Braking Maneuver
If the planner was unable to find a valid trajectory for the next 0.5 seconds, Then the bot immediately implements breaking feature which means the velocities are zeroed out but not all of a sudden but interpolated to ensure smooth behaviour.
When the Spheres turn Red, the Breaking Maneuver Kicks in.
Results
The system achieves 82.5% IK success rate across tested configurations, validated on physical Fetch hardware. Motion planning has been tested on hardware with the full perception-to-execution pipeline running end to end.
