Abstract
This project presents a real-time teleoperation pipeline that enables intuitive control of a UR10 industrial robot using the SO-101 leader arm. The system achieves low-latency Cartesian pose control by combining exponential filtering, velocity clamping, one-step inverse kinematics with Pinocchio, and IPOPT-based constrained optimization for safe execution. The pipeline provides an intuitive way to collect high-quality demonstration data, a foundation for learning-based manipulation policies, and supports both local and remote collaboration.
Introduction
Industrial robots like the UR10 excel in precision but lack intuitive human-like control. Meanwhile, leader devices such as the SO-101 provide natural human input but have different kinematics and joint structures. This project bridges the gap by building a teleoperation system where motions of the SO-101 directly control the UR10 robot in real-time. The result is a robust and intuitive pipeline suitable for demonstration collection and advanced AI research in robot learning.
Motivation
Long-Term Vision: Build robots that can autonomously pick and place objects from cluttered environments.
Why Teleop?: Teleoperation enables us to collect expert demonstration data, a key ingredient for AI training.
Why Cartesian Control?: Cartesian teleop is more generalizable than joint-space control, especially when leader and follower devices differ in kinematics.
SO101 Leader Arm
Desktop-mounted, 6-DoF arm that replicates human motion.
Streams accurate 6DoF pose data in real time.
Used as a leader device to capture and stream end-effector pose.
Critical for intuitive teleoperation and demonstration collection.
Why Cartesian Teleop?
The Problem: The SO-101 has different DoFs, link lengths, and joint limits than the UR10.
Joint Space Limitations: Requires matched joint structures, making it non-generalizable. Demonstrations don’t transfer meaningfully.
Solution: Cartesian control maps poses directly, ensuring generalization across robots and setups.
System Architecture
SO-101 Arm Input – streams 6DoF end-effector pose.
Pose Filtering – exponential smoothing for stability.
Streaming IK (Pinocchio) – real-time one-step inverse kinematics.
IPOPT Optimization – ensures joint limits, smoothness, and constraint satisfaction.
Velocity Clamping – global and per-cycle checks for safe execution.
MoveIt Safety Check – collision avoidance and validity checks.
JointTrajectory Publisher (ROS) – commands UR10 robot joints in real time.
Role of IPOPT Optimization
While Pinocchio provides fast analytical IK solutions, it struggles under constraints like joint limits, singularities, and safety boundaries. To address this, IPOPT was integrated into the pipeline as a nonlinear optimizer:
Inputs: Desired Cartesian pose from SO-101.
Objective: Minimize error between desired and actual end-effector pose.
Constraints:
Joint limits
Velocity/acceleration bounds
Smoothness (minimizing jerk)
Collision avoidance (via MoveIt checks)
Outcome: A stable, constrained joint configuration that guarantees safe execution while retaining responsiveness.
This combination of Pinocchio for speed and IPOPT for robustness provided the best balance of real-time performance and reliability.
Results
Latency: ~70 ms in remote mode, near-zero locally.
Performance: Fastest recorded pick: 5.3 seconds.
Stability: Smooth and repeatable teleoperation sessions.
Intuitiveness: Operators could naturally transfer hand motions to robot actions
Conclusion
The UR10–SO101 teleoperation pipeline demonstrates how combining fast IK (Pinocchio) with robust optimization (IPOPT) can enable intuitive, low-latency control of industrial robots. Beyond immediate success in teleop, this system lays the groundwork for collecting large-scale demonstration datasets, training diffusion-based visuomotor policies, and scaling to remote and collaborative robot learning environments.
At RightHand Robotics HQ 🦾✨
One of the coolest parts of this project was actually getting to work onsite at RightHand Robotics HQ in Boston. Not only did I get to run my teleop pipeline on the real hardware, but I also got to meet the people behind the company — including the CEO!
It was a surreal mix of “I can’t believe my code is driving a giant UR10 in an actual robotics company” and “wow, this is the kind of stuff I used to learn about, and now I’m here doing it.” The visit wasn’t just about debugging robots (though there was plenty of that). I was also competing with myself to beat my best times to pick to teleoperate faster (achieved less than 6 seconds per Pick)
Honestly, this was the moment where the project stopped being just “grad school work” and felt like stepping into the real robotics world.
