DexPort: How a Handheld Skill Interface Solves Dexterous Hand Data Collection Challenges

Dexterous hand manipulation has long been one of the most challenging areas in robot learning. The scarcity of high-quality demonstration data has become the core bottleneck constraining progress in this field. Traditional teleoperation can record executable robot actions, but it requires a complete robot hardware setup, and the operator must simultaneously coordinate multiple finger joints—an extremely heavy burden. Human demonstration collection has a lower barrier to entry, but faces a complex pipeline of motion estimation and retargeting, and cannot directly provide robot actions. The recently proposed DexPort attempts to find a new path between the two: using a handheld skill interface that lets operators collect data for high-DOF robot hands as naturally as holding a tool.

According to the paper accepted at RSS, DexPort is a portable data collection interface for high-DOF robot hands. Its core design philosophy is "using the same hand hardware for both collection and deployment," thereby eliminating the hand embodiment gap and simultaneously recording commanded joint targets and measured joint states.

The Dilemma of Dexterous Hand Data Collection

Teleoperation and human demonstration each have their shortcomings. Teleoperation requires the operator to control the slave robot hand in real time through a master hand or motion capture device; the actions can be used directly for policy learning, but the system is complex and costly, and multi-finger coordination is extremely unfriendly to operators. Human demonstration typically uses vision or motion capture to collect human hand motions, then maps them to the robot hand through retargeting—a process that not only introduces mapping errors but also loses action information at the robot execution level.

DexPort's starting point is straightforward: can we let operators directly manipulate the robot hand itself without needing to control each joint individually? The observation presented in the paper is that a small set of skills is sufficient to support a wide range of dexterous tasks. This insight forms the cornerstone of the entire system design.

Embodiment gap refers to the physical structure and kinematic differences between the embodiment used during demonstration collection (such as a human hand or master-hand device) and the target robot embodiment during deployment (such as a high-DOF robot hand). In human demonstration, this gap is particularly pronounced: a human hand has roughly 20-plus DOF, while most robot hands differ in DOF count, joint coupling, and link dimensions. Therefore, human hand motions must be mapped to the robot joint space through retargeting, which itself introduces errors and loses execution-level information such as contact forces and joint limits. DexPort fundamentally eliminates this gap by sharing the same robot hand hardware for both collection and deployment, so the recorded commands can be used directly for policy learning without any cross-embodiment mapping.

How the Skill Interface Works

During collection, the operator holds the DexPort interface and moves it to specify the robot wrist trajectory; finger motions are coordinated by selecting or adjusting preset skills. This approach eliminates the burden of controlling each joint independently while preserving behaviors such as precise grasping and in-hand manipulation.

From a data perspective, DexPort records the commanded joint targets actually received by the robot hand and the joint states fed back by sensors. According to the paper, using the recorded robot commands rather than states improves task performance from 11% to 64%; and using in-the-wild data raises task success rates from 36% to 78%. These results demonstrate two points: command signals are more suitable for behavior cloning than state signals; and data collected in unstructured environments has significant value.

Behavior cloning is the most fundamental method in imitation learning: it treats observation-action pairs from demonstration data as supervision signals to train a policy network that maps observations directly to actions, without interacting with the environment or designing reward functions. Its key prerequisite is the quality of "action labels"—the labels must be strictly consistent with the action space output by the policy during deployment. This is precisely why DexPort emphasizes recording "commanded joint targets" rather than "measured joint states": commanded targets reflect the desired actions actually received by the controller and align semantically with the policy's output during deployment; whereas measured states are contaminated by execution latency, compliant control, and sensor noise, and using them as labels introduces systematic bias, causing the policy to learn "outcomes" rather than "intentions."

In-hand manipulation refers to adjusting object pose within the palm using only finger motion—such as rotating, translating, or flipping—without relying on the arm or external support. It is the most signature capability of dexterous hands and also the hardest part to cover in data collection: in-hand manipulation involves rapid switching of multi-finger contacts and dynamic adjustment of force closure, and operators can hardly coordinate so many DOF simultaneously during per-joint teleoperation. DexPort uses preset skills to handle finger-level coordination, with the operator only providing global pose context through wrist motion—thereby preserving in-hand manipulation behaviors without per-joint control. This is the core value of its skill abstraction design.

Real Tasks and Training Validation

The research team collected demonstration data across multiple real tasks and unstructured environments without a robot arm. They then directly trained visuomotor policies with behavior cloning, using the recorded wrist motion and hand commands as input, without using skill labels during training.

Evaluation covered four task types: pick-and-place with diverse objects, screwing with an in-hand tool, drawer storage requiring simultaneous grasping and pulling, and a dual-arm scanning task. The team collected over 2,500 demonstrations in both laboratory and in-the-wild settings using the portable device. These tasks impose different requirements on grasp precision, in-hand manipulation, and dual-arm coordination, and also validate the generality of the skill interface.

Takeaways Worth Noting

DexPort's significance goes beyond being a data collection device. It proposes an approach: rather than pursuing full-DOF teleoperation or complex human hand retargeting, it is better to use skill-level abstraction to reduce operator burden while preserving the true action space of the robot hand hardware. Sharing hardware between collection and deployment avoids retargeting errors; skill selection replaces per-joint control, lowering the operator barrier; and prioritizing command signals over state signals provides cleaner supervision for behavior cloning.

This approach also has limitations. Whether the skill set covers enough tasks and how skill-switching granularity affects policy learning have not yet been discussed in the paper. Additionally, while 2,500 demonstrations is substantial for real-world data, cross-task generalization still requires further validation.

For researchers and engineers focused on data infrastructure for robot learning, DexPort offers a direction worth following: making the data collection interface simple enough without sacrificing the authenticity of robot actions.

One-line Summary

DexPort uses a handheld skill interface to collect dexterous hand data, eliminating the embodiment gap and reducing operator burden.

Paragraph Summary

DexPort is a portable data collection interface for high-DOF robot hands, designed to break the bottleneck of scarce dexterous hand demonstration data. It lets operators hold the interface like a tool, use wrist motion to specify global trajectories, and select preset skills with their fingers to accomplish grasping and in-hand manipulation—eliminating the burden of per-joint coordination. Collection and deployment share the same hand hardware, eliminating the hand embodiment gap and simultaneously recording commanded joint targets and measured joint states. Experiments show that using commands rather than states as supervision signals improves task performance from 11% to 64%, and using in-the-wild data raises success rates from 36% to 78%. The team collected over 2,500 demonstrations across four real-world task types and validated the approach by training visuomotor policies with behavior cloning.