1 Shanghai Jiao Tong University 2 ShanghaiTech University 3 Beijing Institute for General Artificial Intelligence (BIGAI) 4 Beihang University
Dexterous object rotation requires a robot hand to repeatedly release and remake contacts without dropping the object or stopping the rotation. We introduce DexMani, a demonstration-guided reinforcement learning framework that transfers contact-transition knowledge across hand morphologies. Rather than retargeting human poses, trajectories, or contact identities, DexMani learns how contact-conditioned rotational manipulability changes during successful human rotation. During robot training, each candidate action is evaluated using the target hand's own kinematics and active contacts, and a frozen energy prior provides residual action guidance. We evaluate DexMani on cap unscrewing, free object rotation, faucet turning, cross-hand cap twisting, and closed-loop real-robot deployment.
DexMani transfers contact-transition knowledge without retargeting human joint motion. It first learns an energy prior over contact-conditioned manipulability evolution from human visual-tactile demonstrations. During robot reinforcement learning, each candidate action is interpreted through the robot's own kinematics and active contacts, and the frozen prior supplies an energy-guided residual signal in the native action space.
RGB observations, finger pose, global hand pose, and a tactile pressure array are synchronized during human rotation. These signals identify active contacts without treating human joint trajectories as robot references.
Contact-aware hand kinematics yield a rotational manipulability descriptor at each time step. A contrastive energy model learns which short-horizon changes match successful human contact transitions.
The robot maps candidate actions to changes in its own manipulability. The frozen prior prefers low-energy candidates and supplies a local residual hint, while the policy is optimized only with the task reward.
High-quality robot demonstrations are difficult to obtain for fast, contact-rich rotation, so we collect demonstrations directly from human participants. Synchronized RGB, hand pose, finger pose, and tactile pressure identify active contacts across more than 100,000 frames; human motions are never retargeted as robot actions.
Three multimodal human rotation demonstrations used to learn the contact-conditioned manipulability prior.
A single frozen human-derived prior is reused across three LEAP-hand rotation tasks. DexMani achieves the highest mean success rate in all six seen/unseen settings, including faucet turning, which is held out from human-prior pretraining.
| Method | Unscrew Cap | Rotate Object | Turn Faucet | Avg. | |||
|---|---|---|---|---|---|---|---|
| Seen | Unseen | Seen | Unseen | Seen | Unseen | ||
| PPO | 11.8 ± 2.1 | 5.0 ± 1.9 | 26.5 ± 0.7 | 21.5 ± 0.6 | 33.2 ± 3.5 | 20.5 ± 1.0 | 19.8 |
| VT Pretraining | 29.0 ± 1.5 | 12.7 ± 3.1 | 42.6 ± 1.4 | 34.0 ± 4.6 | 62.4 ± 4.7 | 53.5 ± 4.8 | 39.0 |
| VTM | 47.0 ± 0.9 | 27.8 ± 1.8 | 49.5 ± 3.0 | 37.9 ± 3.4 | 79.0 ± 1.2 | 70.1 ± 3.2 | 51.9 |
| VTA | 40.1 ± 0.8 | 25.1 ± 1.7 | 37.4 ± 5.8 | 20.3 ± 1.8 | 77.2 ± 2.2 | 63.5 ± 7.7 | 43.9 |
| VTA-E | 24.2 ± 2.0 | 11.6 ± 3.6 | 29.0 ± 6.3 | 19.1 ± 3.0 | 67.6 ± 1.9 | 42.0 ± 5.1 | 32.3 |
| DexMani | 65.1 ± 2.4 | 39.3 ± 2.5 | 50.7 ± 2.2 | 39.2 ± 3.1 | 80.2 ± 2.0 | 70.3 ± 2.9 | 57.5 |
Table 1: Guidance reuse across tasks and object generalization. Each task-split value is mean ± standard deviation of success rate (%) across three independent seeds. Avg. SR is the mean of the six task-split means.
DexMani executions across the three LEAP Hand rotation tasks.
LEAP Hand
LEAP Hand
LEAP Hand
The same frozen human-derived prior is reused across the Shadow, Allegro, and XHand platforms. A separate policy is trained in each hand-specific action space.
| Method | Shadow Hand | Allegro Hand | XHand | Avg. SR | |||
|---|---|---|---|---|---|---|---|
| Seen | Unseen | Seen | Unseen | Seen | Unseen | ||
| PPO | 20.8 ± 2.6 | 6.8 ± 2.6 | 11.8 ± 1.6 | 5.2 ± 2.4 | 32.5 ± 3.3 | 17.2 ± 1.6 | 15.7 |
| VTM | 66.4 ± 4.4 | 40.9 ± 9.2 | 24.3 ± 3.9 | 12.5 ± 4.2 | 58.0 ± 1.4 | 22.7 ± 8.4 | 37.5 |
| VTA-E | 52.7 ± 3.2 | 24.5 ± 6.7 | 17.6 ± 3.5 | 9.8 ± 1.7 | 45.2 ± 2.8 | 18.2 ± 7.5 | 28.0 |
| DexMani | 70.0 ± 3.1 | 48.2 ± 4.8 | 34.6 ± 4.0 | 14.8 ± 2.3 | 63.4 ± 4.5 | 29.5 ± 1.9 | 43.4 |
Table 2: Cross-hand cap-unscrewing transfer. Each hand/setting value is mean ± standard deviation of success rate (%) across three seeds. Avg. SR is the mean of the six hand/split means.
Cap-unscrewing executions using the same frozen human-derived prior across three hand embodiments.
Unscrew Cap
Unscrew Cap
Unscrew Cap
The controlled variants keep the same training setup and differ only in their guidance signal. DexMani uses state-aligned, short-horizon manipulability-evolution guidance.
| Method | Unscrew Cap | Turn Faucet | Avg. SR | ||
|---|---|---|---|---|---|
| Seen | Unseen | Seen | Unseen | ||
| Zero Guidance | 18.6 | 9.4 | 48.6 | 20.2 | 24.2 |
| Context-Shuffled | 14.0 | 5.3 | 22.5 | 7.3 | 12.3 |
| Greedy-M | 54.5 | 36.3 | 67.8 | 44.7 | 50.8 |
| DexMani | 65.1 | 39.3 | 80.2 | 70.3 | 63.7 |
Table 3: Mechanism ablation. Values are mean success rate (%). The final manuscript reports means, but not standard deviations, for this controlled ablation.
We measure task-aligned rotational capability with TCI and motion smoothness with log dimensionless jerk (LDLJ) and spectral arc length (SPARC). Higher values indicate better performance for all three metrics.
| Method | Unscrew Cap | Rotate Object | Turn Faucet | ||||||
|---|---|---|---|---|---|---|---|---|---|
| TCI ↑ | LDLJ ↑ | SPARC ↑ | TCI ↑ | LDLJ ↑ | SPARC ↑ | TCI ↑ | LDLJ ↑ | SPARC ↑ | |
| PPO | 0.29 ± 0.11 | -25.6 ± 1.4 | -6.0 ± 0.5 | 0.43 ± 0.11 | -14.5 ± 0.1 | -2.6 ± 0.2 | 0.22 ± 0.05 | -23.2 ± 2.3 | -4.8 ± 0.8 |
| VT Pretrain | 0.41 ± 0.13 | -24.5 ± 1.1 | -5.5 ± 0.6 | 0.41 ± 0.23 | -14.4 ± 0.1 | -2.8 ± 0.2 | 0.23 ± 0.06 | -23.2 ± 2.0 | -4.5 ± 0.9 |
| VTM | 0.44 ± 0.17 | -24.1 ± 0.8 | -5.3 ± 0.7 | 0.26 ± 0.18 | -15.5 ± 0.8 | -3.3 ± 0.3 | 0.27 ± 0.15 | -23.4 ± 1.8 | -4.4 ± 0.4 |
| VTA | 0.45 ± 0.12 | -23.7 ± 1.6 | -5.0 ± 0.1 | 0.29 ± 0.09 | -15.6 ± 0.9 | -3.1 ± 0.2 | 0.23 ± 0.04 | -23.9 ± 2.4 | -4.1 ± 1.2 |
| VTA-E | 0.43 ± 0.05 | -23.6 ± 0.4 | -5.0 ± 0.3 | 0.23 ± 0.10 | -13.9 ± 0.2 | -2.8 ± 0.5 | 0.20 ± 0.02 | -24.8 ± 0.4 | -4.5 ± 0.8 |
| DexMani | 0.50 ± 0.13 | -23.1 ± 0.5 | -4.5 ± 0.4 | 0.36 ± 0.05 | -11.9 ± 1.8 | -2.3 ± 0.7 | 0.28 ± 0.07 | -22.4 ± 0.5 | -3.9 ± 0.1 |
Table 4: Quantitative comparison of execution quality and efficiency. Every entry is reported as mean ± standard deviation; the columns span three dexterous rotation tasks on the LEAP hand.
We deploy DexMani on a physical LEAP Hand with TwinTac tactile sensors. The policy receives binarized tactile input and predicts joint-position targets online.
These experiments establish qualitative closed-loop feasibility under sensing and dynamics mismatch; they are not presented as a new sim-to-real benchmark.
Select a task to view its successful closed-loop execution on the physical LEAP Hand. All actions are displayed at the original speed, at a frequency of 20 Hz.