Paper Appendix
Loading visualization…

DexMimicA Generalist Dexterous Hand–Object Tracking Policy
via Value-Guided Reset Allocation

Zhexi Luo, Zhixuan Xu, Rui Yang, Zeyu Xiao, Yiwen Hou, Lin Shao

National University of Singapore (NUS)

Pipeline

Human hand–object motion guides robot tracking in simulation. The resulting robot motion is then replayed on hardware.

Motion Capture
DexMimic Policy
Deployment

Simulation

Drag the divider to compare MANO and Wuji; select View 3D to explore.

Both hands pour material from a test tube into a beaker. Pour into a Beaker 5.8s · 01 · 30dbe@14 Both hands extinguish an alcohol lamp with its cap. Lamp Extinguishing 2.3s · 02 · 751fb@16 Both hands coordinate while writing on a whiteboard with a marker. Whiteboard Writing 13.2s · 03 · 7a6dd@2 Both hands stir inside a bowl with a spoon. Stirring 6.0s · 04 · 86e2a@11 Both hands scoop salt from a green teacup into a bowl. Scooping 11.3s · 05 · 86e2a@9 The hand writes on a whiteboard with a marker. Writing on a Whiteboard 16.8s · 06 · 9a028@8 The hand scoops material from a gray-green porcelain bottle into a bowl. Scooping into a Bowl 16.0s · 07 · fe8dc@0 The hand spreads ketchup and sugar on bread with a spoon. Spreading on Bread 9.1s · 08 · 920f6@6 The hand writes on paper with a pen. Writing on Paper 15.9s · 09 · 78c62@2 The hand controls a laptop with a mouse. Using a Computer Mouse 5.2s · 10 · f1029@7 The hand lifts the lid from a porcelain teapot. Uncapping a Teapot 5.3s · 11 · 5a065@0 The hand shakes a mixture inside a conical flask. Shaking a Conical Flask 10.2s · 12 · 751fb@24 The hand ignites an alcohol lamp. Lighting an Alcohol Lamp 9.8s · 13 · 81a95@2 The hand moves a blue teacup to a new position. Moving a Blue Teacup 4.3s · 14 · 29266@0 The hand removes a fork from a tool rack. Taking Out a Fork 6.7s · 15 · 07bb1@3 The hand heats a mixed substance in a beaker with an alcohol lamp. Heating a Beaker 4.4s · 16 · 8e5df@11 The hand shakes material inside a test tube. Shaking a Test Tube 8.2s · 17 · 1be0e@1 The hand presses a button on a scale. Pressing a Scale Button 4.2s · 18 · 0b3d1@2 The hand places a test tube into a rack using a tube holder. Racking a Tube with a Holder 4.0s · 19 · b5fa3@15 The hand places a glass rod onto a test-tube rack. Racking a Glass Rod 3.4s · 20 · c2f83@3 The hand removes a test tube from a rack. Removing a Test Tube 5.7s · 21 · 3b1e6@14 The hand places a donut onto a saucer. Placing a Donut on a Saucer 5.9s · 22 · 9f436@1 The hand places a blue teacup on a scale platform. Placing a Cup on a Scale 3.8s · 23 · 86fb0@1 The hand moves a tripod to a new position. Moving a Tripod 4.0s · 24 · f7d37@15 The hand grasps an orange kitchen knife and holds it steady. Holding a Kitchen Knife 7.3s · 25 · 590d0@3 The hand places a roll of tape inside a box. Putting Tape in a Box 3.1s · 26 · 4b10d@1 The hand moves a red striped bottle to a new position. Moving a Striped Bottle 3.7s · 27 · 86fb0@8 The hand moves a toothbrush to a new position. Moving a Toothbrush 6.3s · 28 · f9335@2

Method

Value-Guided Reset Allocation: reset allocation adapts across Early, Middle, and Late training, followed by rollouts and feedback updates.

Key insight. The bottleneck in training a generalist dexterous policy is collecting useful interaction experience. DexMimic shifts the focus beyond reward design: Value-Guided Reset Allocation shapes this experience by adapting the initialization distribution alongside policy learning.

Results

One generalist policy achieves higher tracking success and lower average training cost than trajectory-specific methods. Scaling co-training strengthens transfer to unseen motions and objects.

50 challenging bimanual trajectories from OakInk-V2 and HOT3D. Time is the average training or optimization time per trajectory; DexMimic's joint-training time is divided by 50. All methods share the evaluation initial states, rollout counts and success criteria. Green cells indicate the best value in each column.

Scaling & Generalization

More training trajectories improve zero-shot transfer. Hover over a bar to see its exact success rate.

L1 · Similar motionsL2 · Unseen motionsL3 · Unseen objects

L1: similar trajectories with seen objects. L2: unseen trajectories with seen objects. L3: trajectories with unseen objects. Policies are frozen during evaluation.

Fast Adaptation

Fine-tuning compared with training from scratch on an unseen trajectory.

Fine-tuningFrom scratch
5.3×faster to reach 80% Training SR
0.24 h vs. 1.27 h

Realworld

Real-robot clips are trimmed and time-scaled to match the simulation.

Mouse manipulation

Spoon manipulation

Roll manipulation

Beaker shaking

Test-tube pouring

Test-tube shaking

Flask shaking

Introduction

Achieving human-like dexterous manipulation remains a central challenge in robotics. Learning from human hand–object interaction (HOI) datasets offers a promising pathway to diverse dexterous skills, yet existing physics-based approaches train specialist policies for individual trajectories, thereby limiting skill reuse at scale. We introduce DexMimic, a generalist dexterous hand–object tracking policy jointly trained across diverse HOI trajectories. Efficient joint training requires allocating each rollout to both an undertrained phase and a starting state that supports continued tracking. DexMimic realizes this insight through Value-Guided Reset Allocation, which adapts the initialization distribution online to prioritize difficult phases and select reset states with higher expected tracking returns.

DexMimic paper teaser showing one policy tracking multiple trajectories, tracking success and training efficiency comparisons, scaling results, and eight real-world dexterous skills.

Loading...