Embodied AI has a data problem. Collecting demonstrations on real robots is expensive, slow, and tied to the specific hardware used to record them. Meanwhile, the world is full of videos of humans manipulating objects, and egocentric footage, recorded from head-mounted cameras, shows exactly what a robot would need to learn: how a hand approaches, grasps, moves, and releases an object. Recent vision-language-action work already treats this kind of footage as a pretraining source alongside simulation and real-robot data. [1][2]
There is a catch. A video is a sequence of flat images with no joint angles, no depth, and no action labels. Turning it into something a robot can consume requires a pipeline of models that each solve one piece of the problem. In this piece of our data collection section, we walk through that pipeline step by step: what each stage does, where it breaks, and why treating it as a system, not as a single model, matters for dataset quality.
The pipeline, stage by stage
Hand localization and 3D keypoints
The first step is finding the hand. Classic pipelines crop the frame around it with a bounding box and pass that crop to a backbone network that predicts the hand's structure. The hand is decomposed into 21 keypoints: the wrist plus four points per finger (the fingertip and the three joints along the finger). Each point has a name, and each connects to its neighbor, with the base of every finger linked to the wrist. The thumb is the exception. It has more freedom of movement than the other fingers, so it is modeled differently.
The 2D keypoints come from networks trained on large collections of human-labeled images. To lift them into 3D, the model relies on a strong prior: the hand is a known rigid structure with constrained joints. A finger joint can bend in one direction but not the other. This prior is formalized in parametric models such as MANO, which describe the hand through a small set of pose and shape parameters. [3] Current models go further. HaMeR scales a Vision Transformer to regress MANO parameters from hand crops, and WiLoR integrates hand localization and 3D reconstruction so it can handle incomplete hands, multiple hands, and cluttered scenes typical of real videos. [4][5]
Joint states: the robot-agnostic layer
Once the hand has 3D coordinates, the next step is converting them into joint states: the angle of each joint, not just "the finger is bent" but "bent by this many degrees." Joint states are useful because they sit one step before robotics. They do not depend on which robot hand will eventually use them. Whether the target is a simple gripper or a humanoid hand, joint states can be translated (retargeted) to that architecture later.
Because the input is video, time is available too. At 30 or 60 frames per second, the displacement of a fingertip across a few frames gives velocity, and a second derivative gives acceleration and deceleration. The catch is that every derivative amplifies noise. A second derivative on jittery keypoints is close to useless unless the model is very good or the frame rate is high. This is why recent work does not trust raw derivatives and instead adds explicit temporal regularization on velocity and acceleration, plus learned motion priors. [6]
Depth and world-space: where the camera stops lying
The hand's position in the image is not its position in the world. In egocentric video the camera is on the person, so camera motion and hand motion are entangled. Recovering true trajectories requires estimating depth and camera pose.
With a stereo camera, depth is a straightforward computation: the pixel disparity of an object between the two views, combined with calibration (focal length and baseline), gives distance. With a monocular camera, the problem is ambiguous. Classic solutions use SLAM, tracking reference points as the camera moves, often fused with IMU data. Newer pipelines use foundation models that predict camera parameters and depth maps directly, such as Depth Anything 3. [6][7] Either way, the goal is a hand trajectory in a stable world coordinate frame, which is what HaWoR and Dyn-HaMR target by decoupling hand motion from camera motion. [8][9]
Beyond the hand: the object
A hand trajectory alone tells a robot little about the task. What matters is the hand-object interaction: what was grasped, how it moved, and whether the contact was physically plausible. EgoGrasp is, to our knowledge, the first method to reconstruct hand-object interactions in world space from monocular egocentric video with a moving camera and open-vocabulary objects. [6] It combines three stages:
The numbers show what this approach can do. On the H2O dataset, EgoGrasp estimates an object's rotation to within 18.6°, while the other methods it was compared against land between 28.9° and 35.1°. It also does a much better job of keeping the hand from sinking into the object: its overlap is just 0.74 mm, compared with 3.2 mm and 10.5 mm for two of the competing methods. [6]
The most revealing test was one where the researchers deliberately switched off parts of the system to see what would happen. When they removed the rule that keeps the wrist's position consistent over time, the error in tracking the hand's joints jumped from about 41 mm to about 238 mm. [6] In other words, a single missing rule was enough to throw the whole hand trajectory off.
That said, EgoGrasp doesn't win on every measure. On H2O, its joint-tracking error in world space is 6.84 mm, a little higher than Dyn-HaMR's 5.75 mm. The authors also point out a downside to how it's built: it relies on a chain of several models working one after another, which makes the overall pipeline more complicated. [6]
Still, the real takeaway isn't about beating a competitor on one number. EgoGrasp shows that when you account for camera movement, hand pose, body motion, and how the hand touches the object all at once, the result is a trajectory that looks and behaves more like real physical motion.
Output: a structured trajectory, not a video
The end product is a per-frame record of 3D hand positions, joint states, and object poses, typically stored as structured data such as JSON. From there, the same data can be rendered in tools like Rerun or Foxglove, or exported to a robot-learning environment, where the joint states are translated into the joint limits of a specific robotic hand. For a flat video overlay the 3D data is projected back down. For a 3D scene it is drawn using the recovered depth.
Where the pipeline breaks
Reconstructing 3D hand-object interaction from a single video is hard, and it breaks in a handful of predictable places. Fingers are constantly hidden behind the objects they hold, so joints and contact points are easy to lose. Motion can jitter, jump, or drift over long sequences. Some methods swap the left and right hand, a single camera can't tell you the real size of what it sees, and chaining all the models needed to get good results gets expensive fast. Errors at one stage tend to carry into the next, which is why long sequences suffer the most. The diagram below lays out these bottlenecks, the ones any hand tracking and pose estimation pipeline has to deal with.
Judging it as a system, not a model
A recent white paper on embodied operators argues that modules like these should be treated as reusable, deployable units with explicit input-output contracts: coordinate frames, timestamps, units, confidence scores, and failure states. [10] It also argues that a hand-motion operator should be benchmarked beyond accuracy: latency at P95 and P99, memory, stability across repeated runs, and behavior under missing frames. It proposes an Operator Card and a Run Manifest so results are reproducible. A speedup only counts if it does not degrade accuracy, determinism, or failure rate. [10] For a data pipeline this reframes the question. The metric that matters is not how good the hand mesh looks, but whether the trajectory is usable by whatever comes next.
Nurvai's closing thoughts
Converting 2D video into 3D hand motion is not one model but a chain: localization, keypoints, joint states, depth, world-space alignment, object tracking, and output formatting. Each link has its own failure modes, and errors propagate downstream. The most promising direction is the one recent work is taking: pairing perception models with generative motion priors that keep trajectories physically plausible, and evaluating the whole pipeline as a deployable system. If egocentric video is going to be a real source of robot training data, the quality of this conversion is what decides whether that data is an asset or a liability.
If you are interested in learning more about how human noise shapes teleoperation data, in exploring these dynamics in greater depth, or in sourcing robotics datasets collected and curated by our team, feel free to connect with us: Free Consultation with Nurvai
Connect with us on socials LinkedIn | X
References
[1] Yang, R., et al. EgoVLA: Learning vision-language-action models from egocentric human videos. arXiv:2507.12440 (2025).
[2] Luo, H., et al. Being-H0: Vision-language-action pretraining from large-scale human videos. arXiv:2507.15597 (2025).
[3] Romero, J., Tzionas, D., Black, M. J. Embodied hands: Modeling and capturing hands and bodies together. ACM TOG (SIGGRAPH Asia), 2017.
[4] Pavlakos, G., et al. Reconstructing hands in 3D with transformers (HaMeR). CVPR 2024.
[5] Potamias, R. A., et al. WiLoR: End-to-end 3D hand localization and reconstruction in-the-wild. CVPR 2025.
[6] Fu, H., et al. EgoGrasp: World-space hand-object interaction estimation from egocentric videos. arXiv:2601.01050 (2026).
[7] Lin, H., et al. Depth Anything 3: Recovering the visual space from any views. arXiv:2511.10647 (2025).
[8] Zhang, J., et al. HaWoR: World-space hand motion reconstruction from egocentric videos. CVPR 2025.
[9] Yu, Z., Zafeiriou, S., Birdal, T. Dyn-HaMR: Recovering 4D interacting hand motion from a dynamic camera. CVPR 2025.
[10] Xiong, J., et al. Embodied Operators and Benchmarking: Toward reusable and deployable embodied intelligence systems. arXiv:2607.03283 (2026).