Robotics: Ground-Truth Motion Capture

Robot foundation models for manipulation need something synthetic data can't fake: ground-truth human motion and 3D hand pose, captured in the real world, at sub-millimetre precision — and across enough task, object, and environment diversity that a policy actually generalizes.

The usual sources all fall short. Teleoperation is the gold standard for clean action labels but it's slow and costly — 1 to 10 minutes of skilled operator time per episode, and it only covers the tasks you can afford to script. Web video is effectively infinite but passive: no joint angles, no gripper state, no metric scale. The scarce resource is real, action-paired, metrically-accurate motion data.

We combined two capture regimes on one synchronized clock. Egocentric capture packs worn by operators on real production floors supply the diversity and first-person viewpoint a robot actually acts from; lab-grade optical motion capture supplies the sub-millimetre reference that validates it. Every stream — RGB, depth, markers, hand pose, object pose — is hardware-clock aligned so a MANO-style 21-keypoint hand, a 33-keypoint body, and the object it manipulates all share a single timeline.

Each episode ships as a bundle of 12+ synchronized modalities, not a single video — so a training pipeline can pick exactly the signals its model consumes.

• Egocentric RGB — first-person, close to the robot's own sensor pose.
• Multi-view RGB-D — spatial reconstruction + metric depth.
• Optical MOCAP markers — sub-mm ground-truth body and hand trajectories.
• MANO 21-keypoint hands + 33-keypoint body — per-frame pose.
• Object masks + 6-DoF pose — track-linked across every frame.
• Verb-noun action segments — what happens, when.
• IMU, audio, and SLAM camera trajectory — full sensor context.

Across the corpus that's 829 hours of reference-grade capture spanning household and commercial robotics tasks.

Precision claims mean nothing without a gate that enforces them. Every capture flows through an 8-model auto-label pipeline that pre-fills hand, body, masks, depth, and verb-noun on all frames, then through 15 machine-checkable hard rules before a human ever looks.

The hard rules catch the failures that quietly poison a policy — stream desync, out-of-range world scale (an object's position ‖t‖ must land inside 0.1–5 m), dropped tracks, idle frames. What survives gets layered human review: annotator, reviewer, audit.

The moat isn't the capture rig. It's the judgment encoded in the QC pipeline — the checks that separate sub-mm ground truth from noise that looks fine until a robot trains on it.

Sub-millimetre precision, validated against peer-reviewed motion-capture benchmarks, across 6+ manipulation use cases. Every episode ships with its QC report and manifest, delivered RLDS- and LeRobot-ready in ≤48 hours — no proprietary format, no conversion tax.