How to build a robot imitation-learning dataset

Imitation learning (also called learning from demonstration) trains a robot policy from examples of a task being performed. The quality of the resulting policy is bounded by the quality of the demonstration data — so what you capture, and how precisely, matters more than the model architecture for most manipulation tasks.

What signals does an imitation-learning dataset need?

A box around an object is not enough. To learn a manipulation task, a policy needs the relationship between the hand (or gripper) and the object over time:

• Hand pose — position and orientation (6DoF), ideally as a quaternion to avoid gimbal lock.
• Hand configuration — per-finger joint angles, so the grasp shape is captured, not just the wrist.
• The manipulated object — which specific object the hand is acting on, and its pose.
• Contact — whether the hand is grasping or merely touching the object, and when that changes.
• Time alignment — all of the above synchronized frame-by-frame across the demonstration.

How do you capture these from video?

You can collect demonstrations with a single camera and recover most of these signals in software. Hand landmarks come from a hand-pose model (e.g. MediaPipe's 21 keypoints); the object comes from an object detector; the contact is inferred from the overlap between the hand and the object it controls. Metric depth — real millimetre distance — needs either a known object dimension (solved via PnP from marked corners) or a device depth sensor (ToF/LiDAR).

PI Recorder does exactly this in its Hand + Object mode: it detects the hand and the object being held, keeps only the object under manipulation, computes the grasp/touch contact, estimates hand 6DoF and finger joint angles, and can recover object metric pose. It is explicit about which values are measured versus monocular estimates, so you know what you are training on.

What format should the dataset be in?

For box-only pipelines, COCO, YOLO and Pascal VOC are standard. But those schemas have no field for pose or contact — so the robotics extras (keypoints, 6DoF pose, joint angles, hand→object contact) belong in a richer format. JSONL is the common choice: one JSON object per sample carrying the box plus the manipulation fields. PI Recorder exports all four; its JSONL carries the robotics extras while COCO/YOLO/VOC stay box-only.

A minimal workflow

• Record the demonstration on video (or import existing footage) and extract clean, hand-visible frames.
• Run hand + object detection; confirm the contact and correct any labels.
• Add metric pose where you need real distance (marked corners or a depth sensor).
• Export to JSONL for the full signal, plus COCO/YOLO if your trainer expects boxes.

The result is a demonstration dataset that carries the manipulation signal a policy actually learns from — not just object boxes.