What is hand–object contact annotation, and why does it matter?

Hand–object contact annotation records the interaction between a hand and the object it is manipulating: not just where each is, but whether they are in contact and how — a firm grasp versus a light touch — at each moment of a demonstration.

Why a box isn't enough

Object detection tells you an object is present and where. Hand detection tells you a hand is present and where. Neither tells you that this hand is acting on that object — which is the entire point of a manipulation demonstration. Contact annotation makes that relationship explicit.

How is contact computed?

A common, honest approach uses geometry: pair each hand with the object it most overlaps, and classify the interaction by the amount of overlap — a high overlap reads as a grasp, a lower but non-zero overlap as a touch. It's a heuristic from 2D boxes, so a human can confirm or correct it, but it produces the grasp/touch signal directly and cheaply.

Why it matters for robots

Robot imitation-learning policies need to know when a grasp begins and ends to segment a task and learn the right timing. Contact is the label that encodes that transition. Combined with hand pose and object pose, it gives a policy the full picture: which object, held how, and when.

PI Recorder captures this in its robotics mode — pairing the hand with the object it controls, labeling the contact as grasp or touch, and saving it alongside hand pose and joint angles so the exported dataset carries the manipulation signal end-to-end.