Match annotation depth to your research objective

Start with the question a label must answer

If the immediate need is to retrieve complete examples of a task, a task identifier and a short description may be a useful starting point. If the need is to study action order, time segments may matter. If the team needs to distinguish which object was handled, the schema must identify those objects consistently. Write one example question for every proposed field. Fields without a clear use can create review work without improving the current experiment.

Separate task, action and object

A task describes the overall activity, such as packing an item. Actions describe observable steps, such as placing an item into a box or closing a flap. Object references clarify what those actions involve. Keep these concepts separate so a single phrase does not carry several meanings. Agree how repeated objects are distinguished and whether labels describe only what is visible. Avoid asking annotators to infer intent, force or hidden physical state from video alone.

Define time boundaries with examples

An action name is incomplete without a rule for where it begins and ends. Does a placement start when the hand reaches toward the object or when the object begins to move? Does it end at first contact with the surface or after release? Neither convention is universally right. Choose the one that suits your objective, illustrate it and specify how pauses, overlapping actions and partially obscured moments should be handled.

Make uncertainty part of the schema

Reviewers should be able to flag a decision they cannot support. Agree whether uncertain examples need an explicit status, a reason or a second review. Use a pilot to find labels that people repeatedly confuse. You may need clearer definitions, a different level of detail or a merged category. A smaller vocabulary with stable meanings can be more useful than fine distinctions that reviewers cannot reliably observe.

• Keep a short definition and an example for each label.
• Record how ambiguous, incomplete or out-of-view actions are represented.
• Version the vocabulary so later changes can be traced.

Agree the delivery contract before scaling

Specify how labels connect to recording identifiers, which time units are used and how revisions are recorded. Test a small export in your ingestion workflow before annotating the full collection. If your project requires spatial labels, motion estimates or robot-specific information, discuss those as separate requirements with their own feasibility and review criteria. Adding detailed video annotations does not turn a human recording into robot control data.