Writing Instructions: Best Practices

Below are some best practices to consider as you write instructions:

  1. Dive into your data: Before writing instructions, it’s important to review a representative sample of your data to understand what global and edge case rules need to be written. As you’re reviewing the data, put yourself in the labeler's shoes; document areas you’d need guidance on, as well as any tricky cases that need to be addressed in the instructions.

  2. Writing instructions:

    • Structure: Following a clear structure in your instructions will help labelers better understand the rules, leading to higher quality annotations. The below structure is a good starting point:
    • Summary of Task: Provide a few pithy bullets describing the basics of your task.
    • Workflow: Provide the steps labelers must take to annotate your task from start to finish.
    • Annotation Rules: Start with the global rules (i.e. rules that apply to all tasks, then zoom in to more specific rules). Areas you’ll likely want to touch on:
      • Minimum pixel size
      • What to label and what NOT to label
      • How to manage occlusion, truncation, and low visibility cases
      • Geometry sizing if relevant
      • How to manage cases you’re not sure about (i.e. “err on the side of selecting XXX”)
      • Etc.
    • Label and Attribute Definitions and Examples
    • Common Errors
    • Edge Cases
  3. Communication:

    • Communicate succinctly and clearly (ie pithy bullets, use spaces to decrease cognitive load for the labeler)
    • Include at least 1 image for each point you’re looking to make. Make sure each image has a description of WHY it is correctly or incorrectly annotated.
  4. Pressure test your instructions: After writing the instructions, review your data again and make sure your instructions cover the majority (90+% of cases you see). Continue to iterate on instructions until you have hit the 90+% number.

  5. Final tips:

    • It can be helpful to include a short video walking through the instructions
    • If there are certain tools / features that you expect labelers to use when annotating your data, we’d suggest mentioning them in the instructions and providing a description of how to use the tool (e.g. interpolation)