Calibration Batches are a critical step in the iteration of your project. Getting a good Calibration Score indicates a healthy project where taskers are able to understand your instructions and label your data.
Calibration Score is a function of audit results and Tasker confidence
It is recommended that your Calibration Score is above 80% before moving on to a Production Batch
Inputs that go into calculating the Calibration Score include:
- The number of rejections you made while auditing the returned completed tasks
- How confident the Tasker self-reported after reading the instructions & completing your task
Iterate on instructions
Write Instructions that cater better to the moments of confusion taskers may be having
For each Calibration Batch you will receive feedback on each task, based on how easy or confusing the Tasker found the process of reading the instructions and executing on the task.
Per feedback, you may want to view the Audit Response (to see tasker response and any of your audit versions) and see the instructions attached to that task. To edit the Instructions, you'll need to go to
Task Setup >
Current Instructions, where you can view the feedback and Calibration Score next to your instruction workspace.
Commonly, the feedback may point to adding more context, explanations, and examples of properly and improperly labelled tasks.
In general, you should aim to write instructions in a way that shows how one should do the task as opposed to just describing it.
🐶 Example Image Annotation Project
Dogs & their coat color
❌ Don't do: Describing the task
Label dogs and their coat colors [solid, merle, brindle, harlequin, ticked, spotted, roan, tricolor]
✔️ Do: Show how to do the task
Describe overall goal of labelling dogs and their coat colors.
"The goal of this project is to properly distinguish between dog coat colors across a variety of breeds. We would like to have each dog in the dataset labeled and also have their coat coloring marked."
Give an overview of how to label dogs:
"Draw a bounding polygon around the dogs, making sure to include all legs, ears, and tails"
→ Include examples of well drawn and poorly drawn polygons.
Give an overview of coat colors:
"Here are multiple examples of all of the coat colors across different breeds..."
→ Include various examples for coat colors.
Updated 12 months ago