Before you can send off loads of data to the labelers, you're required to calibrate your project by running a Calibration Batch. This will allow taskers to test your project, and they'll give you feedback on the instructions and attempt to label a small subset of your data.
Calibration Batches serve the purpose of making sure
- You have a properly functioning Project
- You have completed tasks that can be turned into Training tasks for the labelers
- You have completed tasks that can be turned in Evaluation tasks to track labeler performance
Note that this is an iterative process. You may have to create more than one Calibration Batch to properly fine-tune your instructions and create the necessary number of Quality Tasks.
You can create a Calibration Batch from the
Batches page of your project.
You'll select a subset of your data for the Calibration Batch.
Once you've created your Calibration Batch, it should be completed within a few hours.
The second part of calibrating your project is to view the results. It is important to take the time to review instruction feedback as well as the actual labelling attempt.
Calibration Batches provide the inputs you'll need to refine your instructions. The higher the clarity of your instructions, the better each Tasker will be able to perform on your project. You can view this feedback in the Calibration Batch view under
Any available feedback can also be seen in the Instruction Editor, so you can view the feedback as you refine your project's instructions.
At the top of the Calibration Batch view you should be able to open the Audit Tool to go through the tasks and grade the attempt.
Once you've finished your audit, you'll have a returned Calibration Score that is represents how accurately the taskers labeled.
You might need to run more than one Calibration Batches until the calibration scores are high. Calibration Batches help to resolve initial confusions about the instructions and help you iterate.
Note, for tasks that require a higher level of english understanding or other language requirements, you can use advanced controls to specify
Updated about 1 year ago