When a task has been completed, it can be audited. The outcome of an audit can be that the task is one of accepted
, rejected
, or fixed
. Fixing tasks is only available for certain task types and plans.
We have information about the auditing workflow inside the web application in our documentation.
Audit Example
{
"task_id": "576ba74eec471ff9b01557cc",
"created_at": "2021-06-22T09:09:34.752Z",
"completed_at": "2021-06-23T09:10:02.798Z",
"type": "imageannotation",
"status": "completed",
"params": {
...
},
"response": {
"category": "big"
},
"customer_review_status": "fixed",
"customer_review_comments": "Was small, actually is big",
"prior_responses": [
{
"category": "small"
}
],
"audits": [
{
"audited_by": "[email protected]",
"audited_at": "2021-06-24T15:32:03.585Z",
"audit_time_secs": 120,
"audit_result": "accepted",
"audit_source": "customer"
},
{
"audited_by": "[email protected]",
"audited_at": "2021-06-23T10:01:02.352Z",
"audit_time_secs": 511,
"audit_result": "fixed",
"audit_source": "customer"
}
],
...
}
What happens after a Task Audit
When an audit is submitted, several new fields will be added to the task object:
Property | Type | Description |
---|---|---|
| string | Reflects the result of the audit as |
| string | Reflects any comments saved during the audit |
| array | If a task is |
If a task is fixed through multiple audits:
response
: will be updated with the most recent audit
customer_review_status
: will reflect the most recent audit
customer_review_comments
: will append the comment to the array
prior_responses
: will append previous responses to the array
The audits
array
audits
arrayThe audits
field is an array of audit records. It is only available for tasks that have been completed and audited. Each record has the following properties:
Property | Type | Description |
---|---|---|
| string (email) | The person who performed the audit. |
| timestamp | The time an audit is performed. |
| number | The total time, in seconds, the audit tab is in focus in a browser. |
| number | The total time, in seconds, when the audit tab is up in a browser, and there hasn't been any mouse or keyboard movements for over 60 seconds. |
| string | The result of the audit. It is one of |
| string | The origin of the audit. It is either |
| array, optional | If per annotation error auditing is enabled, the error array. More details in the next section. |
The audit records are sorted by audited_at
in descending order, in other words, the most recent audit is the first one in the array.
In most cases, the audit_time_secs
and 'audit_active_time_secs' values should match, but in other cases, the audit_active_time_secs
can be more precise, depending on one's workflows and how one wants to measure the audit time.
For audit_active_time_secs
, after the 60 seconds threshold, the audit time would stop accumulating.
Per Annotation Errors
The audit_per_annotation_errors
is an array of per annotation error records. It is only available for audits that have per annotation errors enabled. Each record has the following properties:
Property | Type | Description |
---|---|---|
| string | The UUID of the annotation associated with the error. |
| string | The severity of the error. It is either |
| string | The type of the error. It is one of |
| string | The comment added to the error. |
| string, optional | For attribute errors, the erroneous attribute. |
| boolean, optional | True if the annotation is fixed with the audit. |
| string, optional | For class errors, the original label. |
| string, optional | For class errors, the fixed audit label. |
| number, optional | For video audits, the frame that the error appears on. |
| array, optional | The reasons tagged by the auditor. |
| Box, optional | For geometry errors, the original annotation's box. Fields include |
| Box, optional | For geometry errors, the fixed audit annotation's box. Fields include |
| number, optional | For semantic segmentation audits, the index of the label. |
| array, optional | For semantic segmentation audits, the locations of the erroneous pixels. |
| boolean, optional | True if the error was tagged without drawing an annotation, so |