When a task has been completed, it can be audited. The outcome of an audit can be that the task is one of accepted
, rejected
, or fixed
. Fixing tasks is only available for certain task types and plans.
We have information about the auditing workflow inside the web application in our documentation.
Audit Example
{
"task_id": "576ba74eec471ff9b01557cc",
"created_at": "2021-06-22T09:09:34.752Z",
"completed_at": "2021-06-23T09:10:02.798Z",
"type": "imageannotation",
"status": "completed",
"params": {
...
},
"response": {
"category": "big"
},
"customer_review_status": "fixed",
"customer_review_comments": "Was small, actually is big",
"prior_responses": [
{
"category": "small"
}
],
"audits": [
{
"audited_by": "[email protected]",
"audited_at": "2021-06-24T15:32:03.585Z",
"audit_time_secs": 120,
"audit_result": "accepted",
"audit_source": "customer"
},
{
"audited_by": "[email protected]",
"audited_at": "2021-06-23T10:01:02.352Z",
"audit_time_secs": 511,
"audit_result": "fixed",
"audit_source": "customer"
}
],
...
}
What happens after a Task Audit
When an audit is submitted, several new fields will be added to the task object:
Property | Type | Description |
---|---|---|
customer_review_status | string | Reflects the result of the audit as accepted , rejected or fixed |
customer_review_comments | string | Reflects any comments saved during the audit |
prior_responses | array | If a task is fixed , this will contain all prior, original responses. The previous response will be appended to the list of prior_responses such that the first prior_response is the oldest and the last is the newest.A new boolean field is added, is_customer_fix , to denote tasks you have fixed vs. prior responses that Scale submitted. |
If a task is fixed through multiple audits:
response
: will be updated with the most recent audit
customer_review_status
: will reflect the most recent audit
customer_review_comments
: will append the comment to the array
prior_responses
: will append previous responses to the array
The audits
array
audits
arrayThe audits
field is an array of audit records. It is only available for tasks that have been completed and audited. Each record has the following properties:
Property | Type | Description |
---|---|---|
audited_by | string (email) | The person who performed the audit. |
audited_at | timestamp | The time an audit is performed. |
audit_time_secs | number | The total time, in seconds, the audit tab is in focus in a browser. |
audit_active_time_secs | number | The total time, in seconds, when the audit tab is up in a browser, and there hasn't been any mouse or keyboard movements for over 60 seconds. |
audit_result | string | The result of the audit. It is one of accepted , fixed , or rejected . |
audit_source | string | The origin of the audit. It is either customer , or scale . |
audit_per_annotation_errors | array, optional | If per annotation error auditing is enabled, the error array. More details in the next section. |
The audit records are sorted by audited_at
in descending order, in other words, the most recent audit is the first one in the array.
In most cases, the audit_time_secs
and 'audit_active_time_secs' values should match, but in other cases, the audit_active_time_secs
can be more precise, depending on one's workflows and how one wants to measure the audit time.
For audit_active_time_secs
, after the 60 seconds threshold, the audit time would stop accumulating.
Per Annotation Errors
The audit_per_annotation_errors
is an array of per annotation error records. It is only available for audits that have per annotation errors enabled. Each record has the following properties:
Property | Type | Description |
---|---|---|
uuid | string | The UUID of the annotation associated with the error. |
severity | string | The severity of the error. It is either default or critical . |
error_type | string | The type of the error. It is one of class , geometry , attribute , extraneous , missing , or other . |
comment | string | The comment added to the error. |
attribute | string, optional | For attribute errors, the erroneous attribute. |
is_fixed | boolean, optional | True if the annotation is fixed with the audit. |
original_label | string, optional | For class errors, the original label. |
audit_label | string, optional | For class errors, the fixed audit label. |
frame_num | number, optional | For video audits, the frame that the error appears on. |
tagged_reasons | array, optional | The reasons tagged by the auditor. |
original_box | Box, optional | For geometry errors, the original annotation's box. Fields include left , top , width , height , and rotation . |
audit_box | Box, optional | For geometry errors, the fixed audit annotation's box. Fields include left , top , width , height , and rotation . |
label_index | number, optional | For semantic segmentation audits, the index of the label. |
pixel_location | array, optional | For semantic segmentation audits, the locations of the erroneous pixels. |
is_without_annotation | boolean, optional | True if the error was tagged without drawing an annotation, so uuid doesn't correspond to an actual annotation. |