Task Audits

When a task has been completed, it can be audited. The outcome of an audit can be that the task is one of accepted, rejected, or fixed. Fixing tasks is only available for certain task types and plans.

We have information about the auditing workflow inside the web application in our documentation.

Audit Example

{
  "task_id": "576ba74eec471ff9b01557cc",
  "created_at": "2021-06-22T09:09:34.752Z",
  "completed_at": "2021-06-23T09:10:02.798Z",  
  "type": "imageannotation",
  "status": "completed",
  "params": {
    ...
  },
  "response": {
    "category": "big"
  },
  "customer_review_status": "fixed",
  "customer_review_comments": "Was small, actually is big",
  "prior_responses": [
    {
      "category": "small"
    }
  ],
  "audits": [
    {
      "audited_by": "[email protected]",
      "audited_at": "2021-06-24T15:32:03.585Z",
      "audit_time_secs": 120,
      "audit_result": "accepted",
      "audit_source": "customer"
    },
    {
      "audited_by": "[email protected]",
      "audited_at": "2021-06-23T10:01:02.352Z",
      "audit_time_secs": 511,
      "audit_result": "fixed",
      "audit_source": "customer"
    }
  ],
  ...
}

What happens after a Task Audit

When an audit is submitted, several new fields will be added to the task object:

Property	Type	Description
`customer_review_status`	string	Reflects the result of the audit as `accepted`, `rejected` or `fixed`
`customer_review_comments`	string	Reflects any comments saved during the audit
`prior_responses`	array	If a task is `fixed`, this will contain all prior, original responses. The previous response will be appended to the list of `prior_responses` such that the first `prior_response` is the oldest and the last is the newest. A new boolean field is added, `is_customer_fix`, to denote tasks you have fixed vs. prior responses that Scale submitted.

If a task is fixed through multiple audits:
response: will be updated with the most recent audit
customer_review_status: will reflect the most recent audit
customer_review_comments: will append the comment to the array
prior_responses: will append previous responses to the array

The `audits` array

The audits field is an array of audit records. It is only available for tasks that have been completed and audited. Each record has the following properties:

Property	Type	Description
`audited_by`	string (email)	The person who performed the audit.
`audited_at`	timestamp	The time an audit is performed.
`audit_time_secs`	number	The total time, in seconds, the audit tab is in focus in a browser.
`audit_active_time_secs`	number	The total time, in seconds, when the audit tab is up in a browser, and there hasn't been any mouse or keyboard movements for over 60 seconds.
`audit_result`	string	The result of the audit. It is one of `accepted`, `fixed`, or `rejected`.
`audit_source`	string	The origin of the audit. It is either `customer`, or `scale`.
`audit_per_annotation_errors`	array, optional	If per annotation error auditing is enabled, the error array. More details in the next section.

The audit records are sorted by audited_at in descending order, in other words, the most recent audit is the first one in the array.

In most cases, the audit_time_secs and 'audit_active_time_secs' values should match, but in other cases, the audit_active_time_secs can be more precise, depending on one's workflows and how one wants to measure the audit time.

For audit_active_time_secs, after the 60 seconds threshold, the audit time would stop accumulating.

Per Annotation Errors

The audit_per_annotation_errors is an array of per annotation error records. It is only available for audits that have per annotation errors enabled. Each record has the following properties:

Property	Type	Description
`uuid`	string	The UUID of the annotation associated with the error.
`severity`	string	The severity of the error. It is either `default` or `critical`.
`error_type`	string	The type of the error. It is one of `class`, `geometry`, `attribute`, `extraneous`, `missing`, or `other`.
`comment`	string	The comment added to the error.
`attribute`	string, optional	For attribute errors, the erroneous attribute.
`is_fixed`	boolean, optional	True if the annotation is fixed with the audit.
`original_label`	string, optional	For class errors, the original label.
`audit_label`	string, optional	For class errors, the fixed audit label.
`frame_num`	number, optional	For video audits, the frame that the error appears on.
`tagged_reasons`	array, optional	The reasons tagged by the auditor.
`original_box`	Box, optional	For geometry errors, the original annotation's box. Fields include `left`, `top`, `width`, `height`, and `rotation`.
`audit_box`	Box, optional	For geometry errors, the fixed audit annotation's box. Fields include `left`, `top`, `width`, `height`, and `rotation`.
`label_index`	number, optional	For semantic segmentation audits, the index of the label.
`pixel_location`	array, optional	For semantic segmentation audits, the locations of the erroneous pixels.
`is_without_annotation`	boolean, optional	True if the error was tagged without drawing an annotation, so `uuid` doesn't correspond to an actual annotation.

Audit Example

What happens after a Task Audit

The audits array

Per Annotation Errors

The `audits` array