Task Audits

When a task has been completed, it can be audited. The outcome of an audit can be that the task is one of accepted, rejected, or fixed. Fixing tasks is only available for certain task types and plans.

We have information about the auditing workflow inside the web application in our documentation.

Audit Example

{
  "task_id": "576ba74eec471ff9b01557cc",
  "created_at": "2021-06-22T09:09:34.752Z",
  "completed_at": "2021-06-23T09:10:02.798Z",  
  "type": "imageannotation",
  "status": "completed",
  "params": {
    ...
  },
  "response": {
    "category": "big"
  },
  "customer_review_status": "fixed",
  "customer_review_comments": "Was small, actually is big",
  "prior_responses": [
    {
      "category": "small"
    }
  ],
  "audits": [
    {
      "audited_by": "[email protected]",
      "audited_at": "2021-06-24T15:32:03.585Z",
      "audit_time_secs": 120,
      "audit_result": "accepted",
      "audit_source": "customer"
    },
    {
      "audited_by": "[email protected]",
      "audited_at": "2021-06-23T10:01:02.352Z",
      "audit_time_secs": 511,
      "audit_result": "fixed",
      "audit_source": "customer"
    }
  ],
  ...
}

What happens after a Task Audit

When an audit is submitted, several new fields will be added to the task object:

PropertyTypeDescription
customer_review_statusstringReflects the result of the audit as accepted, rejected or fixed
customer_review_commentsstringReflects any comments saved during the audit
prior_responsesarrayIf a task is fixed, this will contain all prior, original responses.

If a task is fixed through multiple audits:
response: will be updated with the most recent audit
customer_review_status: will reflect the most recent audit
customer_review_comments: will append the comment to the array
prior_responses: will append previous responses to the array

The audits array

The audits field is an array of audit records. It is only available for tasks that have been completed and audited. Each record has the following properties:

PropertyTypeDescription
audited_bystring (email)The person who performed the audit.
audited_attimestampThe time an audit is performed.
audit_time_secsnumberThe total time, in seconds, the audit tab is in focus in a browser.
audit_active_time_secsnumberThe total time, in seconds, when the audit tab is up in a browser, and there hasn't been any mouse or keyboard movements for over 60 seconds.
audit_resultstringThe result of the audit. It is one of accepted, fixed, or rejected.
audit_sourcestringThe origin of the audit. It is either customer, or scale.
audit_per_annotation_errorsarray, optionalIf per annotation error auditing is enabled, the error array. More details in the next section.

The audit records are sorted by audited_at in descending order, in other words, the most recent audit is the first one in the array.

In most cases, the audit_time_secs and 'audit_active_time_secs' values should match, but in other cases, the audit_active_time_secs can be more precise, depending on one's workflows and how one wants to measure the audit time.

For audit_active_time_secs, after the 60 seconds threshold, the audit time would stop accumulating.

Per Annotation Errors

The audit_per_annotation_errors is an array of per annotation error records. It is only available for audits that have per annotation errors enabled. Each record has the following properties:

PropertyTypeDescription
uuidstringThe UUID of the annotation associated with the error.
severitystringThe severity of the error. It is either default or critical.
error_typestringThe type of the error. It is one of class, geometry, attribute, extraneous, missing, or other.
commentstringThe comment added to the error.
attributestring, optionalFor attribute errors, the erroneous attribute.
is_fixedboolean, optionalTrue if the annotation is fixed with the audit.
original_labelstring, optionalFor class errors, the original label.
audit_labelstring, optionalFor class errors, the fixed audit label.
frame_numnumber, optionalFor video audits, the frame that the error appears on.
tagged_reasonsarray, optionalThe reasons tagged by the auditor.
original_boxBox, optionalFor geometry errors, the original annotation's box. Fields include left, top, width, height, and rotation.
audit_boxBox, optionalFor geometry errors, the fixed audit annotation's box. Fields include left, top, width, height, and rotation.
label_indexnumber, optionalFor semantic segmentation audits, the index of the label.
pixel_locationarray, optionalFor semantic segmentation audits, the locations of the erroneous pixels.
is_without_annotationboolean, optionalTrue if the error was tagged without drawing an annotation, so uuid doesn't correspond to an actual annotation.