Quality Assurance

When your QA team edits or accepts a completed sensor fusion task, your callback will be invoked again.

📘
Notice
editing sensor fusion tasks in customer audit is only enabled for qualified enterprise customers who request this ability.

This second callback will include some additional fields describing the result of the audit:

Parameter	Type	Description
customer_review_status	string	The result of auditing this task. This will be one of `pending`, `accepted`, `fixed` or `rejected`.
customer_review_comments	list of `string`	The comments written when auditing this task, in chronological order.
prior_responses	list of `Response`	An ordered list of all the previous responses for this task. A common way to have multiple responses is if the `customer_review_status` is `fixed`.

📘
Notice
you can specify a different URL to be used when re-sending your task data upon being audited, by passing it in the customer_review_callback_ur field when creating the task. If you don't specify one, your regular callback URL will be invoked.

Quality reports

{
  "score": 0.99,
  "width": {
    "errors": [
      {
        "target": "0F81F763-A35C-4B11-9791-019F8CDA6524",
        "frame": 0,
        "errorType": "width",
        "metadata": {
          "originalValue": 165.33,
          "correctedValue": 169.11
        }
      }
    ]
  },
  "length": {
    "errors": []
  },
  "height": {
    "errors": []
  },
  "position": {
    "errors": []
  },
  "label": {
    "errors": []
  },
  "attribute": {
    "errors": []
  },
  "heading": {
    "errors": []
  },
  "extraneous": {
    "errors": []
  },
  "missing": {
    "errors": []
  }
}

If the task was fixed, the response field in the new callback will include the changes made during the audit. Additionally, it will include a quality_report field indicating the detail of what was adjusted in the audit, in the following format:

Parameter	Type	Description
score	`float` between 0 and 1	The aggregated score for this task, as defined by the applicable grading rubric. A higher score means that fewer changes were made to this task.
width.errors	list of `CategorizedError`	A list of the errors where a cuboid had its `dimensions.x` changed.
length.errors	list of `CategorizedError`	A list of the errors where a cuboid had its `dimensions.y` changed.
height.errors	list of `CategorizedError`	A list of the errors where a cuboid had its `dimensions.z` changed.
position.errors	list of `CategorizedError`	A list of the errors where a cuboid had its `position` changed.
label.errors	list of `CategorizedError`	A list of the errors where a cuboid had its `label` changed.
attribute.errors	list of `CategorizedError`	A list of the errors where a cuboid had its `attributes` changed.
heading.errors	list of `CategorizedError`	A list of the errors where a cuboid had its `yaw` changed.
extraneous.errors	list of `CategorizedError`	A list of the errors where a cuboid was present in the response but it got deleted (i.e. it was a false positive).
missing.errors	list of `CategorizedError`	A list of the errors where a cuboid was not present in the response but it got added (i.e. it was a false negative).

Categorized errors contain the detail of what cuboid was modified, and which were the original and the corrected values:

Parameter	Type	Description
target	string	The ID of the cuboid where this error was encountered.
errorType	string	The kind of error.
frame	integer	The frame number where this error was found. This field will only be present if the error is frame-specific.
metadata	Object	An object containing the detail of the error, including both the `originalValue` and the `correctedValue`. This field is not present for `missing` and `extraneous` errors.

📘
Notice
If multiple attributes are changed on the same cuboid, a single attribute error will be created; the original and corrected values will be objects where only the keys that were changed will be included.
In the case of position errors, the metadata will also contain a distance field.

📘
Notice
The label, width, length, and height error types are never frame-specific, since these attributes always stay the same throughout the frames for a given object.
The missing, extraneous, position and heading error types will only be frame-specific if the cuboid is not stationary.
The attribute error type is always frame-specific, since attributes can be different across frames even for stationary objects.

Quality Assurance