Quality Assurance

Quality Assurance

When your QA team edits or accepts a completed sensor fusion task, your callback will be invoked again.

📘

Notice

editing sensor fusion tasks in customer audit is only enabled for qualified enterprise customers who request this ability.

This second callback will include some additional fields describing the result of the audit:

Parameter

Type

Description

customer_review_status

string

The result of auditing this task. This will be one of pending, accepted, fixed or rejected.

customer_review_comments

list of string

The comments written when auditing this task, in chronological order.

prior_responses

list of Response

An ordered list of all the previous responses for this task. A common way to have multiple responses is if the customer_review_status is fixed.

📘

Notice

you can specify a different URL to be used when re-sending your task data upon being audited, by passing it in the customer_review_callback_ur field when creating the task. If you don't specify one, your regular callback URL will be invoked.

Quality reports

{
  "score": 0.99,
  "width": {
    "errors": [
      {
        "target": "0F81F763-A35C-4B11-9791-019F8CDA6524",
        "frame": 0,
        "errorType": "width",
        "metadata": {
          "originalValue": 165.33,
          "correctedValue": 169.11
        }
      }
    ]
  },
  "length": {
    "errors": []
  },
  "height": {
    "errors": []
  },
  "position": {
    "errors": []
  },
  "label": {
    "errors": []
  },
  "attribute": {
    "errors": []
  },
  "heading": {
    "errors": []
  },
  "extraneous": {
    "errors": []
  },
  "missing": {
    "errors": []
  }
}

If the task was fixed, the response field in the new callback will include the changes made during the audit. Additionally, it will include a quality_report field indicating the detail of what was adjusted in the audit, in the following format:

Parameter

Type

Description

score

float between 0 and 1

The aggregated score for this task, as defined by the applicable grading rubric. A higher score means that fewer changes were made to this task.

width.errors

list of CategorizedError

A list of the errors where a cuboid had its dimensions.x changed.

length.errors

list of CategorizedError

A list of the errors where a cuboid had its dimensions.y changed.

height.errors

list of CategorizedError

A list of the errors where a cuboid had its dimensions.z changed.

position.errors

list of CategorizedError

A list of the errors where a cuboid had its position changed.

label.errors

list of CategorizedError

A list of the errors where a cuboid had its label changed.

attribute.errors

list of CategorizedError

A list of the errors where a cuboid had its attributes changed.

heading.errors

list of CategorizedError

A list of the errors where a cuboid had its yaw changed.

extraneous.errors

list of CategorizedError

A list of the errors where a cuboid was present in the response but it got deleted (i.e. it was a false positive).

missing.errors

list of CategorizedError

A list of the errors where a cuboid was not present in the response but it got added (i.e. it was a false negative).

Categorized errors contain the detail of what cuboid was modified, and which were the original and the corrected values:

Parameter

Type

Description

target

string

The ID of the cuboid where this error was encountered.

errorType

string

The kind of error.

frame

integer

The frame number where this error was found. This field will only be present if the error is frame-specific.

metadata

Object

An object containing the detail of the error, including both the originalValue and the correctedValue. This field is not present for missing and extraneous errors.

📘

Notice

If multiple attributes are changed on the same cuboid, a single attribute error will be created; the original and corrected values will be objects where only the keys that were changed will be included.

In the case of position errors, the metadata will also contain a distance field.

📘

Notice

The label, width, length, and height error types are never frame-specific, since these attributes always stay the same throughout the frames for a given object.

The missing, extraneous, position and heading error types will only be frame-specific if the cuboid is not stationary.

The attribute error type is always frame-specific, since attributes can be different across frames even for stationary objects.