Data Types and the Frame Objects

Definition: Vector2

{
  "x": 1,
  "y": 2
}

Vector2 objects are used to represent positions, and are JSON objects with 2 properties.

Definition: Vector3

{
  "x": 1,
  "y": 2,
  "z": 3
}

Vector3 objects are used to represent positions, and are JSON objects with 3 properties.

Definition: LidarPoint

{
  "x": 1,
  "y": 2,
  "z": 3,
  "i": 0.5,
  "d": 2,
  "t": 1541196976735462000,
  "is_ground": true
}

LidarPoint objects are mainly used to represent LIDAR points, and are JSON objects.

📘

Good things to note:

z is up, so the x-y plane should be flat with the ground

Scale processes point coordinates as 32-bit floats. If your coordinates are greater than 10^5, your point cloud may suffer from rounding effects.

What we recommend as a best practice is applying the negative position of the first frame as an offset to all points/calibrations to avoid this, such that the first frame is at position 0,0,0 and all subsequent frames are just the offsets.

Why 10^5 as a suggested limit?
Given that 32 bit floats support 23 bits of precision, when using 10e5 order coordinates, that only leaves you about 2 decimal places of precision. (2e23 ~= 10e7)

Assuming a meter-based unit of measure, this means the point precision would be to the nearest centimeter. Going above 10e5 reduces the precision to an often unacceptable degree, hence the warning here.

Definition: Quaternion

{
  "x": 1,
  "y": 1,
  "z": 1,
  "w": 1
}

Quaternion objects are used to represent rotation. We use the Hamilton quaternion convention, where i^2 = j^2 = k^2 = ijk = -1, i.e. the right-handed convention.
The quaternion represented by the tuple (x, y, z, w) is equal to w + x*i + y*j + z*k.

Definition: BoundingBox

{
  "top": -85,
  "left": -170,
  "bottom": 85,
  "right": 170
}

Specifies the lat/lon (in degrees) of a BoundingBox. Used for transforming from world coordinates to pixel coordinates

Definition: GPSPose

{
  "lat": 1,
  "lon": 1,
  "bearing": 1
}

GPSPose objects are used to represent the pose (location and direction) of the robot in the world.

Definition: CameraImage

CameraImage objects represent an image and the camera position/heading used to record the image.

Camera models supported:

• Pinhole model with Brown-Conrady distortion (supports k1, k2, k3, p1, p2 distortion parameters)
• OpenCV fisheye model with radial distortion (supports k1, k2, k3, k4 distortion parameters)
• OpenCV omnidirectional model (supports k1, k2, k3, p1, p2 distortion parameters)

For example, to represent a camera pointing along the positive x-axis (i.e. the camera's z-axis points along the world's x-axis) and oriented normally (the camera's y-axis points along the world's negative z-axis), the corresponding heading is 0.5 - 0.5i + 0.5j - 0.5k.

Definition: RadarPoint

{
  "position": {
    "x": 100.1,
    "y": 150.12,
    "z": 200.2
  },
  "direction": {
    "x": 1,
    "y": 0,
    "z": 0
  },
  "size": 0.5
}

RadarPoint objects are used to define an individual radar point and Doppler in a particular frame.

Definition: AnnotationRule

This object enforces certain annotation relationships.

Definition: DeriveFrom

This object enforces that if line annotations are used to form, or in other words, "derive" a polygon annotation, then the labels of the involved annotations must be of a certain set.

Definition: RegionOfInterest2d

RegionOfInterest2d This object allows Scale to perform the correct transformation from lon/lat world coordinates to pixels. It allows Scale to identify the pixel coordinates of the camera location on the provided aerial imagery task. If this is not provided, the camera context images will not render on the task.

Definition: RegionOfInterest3d

RegionOfInterest3d This Object crops the attachments’ points to a rectangle on the XY plane centered around position with rotation counterclockwise to the z-axis. This must be submitted for any LiDAR TopDown annotation tasks, and defines the bounds to which the point cloud should be restricted to for annotation. The greater of the x and y dimensions are meter measurements, and are used to create a square on the XY plane. The RegionOfInterest3d differs from geofencing in that it crops in 3D world space as opposed to 2D orthographic image space. If the RegionOfInterest3d specified is larger than the size of the point cloud, the orthographic image will contain empty/black space where there are no points. If the RegionOfInterest3d contains no points, then an error will be thrown. Annotations will be translated to the RegionOfInterest3d coordinate frame, but will be translated back before the response is sent to the customer.

Definition: CameraContext

CameraContext objects are non-primary images that can be referenced during labeling. They are used to provide additional information to annotators, but are not annotated themselves.

Definition: Link

Links are created to represent a relationship between two Annotation objects.

Definition: OrthoResponse

{
 "response_type": "ortho_response",
 "annotations": [
   {
     "label": "Lane Line",
     "uuid": "3927f821-bfca-4cc9-8f6c-4fcf3c9e524e",
     "vertices": [
       {
         "x": 1.400146484375,
         "y": 5.218505859375
       },
       {
         "x": 1003.986572265625,
         "y": 1053.152587890625
       },
       {
         "x": 2984.986572265625,
         "y": 2701.152587890625
       }
     ],
     "type": "line"
   }
 ]
}

The OrthoResponse contains annotations in local coordinates of the task. Note that the Annotations returned in the OrthoResponse are in 2D.

Definition: WorldResponse

{
 "response_type": "world_response",
 "annotations": [
   {
     "label": "Lane Line",
     "uuid": "3927f821-bfca-4cc9-8f6c-4fcf3c9e524e",
     "vertices_3d": [
       {
         "x": 10014.00146484375,
         "y": 10052.18505859375,
         "z": 5.153
       },
       ...
     ],
     "type": "line"
   }
 ]
}

The WorldResponse contains annotations that are re-projected from the region_of_interest_3d into the World scene coordinates of the original Lidar data. The GroundMesh is used to assign altitude (Z-coordinates) to all Annotation vertices.
Note that aerial imagery tasks have their WorldResponse and OrthoResponse in the same coordinate space as no region_of_interest_3d is specified.

Definition: CameraResponse

[
 {
   "response_type": "camera_response",
   "annotations": [
     {
       "label": "Lane Line",
       "uuid": "3927f821-bfca-4cc9-8f6c-4fcf3c9e524e",
       "vertices_3d": [
         ...
       ],
       "type": "line"
     }
   ],
   "frame_number": 0,
   "camera_index": 0,
   "metadata": {}
 },
 ...
]

For each Lidar Camera Context Attachment, a CameraResponse is generated, which holds an array of all World annotations projected into the 3D coordinate space from the PoV of the Lidar camera. In the LidarTopdown final task response, the “camera” field holds an array of CameraResponse, one for each Lidar Camera Context Attachment.

Definition: ImageResponse

[
 {
   "response_type": "image_response",
   "annotations": [
     {
       "label": "Lane Line",
       "uuid": "3927f821-bfca-4cc9-8f6c-4fcf3c9e524e",
       "vertices": [
         ...
       ],
       "type": "line"
     }
   ],
   "frame_number": 0,
   "camera_index": 0,
   "metadata": {}
 },
 ...
]

For each Lidar Camera Context Attachment, the ImageResponse holds an array of all World annotations projected onto the camera context image itself. These annotations are in the image space of the camera context image, and as such are 2D annotations. As with the ImageResponse, in the LidarTopdown final task response, the CameraResponse is an array holding CameraResponses for each Lidar Camera Context Attachment.

Definition: Frame

Frame objects represent all the point cloud, image, and other data that is sent to the annotator.

Example JSON file for a Frame

https://static.scale.com/scaleapi-lidar-pointclouds/example.json

Steps to Process a series of LIDAR data

1. Create a Frame JSON object and save to a file; alternatively, create a LidarFrame protobuf message and save to a file. Repeat for each lidar frame.
2. Upload the files to a Scale accessible location (e.g. an S3 bucket) and record the URLs.
3. Send a POST request to https://api.scale.com/v1/task/lidarannotation