Annotations¶

LightlyStudio supports three annotation types:

classification,
object detection, and
segmentation.

You can inspect and edit annotations in the GUI, or add and read them in Python. For dataset-level imports such as COCO, YOLO, and Label Studio format, see Image Dataset and Video Dataset.

Terminology

Annotation: A classification, object-detection box, or segmentation mask attached to a sample.
Prediction: An annotation with an optional confidence score.
Annotation class: The category of an annotation, e.g. "dog" or "cat".
Annotation source: A named group of annotations, e.g. ground_truth or model_a.

Annotations in the GUI¶

Annotations are shown in sample detail view and in the annotation-focused views. Use Edit Annotations to create, update, or delete them.

Annotations in Python¶

Use the Python API to create annotations and predictions directly, import them from model outputs, or inspect them programmatically.

Classification¶

Use CreateClassification for sample-level labels:

from lightly_studio.core.annotation import CreateClassification

sample.add_annotation(
    CreateClassification(
        class_name="cat",
        confidence=0.95,  # optional
    )
)

Object Detection¶

Use CreateObjectDetection for bounding boxes:

from lightly_studio.core.annotation import CreateObjectDetection

sample.add_annotation(
    CreateObjectDetection(
        class_name="car",
        x=10,
        y=20,
        width=30,
        height=40,
        confidence=0.9,  # optional
    )
)

To import object detection annotations at the dataset level, use add_annotations_from_coco, add_annotations_from_yolo, or add_annotations_from_labelformat.

Segmentation¶

Use CreateSegmentationMask for segmentation masks. In most cases, from_binary_mask(...) is the easiest option:

import numpy as np
from lightly_studio.core.annotation import CreateSegmentationMask

mask = np.array([
    [0, 0, 0, 0],
    [0, 1, 1, 0],
    [0, 1, 1, 0],
    [0, 0, 0, 0],
])

sample.add_annotation(
    CreateSegmentationMask.from_binary_mask(
        class_name="car",
        binary_mask=mask,
        confidence=0.85,  # optional
    )
)

If you already have RLE masks, use from_rle_mask.

RLE mask format details

For segmentation annotations, CreateSegmentationMask expects segmentation_mask as row-wise Run-Length Encoding (RLE):

Flatten the mask row by row.
The first number counts leading background pixels.
If the mask starts with foreground, the first number must be 0.
Counts then alternate between foreground and background.

Example 2x4 mask:

[[0, 1, 1, 0],
 [1, 1, 1, 1]]

Flattened:

[0, 1, 1, 0, 1, 1, 1, 1]

Encoded:

[1, 2, 1, 4]

To import segmentation annotations at the dataset level, use add_annotations_from_coco, add_annotations_from_labelformat, or add_annotations_from_pascal_voc_segmentations.

Predictions¶

Predictions are represented by the same objects as annotations. The only difference is that predictions can include a confidence value and are usually stored in their own annotation source.

For image datasets, the add_annotations_from_* methods are the easiest way to import predictions into a named source:

import lightly_studio as ls

dataset = ls.ImageDataset.create()
dataset.add_images_from_path(path="./path/to/images")

dataset.add_annotations_from_coco(
    annotations_json="./ground_truth.json",
    images_root="./path/to/images",
    annotation_source="ground_truth",
)

dataset.add_annotations_from_coco(
    annotations_json="./predictions_model_a.json",
    images_root="./path/to/images",
    annotation_source="model_a",
)

Supported methods:

If the input is a COCO prediction file, LightlyStudio reads the score field and stores it as annotation confidence.

Reading annotations¶

Each sample exposes its annotations through sample.annotations:

from lightly_studio.core.annotation import ObjectDetectionAnnotation

for sample in dataset:
    for annotation in sample.annotations:
        if isinstance(annotation, ObjectDetectionAnnotation):
            print(annotation.x, annotation.y, annotation.width, annotation.height)

Available annotation classes:

See Annotation API Reference for the full API and Search and Filter for annotation-based queries.