Videos as Input

The Lightly Worker can also directly process video data instead of images. It will download the video files from your datasource and decode them on the fly. To use video files as the input you have to create a new dataset and set the dataset_type to DatasetType.VIDEOS:

from lightly.api import ApiWorkflowClient
from lightly.openapi_generated.swagger_client import DatasetType, DatasourcePurpose

# Create the Lightly client to connect to the API.
client = ApiWorkflowClient(token="MY_LIGHTLY_TOKEN")

# Create a new dataset on the Lightly Platform.
client.create_dataset(
    dataset_name="dataset-name",
    dataset_type=DatasetType.VIDEOS,
)

# Configure the Input datasource.
client.set_s3_config(
    resource_path="s3://bucket/input/",
    region="eu-central-1",
    access_key="S3-ACCESS-KEY",
    secret_access_key="S3-SECRET-ACCESS-KEY",
    purpose=DatasourcePurpose.INPUT,
)
# Configure the Lightly datasource.
client.set_s3_config(
    resource_path="s3://bucket/lightly/",
    region="eu-central-1",
    access_key="S3-ACCESS-KEY",
    secret_access_key="S3-SECRET-ACCESS-KEY",
    purpose=DatasourcePurpose.LIGHTLY,
)

Supported Formats

The following video containers and codecs are currently supported. Other formats might work as well.

Containers:

Codecs:

AV1
Cinepak
H.264
H.265
Indeo

Selection

When using video data, the Lightly Worker will run the selection on the video frames. For example, the following code will select a set of diverse video frames:

scheduled_run_id = client.schedule_compute_worker_run(
    selection_config={
        "n_samples": 50,
        "strategies": [
            {
                "input": {
                    "type": "EMBEDDINGS"
                },
                "strategy": {
                    "type": "DIVERSITY"
                }
            }
        ]
    }
)

The selected video frames will be uploaded as images to the Lightly bucket. By default, the uploaded video frames are saved in png format. You can configure the format and size of the video frames with the output_image_format and resize configuration options, respectively:

scheduled_run_id = client.schedule_compute_worker_run(
    worker_config={
        # Image format for uploaded video frames.
        "output_image_format": "png",
    },
    selection_config={
        "n_samples": 50,
        "strategies": [
            {
                "input": {
                    "type": "EMBEDDINGS"
                },
                "strategy": {
                    "type": "DIVERSITY"
                }
            }
        ]
    },
    lightly_config={
        # Size of uploaded video frames. If negative, default size is used.
        # If size is a sequence like [h, w], output size will be matched to this.
        # If size is an int, smaller edge of the frame will be matched to this number.
        # For example, if height > width, then frame will be rescaled to
        # (size * height / width, size).
        "resize": -1,
    },
)