Videos as Input

The Lightly Worker can also directly process video data instead of images. It will load the video files from your datasource and decode them on the fly. To use video files as the input you have to create a new dataset and set the dataset_type to DatasetType.VIDEOS:

from lightly.api import ApiWorkflowClient
from lightly.openapi_generated.swagger_client import DatasetType, DatasourcePurpose

# Create the Lightly client to connect to the API.
client = ApiWorkflowClient(token="MY_LIGHTLY_TOKEN")

# Create a new dataset on the Lightly Platform.
client.create_dataset(
    dataset_name="dataset-name",
    dataset_type=DatasetType.VIDEOS,
)

# Configure the Input datasource.
client.set_s3_config(
    resource_path="s3://bucket/input/",
    region="eu-central-1",
    access_key="S3-ACCESS-KEY",
    secret_access_key="S3-SECRET-ACCESS-KEY",
    purpose=DatasourcePurpose.INPUT,
)
# Configure the Lightly datasource.
client.set_s3_config(
    resource_path="s3://bucket/lightly/",
    region="eu-central-1",
    access_key="S3-ACCESS-KEY",
    secret_access_key="S3-SECRET-ACCESS-KEY",
    purpose=DatasourcePurpose.LIGHTLY,
)

Supported Formats

The following video containers and codecs are currently supported. Other formats might work as well.

Containers:

Codecs:

AV1
Cinepak
H.264
H.265
Indeo

🚧
It is not recommended to use MOV videos with H.265 codec or videos captured by iPhone cameras due to slow processing speeds. We recommend to convert such videos to MP4.

Selection

When using video data, the Lightly Worker will run the selection on the video frames. For example, the following code will select a set of diverse video frames:

scheduled_run_id = client.schedule_compute_worker_run(
    selection_config={
        "n_samples": 50,
        "strategies": [
            {
                "input": {
                    "type": "EMBEDDINGS"
                },
                "strategy": {
                    "type": "DIVERSITY"
                }
            }
        ]
    }
)

The selected video frames will be uploaded as images to the Lightly bucket. By default, the uploaded video frames are saved in png format. You can configure the format and size of the video frames with the output_image_format and resize configuration options, respectively:

scheduled_run_id = client.schedule_compute_worker_run(
    worker_config={
        # Image format for uploaded video frames.
        "output_image_format": "png",
    },
    selection_config={
        "n_samples": 50,
        "strategies": [
            {
                "input": {
                    "type": "EMBEDDINGS"
                },
                "strategy": {
                    "type": "DIVERSITY"
                }
            }
        ]
    },
)