Videos as Input

The LightlyOne Worker can also directly process video data instead of images. It will load the video files from your datasource and decode them on the fly. To use video files as the input you have to create a new dataset and set the dataset_type to DatasetType.VIDEOS:

from lightly.api import ApiWorkflowClient
from lightly.openapi_generated.swagger_client import DatasetType, DatasourcePurpose

# Create the LightlyOne client to connect to the API.
client = ApiWorkflowClient(token="MY_LIGHTLY_TOKEN")

# Create a new dataset on the LightlyOne Platform.
client.create_dataset(
    dataset_name="dataset-name",
    dataset_type=DatasetType.VIDEOS,
)

# Configure the Input datasource.
client.set_s3_config(
    resource_path="s3://bucket/input/project_A/",
    region="eu-central-1",
    access_key="S3-ACCESS-KEY",
    secret_access_key="S3-SECRET-ACCESS-KEY",
    purpose=DatasourcePurpose.INPUT,
)
# Configure the Lightly datasource.
client.set_s3_config(
    resource_path="s3://bucket/lightly/project_A/",
    region="eu-central-1",
    access_key="S3-ACCESS-KEY",
    secret_access_key="S3-SECRET-ACCESS-KEY",
    purpose=DatasourcePurpose.LIGHTLY,
)

Supported Formats

The following video container formats are supported. The list of filename extensions is exhaustive. Files with other extensions are not recognised as video files. The check is case insensitive, e.g. a file named video.MP4 is still recognised as an .mp4 file.

MP4 - .mp4, .m4v and .hevc extensions
MPEG - .mpg and.mpeg extensions
AVI - .avi extension
MOV - .mov extension
WebM - .webm extension

The following codecs are supported. The list is non-exhaustive. Other codecs might be supported as well.

AV1
Cinepak
H.264
H.265
Indeo

🚧
It is not recommended to use MOV videos with H.265 codec or videos captured by iPhone cameras due to slow processing speeds. We recommend to convert such videos to MP4.

Selection

When using video data, the LightlyOne Worker will run the selection on the video frames. For example, the following code will select a set of diverse video frames:

scheduled_run_id = client.schedule_compute_worker_run(
    selection_config={
        "n_samples": 50,
        "strategies": [
            {
                "input": {
                    "type": "EMBEDDINGS"
                },
                "strategy": {
                    "type": "DIVERSITY"
                }
            }
        ]
    }
)

The selected video frames will be uploaded as images to the Lightly bucket. By default, the uploaded video frames are saved in png format. You can configure the format and size of the video frames with the output_image_format and resize configuration options, respectively:

scheduled_run_id = client.schedule_compute_worker_run(
    worker_config={
        # Image format for uploaded video frames.
        "output_image_format": "png",
    },
    selection_config={
        "n_samples": 50,
        "strategies": [
            {
                "input": {
                    "type": "EMBEDDINGS"
                },
                "strategy": {
                    "type": "DIVERSITY"
                }
            }
        ]
    },
)