Videos as Input
The Lightly Worker can also directly process video data instead of images. It will load the video files from your datasource and decode them on the fly. To use video files as the input you have to create a new dataset and set the dataset_type
to DatasetType.VIDEOS
:
from lightly.api import ApiWorkflowClient
from lightly.openapi_generated.swagger_client import DatasetType, DatasourcePurpose
# Create the Lightly client to connect to the API.
client = ApiWorkflowClient(token="MY_LIGHTLY_TOKEN")
# Create a new dataset on the Lightly Platform.
client.create_dataset(
dataset_name="dataset-name",
dataset_type=DatasetType.VIDEOS,
)
# Configure the Input datasource.
client.set_s3_config(
resource_path="s3://bucket/input/",
region="eu-central-1",
access_key="S3-ACCESS-KEY",
secret_access_key="S3-SECRET-ACCESS-KEY",
purpose=DatasourcePurpose.INPUT,
)
# Configure the Lightly datasource.
client.set_s3_config(
resource_path="s3://bucket/lightly/",
region="eu-central-1",
access_key="S3-ACCESS-KEY",
secret_access_key="S3-SECRET-ACCESS-KEY",
purpose=DatasourcePurpose.LIGHTLY,
)
Supported Formats
The following video containers and codecs are currently supported. Other formats might work as well.
Containers:
- AVI
- MOV
- MP4
Codecs:
- AV1
- Cinepak
- H.264
- H.265
- Indeo
It is not recommended to use MOV videos with H.265 codec or videos captured by iPhone cameras due to slow processing speeds. We recommend to convert such videos to MP4.
Selection
When using video data, the Lightly Worker will run the selection on the video frames. For example, the following code will select a set of diverse video frames:
scheduled_run_id = client.schedule_compute_worker_run(
selection_config={
"n_samples": 50,
"strategies": [
{
"input": {
"type": "EMBEDDINGS"
},
"strategy": {
"type": "DIVERSITY"
}
}
]
}
)
The selected video frames will be uploaded as images to the Lightly bucket. By default, the uploaded video frames are saved in png
format. You can configure the format and size of the video frames with the output_image_format
and resize
configuration options, respectively:
scheduled_run_id = client.schedule_compute_worker_run(
worker_config={
# Image format for uploaded video frames.
"output_image_format": "png",
},
selection_config={
"n_samples": 50,
"strategies": [
{
"input": {
"type": "EMBEDDINGS"
},
"strategy": {
"type": "DIVERSITY"
}
}
]
},
)
Updated 3 months ago