Videos as Input
The LightlyOne Worker can also directly process video data instead of images. It will load the video files from your datasource and decode them on the fly. To use video files as the input you have to create a new dataset and set the dataset_type
to DatasetType.VIDEOS
:
from lightly.api import ApiWorkflowClient
from lightly.openapi_generated.swagger_client import DatasetType, DatasourcePurpose
# Create the LightlyOne client to connect to the API.
client = ApiWorkflowClient(token="MY_LIGHTLY_TOKEN")
# Create a new dataset on the LightlyOne Platform.
client.create_dataset(
dataset_name="dataset-name",
dataset_type=DatasetType.VIDEOS,
)
# Configure the Input datasource.
client.set_s3_config(
resource_path="s3://bucket/input/",
region="eu-central-1",
access_key="S3-ACCESS-KEY",
secret_access_key="S3-SECRET-ACCESS-KEY",
purpose=DatasourcePurpose.INPUT,
)
# Configure the Lightly datasource.
client.set_s3_config(
resource_path="s3://bucket/lightly/",
region="eu-central-1",
access_key="S3-ACCESS-KEY",
secret_access_key="S3-SECRET-ACCESS-KEY",
purpose=DatasourcePurpose.LIGHTLY,
)
Supported Formats
The following video container formats are supported. The list of filename extensions is exhaustive. Files with other extensions are not recognised as video files. The check is case insensitive, e.g. a file named video.MP4
is still recognised as an .mp4
file.
- MP4 -
.mp4
,.m4v
and.hevc
extensions - MPEG -
.mpg
and.mpeg
extensions - AVI -
.avi
extension - MOV -
.mov
extension - WebM -
.webm
extension
The following codecs are supported. The list is non-exhaustive. Other codecs might be supported as well.
- AV1
- Cinepak
- H.264
- H.265
- Indeo
It is not recommended to use MOV videos with H.265 codec or videos captured by iPhone cameras due to slow processing speeds. We recommend to convert such videos to MP4.
Selection
When using video data, the LightlyOne Worker will run the selection on the video frames. For example, the following code will select a set of diverse video frames:
scheduled_run_id = client.schedule_compute_worker_run(
selection_config={
"n_samples": 50,
"strategies": [
{
"input": {
"type": "EMBEDDINGS"
},
"strategy": {
"type": "DIVERSITY"
}
}
]
}
)
The selected video frames will be uploaded as images to the Lightly bucket. By default, the uploaded video frames are saved in png
format. You can configure the format and size of the video frames with the output_image_format
and resize
configuration options, respectively:
scheduled_run_id = client.schedule_compute_worker_run(
worker_config={
# Image format for uploaded video frames.
"output_image_format": "png",
},
selection_config={
"n_samples": 50,
"strategies": [
{
"input": {
"type": "EMBEDDINGS"
},
"strategy": {
"type": "DIVERSITY"
}
}
]
},
)
Updated 3 months ago