Embeddings

Beta API

The Embeddings API is in beta. Its interface may change in future releases without a deprecation period.

LightlyStudio embeds your data automatically on ingestion. To supply your own embeddings — either computed on the fly or loaded from a precomputed store — implement one of the generator protocols below and register it with set_default_embedding_model. The registration must happen before you load a dataset or before the GUI is started.

See the Embeddings page for more details.

set_default_embedding_model

Embedding manager for dataset processing.

set_default_embedding_model

set_default_embedding_model(embedding_generator: EmbeddingGenerator) -> None

Register a custom embedding model that overrides the env-var default.

Beta

Call this before you add a dataset (for example, before ImageDataset.load_or_create) or before you launch the GUI to use your own generator instead of the model set by LIGHTLY_STUDIO_EMBEDDINGS_MODEL_TYPE. The override applies to every collection.

Parameters:

Name	Type	Description	Default
embedding_generator	EmbeddingGenerator	A generator that implements ImageEmbeddingGenerator and/or VideoEmbeddingGenerator.	required

Generator protocols

EmbeddingGenerator

EmbeddingGenerator implementations.

EmbeddingGenerator

Bases: Protocol

Base protocol shared by every embedding generator.

Beta

An EmbeddingGenerator provides embeddings for images, image crops, videos, video frames or other sample types. Every sample collection can have a different associated embedding generator. The generator is used at two points:

• During data loading to compute sample embeddings
• During a GUI run to embed text or image search queries

Generators are loaded at startup and need to identify themselves with get_embedding_model_input which returns metadata about the loaded model.

To provide custom embeddings, implement one of the protocols below (ImageEmbeddingGenerator and/or VideoEmbeddingGenerator) and register it with set_default_embedding_model before you add a dataset or start the GUI.

embed_text

embed_text(text: str) -> list[float]

Generate an embedding for a text sample.

Beta

Parameters:

Name	Type	Description	Default
text	str	The text to embed.	required

Returns:

Type	Description
list[float]	A list of floats representing the generated embedding.

get_embedding_model_input

get_embedding_model_input(collection_id: UUID) -> EmbeddingModelCreate

Generate an EmbeddingModelCreate instance.

Beta

Returns metadata about the model to be stored in the database. The embedding_model_hash field is used to match the same EmbeddingGenerator across multiple LightlyStudio runs.

Parameters:

Name	Type	Description	Default
collection_id	UUID	The ID of the collection.	required

Returns:

Type	Description
EmbeddingModelCreate	An EmbeddingModelCreate instance with the model details.

ImageEmbeddingGenerator

EmbeddingGenerator implementations.

ImageEmbeddingGenerator

Bases: EmbeddingGenerator, Protocol

Protocol for embedding images, image crops, and text.

Beta

Implement this to use your own image model in LightlyStudio. Inside embed_images you can run the model on the given file paths or look up precomputed vectors. A registered image generator replaces the built-in image, crop, and text embeddings. It inherits embed_text from EmbeddingGenerator, so keep the text and image encoders in the same embedding space for text-based image and crop search to work.

embed_image_crops

embed_image_crops(image_crops: list[ImageCrop], show_progress: bool = True) -> EmbeddingResult

Generate embeddings for image crops.

Beta

Parameters:

Name	Type	Description	Default
image_crops	list[ImageCrop]	A list of image crop definitions to embed.	required
show_progress	bool	Whether to show a progress bar during embedding.	True

Returns:

Type	Description
EmbeddingResult	An EmbeddingResult with embeddings for the crops of readable files, in the same order as the corresponding input crops.

embed_images

embed_images(filepaths: list[str], show_progress: bool = True) -> EmbeddingResult

Generate embeddings for multiple image samples.

Beta

Parameters:

Name	Type	Description	Default
filepaths	list[str]	A list of fsspec-compatible file paths to the images to embed. Each one is an absolute local path with forward slashes (e.g. /data/cat.jpg) or a remote URI (e.g. s3://bucket/cat.jpg).	required
show_progress	bool	Whether to show a progress bar during embedding.	True

Returns:

Type	Description
EmbeddingResult	An EmbeddingResult with embeddings for the readable files, in the same order as the corresponding input file paths. Use kept_indices to skip any file paths you cannot or do not want to embed.

embed_pil_images

embed_pil_images(images: list[Image], show_progress: bool = True) -> NDArray[float32]

Generate embeddings for in-memory PIL images.

Beta

Used for video frame embedding.

Parameters:

Name	Type	Description	Default
images	list[Image]	PIL images to embed.	required
show_progress	bool	Whether to show a progress bar during embedding.	True

Returns:

Type	Description
NDArray[float32]	A numpy array representing the generated embeddings in the same order as the input images.

embed_text

embed_text(text: str) -> list[float]

Generate an embedding for a text sample.

Beta

Parameters:

Name	Type	Description	Default
text	str	The text to embed.	required

Returns:

Type	Description
list[float]	A list of floats representing the generated embedding.

get_embedding_model_input

get_embedding_model_input(collection_id: UUID) -> EmbeddingModelCreate

Generate an EmbeddingModelCreate instance.

Beta

Returns metadata about the model to be stored in the database. The embedding_model_hash field is used to match the same EmbeddingGenerator across multiple LightlyStudio runs.

Parameters:

Name	Type	Description	Default
collection_id	UUID	The ID of the collection.	required

Returns:

Type	Description
EmbeddingModelCreate	An EmbeddingModelCreate instance with the model details.

VideoEmbeddingGenerator

EmbeddingGenerator implementations.

VideoEmbeddingGenerator

Bases: EmbeddingGenerator, Protocol

Protocol for embedding videos (and text).

Beta

Implement this to use your own video model in LightlyStudio. A registered video generator replaces the built-in video and text embeddings. It inherits embed_text from EmbeddingGenerator, so keep the text and video encoders in the same embedding space for text-based video search to work.

embed_text

embed_text(text: str) -> list[float]

Generate an embedding for a text sample.

Beta

Parameters:

Name	Type	Description	Default
text	str	The text to embed.	required

Returns:

Type	Description
list[float]	A list of floats representing the generated embedding.

embed_videos

embed_videos(filepaths: list[str]) -> EmbeddingResult

Generate embeddings for multiple video samples.

Beta

Parameters:

Name	Type	Description	Default
filepaths	list[str]	A list of fsspec-compatible file paths to the videos to embed. Each one is an absolute local path with forward slashes (e.g. /data/clip.mp4) or a remote URI (e.g. s3://bucket/clip.mp4).	required

Returns:

Type	Description
EmbeddingResult	An EmbeddingResult with embeddings for the readable videos, in the same order as the corresponding input file paths.

get_embedding_model_input

get_embedding_model_input(collection_id: UUID) -> EmbeddingModelCreate

Generate an EmbeddingModelCreate instance.

Beta

Returns metadata about the model to be stored in the database. The embedding_model_hash field is used to match the same EmbeddingGenerator across multiple LightlyStudio runs.

Parameters:

Name	Type	Description	Default
collection_id	UUID	The ID of the collection.	required

Returns:

Type	Description
EmbeddingModelCreate	An EmbeddingModelCreate instance with the model details.

Supporting types

EmbeddingResult

Result type shared by the batched embedding paths.

Holds EmbeddingResult, the model-agnostic return type used across the image, image-crop, and video embedding paths. It lives in its own module so any embedding path can depend on it without pulling in a path-specific module.

EmbeddingResult dataclass

EmbeddingResult(embeddings: NDArray[float32], kept_indices: list[int])

Embeddings for the inputs that could be read, plus which inputs they cover.

Beta

A generator skips broken inputs (files it cannot read or decode) instead of failing the whole run, so embeddings can have fewer rows than the input list. kept_indices gives the position of each row in the input list, in input order. Use it to line up the embeddings with any per-input data you keep on the side, such as sample IDs.

embeddings instance-attribute

embeddings: NDArray[float32]

Float32 array of shape (len(kept_indices), embedding_dimension).

kept_indices instance-attribute

kept_indices: list[int]

Indices into the input list of the inputs that were embedded, in input order.

ImageCrop

EmbeddingGenerator implementations.

ImageCrop dataclass

ImageCrop(filepath: str, x: int, y: int, width: int, height: int)

A rectangular region of an image to embed, given in pixel coordinates.

Beta

filepath instance-attribute

filepath: str

Path to the image the crop is taken from.

height instance-attribute

height: int

Crop height in pixels.

width instance-attribute

width: int

Crop width in pixels.

x instance-attribute

x: int

Left edge of the crop, in pixels from the image's left.

y instance-attribute

y: int

Top edge of the crop, in pixels from the image's top.