Embeddings

Beta API

The Embeddings API is in beta. Its interface may change in future releases without a deprecation period.

LightlyStudio embeds your data automatically on ingestion. To supply your own embeddings — either computed on the fly or loaded from a precomputed store — implement one of the generator protocols below and register it with set_default_embedding_model. The registration must happen before you load a dataset or before the GUI is started.

See the Embeddings page for more details.

set_default_embedding_model

Embedding manager for dataset processing.

set_default_embedding_model

set_default_embedding_model(embedding_generator: EmbeddingGenerator) -> None

Register a custom embedding model that overrides the env-var default.

Beta

Call this before you add a dataset (for example, before ImageDataset.load_or_create) or before you launch the GUI to use your own generator instead of the model set by LIGHTLY_STUDIO_EMBEDDINGS_MODEL_TYPE. The override applies to every collection.

Parameters:

Name	Type	Description	Default
embedding_generator	EmbeddingGenerator	A generator that implements ImageEmbeddingGenerator and/or VideoEmbeddingGenerator.	required

Generator protocols

EmbeddingGenerator

EmbeddingGenerator implementations.

EmbeddingGenerator

Bases: Protocol

Base protocol shared by every embedding generator.

Beta

An EmbeddingGenerator provides embeddings for images, image crops, videos, video frames or other sample types. Every sample collection can have a different associated embedding generator. The generator is used at two points:

• During data loading to compute sample embeddings
• During a GUI run to embed text or image search queries

Generators are loaded at startup and need to identify themselves with embedding_space_spec which returns metadata about the loaded model.

To provide custom embeddings, implement one of the protocols below (ImageEmbeddingGenerator and/or VideoEmbeddingGenerator) and register it with set_default_embedding_model before you add a dataset or start the GUI.

embed_text

embed_text(text: str) -> list[float]

Generate an embedding for a text sample.

Beta

Parameters:

Name	Type	Description	Default
text	str	The text to embed.	required

Returns:

Type	Description
list[float]	A list of floats representing the generated embedding.

embedding_space_spec

embedding_space_spec() -> EmbeddingSpaceSpec

Describe the embedding space produced by this generator.

Beta

Returns metadata about the embedding space to be stored in the database. The space_key field is used to match the same embedding space across multiple LightlyStudio runs.

Returns:

Type	Description
EmbeddingSpaceSpec	A specification of the embedding space.

ImageEmbeddingGenerator

EmbeddingGenerator implementations.

ImageEmbeddingGenerator

Bases: EmbeddingGenerator, Protocol

Protocol for embedding images, image crops, and text.

Beta

Implement this to use your own image model in LightlyStudio. Inside embed_images you can run the model on the given file paths or look up precomputed vectors. A registered image generator replaces the built-in image, crop, and text embeddings. It inherits embed_text from EmbeddingGenerator, so keep the text and image encoders in the same embedding space for text-based image and crop search to work.

embed_image_crops

embed_image_crops(image_crops: list[ImageCrop], show_progress: bool = True) -> EmbeddingResult

Generate embeddings for image crops.

Beta

Parameters:

Name	Type	Description	Default
image_crops	list[ImageCrop]	A list of image crop definitions to embed.	required
show_progress	bool	Whether to show a progress bar during embedding.	True

Returns:

Type	Description
EmbeddingResult	An EmbeddingResult with embeddings for the crops of readable files, in the same order as the corresponding input crops.

embed_images

embed_images(filepaths: list[str], show_progress: bool = True) -> EmbeddingResult

Generate embeddings for multiple image samples.

Beta

Parameters:

Name	Type	Description	Default
filepaths	list[str]	A list of fsspec-compatible file paths to the images to embed. Each one is an absolute local path with forward slashes (e.g. /data/cat.jpg) or a remote URI (e.g. s3://bucket/cat.jpg).	required
show_progress	bool	Whether to show a progress bar during embedding.	True

Returns:

Type	Description
EmbeddingResult	An EmbeddingResult with embeddings for the readable files, in the same order as the corresponding input file paths. Use kept_indices to skip any file paths you cannot or do not want to embed.

embed_pil_images

embed_pil_images(images: list[Image], show_progress: bool = True) -> NDArray[float32]

Generate embeddings for in-memory PIL images.

Beta

Used for video frame embedding.

Parameters:

Name	Type	Description	Default
images	list[Image]	PIL images to embed.	required
show_progress	bool	Whether to show a progress bar during embedding.	True

Returns:

Type	Description
NDArray[float32]	A numpy array representing the generated embeddings in the same order as the input images.

embed_text

embed_text(text: str) -> list[float]

Generate an embedding for a text sample.

Beta

Parameters:

Name	Type	Description	Default
text	str	The text to embed.	required

Returns:

Type	Description
list[float]	A list of floats representing the generated embedding.

embedding_space_spec

embedding_space_spec() -> EmbeddingSpaceSpec

Describe the embedding space produced by this generator.

Beta

Returns metadata about the embedding space to be stored in the database. The space_key field is used to match the same embedding space across multiple LightlyStudio runs.

Returns:

Type	Description
EmbeddingSpaceSpec	A specification of the embedding space.

VideoEmbeddingGenerator

EmbeddingGenerator implementations.

VideoEmbeddingGenerator

Bases: EmbeddingGenerator, Protocol

Protocol for embedding videos (and text).

Beta

Implement this to use your own video model in LightlyStudio. A registered video generator replaces the built-in video and text embeddings. It inherits embed_text from EmbeddingGenerator, so keep the text and video encoders in the same embedding space for text-based video search to work.

embed_text

embed_text(text: str) -> list[float]

Generate an embedding for a text sample.

Beta

Parameters:

Name	Type	Description	Default
text	str	The text to embed.	required

Returns:

Type	Description
list[float]	A list of floats representing the generated embedding.

embed_videos

embed_videos(filepaths: list[str]) -> EmbeddingResult

Generate embeddings for multiple video samples.

Beta

Parameters:

Name	Type	Description	Default
filepaths	list[str]	A list of fsspec-compatible file paths to the videos to embed. Each one is an absolute local path with forward slashes (e.g. /data/clip.mp4) or a remote URI (e.g. s3://bucket/clip.mp4).	required

Returns:

Type	Description
EmbeddingResult	An EmbeddingResult with embeddings for the readable videos, in the same order as the corresponding input file paths.

embedding_space_spec

embedding_space_spec() -> EmbeddingSpaceSpec

Describe the embedding space produced by this generator.

Beta

Returns metadata about the embedding space to be stored in the database. The space_key field is used to match the same embedding space across multiple LightlyStudio runs.

Returns:

Type	Description
EmbeddingSpaceSpec	A specification of the embedding space.

Supporting types

EmbeddingSpaceSpec

EmbeddingGenerator implementations.

EmbeddingSpaceSpec dataclass

EmbeddingSpaceSpec(space_key: str, dimension: int)

Identity and shape of the embedding space an embedder produces.

Beta

Stored in the database so the same embedding space can be recognized across LightlyStudio runs.

dimension instance-attribute

dimension: int

Length of each embedding vector this embedder produces.

space_key instance-attribute

space_key: str

Stable identifier for the embedding space.

Two embedders that share a space_key are treated as producing the same embedding space, so their vectors are comparable. Change it whenever the produced vectors become incomparable, e.g. for a model version change. Can be any string, e.g. your-company/model-family@version.

EmbeddingResult

Shared value types for the embedding paths.

Holds the model-agnostic types passed to and returned by embedders: EmbeddingSpaceSpec, EmbeddingResult and ImageCrop. They live in their own module so any caller can depend on them without importing the Embedder classes.

EmbeddingResult dataclass

EmbeddingResult(embeddings: NDArray[float32], kept_indices: list[int])

Embeddings for the inputs that could be read, plus which inputs they cover.

Beta

An embedder skips broken inputs (files it cannot read or decode) instead of failing the whole batch, so embeddings can have fewer rows than the input list. kept_indices gives the position of each row in the input list, in input order. Use it to line up the embeddings with any per-input data you keep on the side, such as sample IDs.

embeddings instance-attribute

embeddings: NDArray[float32]

Float32 array of shape (len(kept_indices), embedding_dimension).

kept_indices instance-attribute

kept_indices: list[int]

Indices into the input list of the inputs that were embedded, in input order.

ImageCrop

EmbeddingGenerator implementations.

ImageCrop dataclass

ImageCrop(filepath: str, x: int, y: int, width: int, height: int)

A rectangular region of an image to embed, given in pixel coordinates.

Beta

filepath instance-attribute

filepath: str

Path to the image the crop is taken from.

height instance-attribute

height: int

Crop height in pixels.

width instance-attribute

width: int

Crop width in pixels.

x instance-attribute

x: int

Left edge of the crop, in pixels from the image's left.

y instance-attribute

y: int

Top edge of the crop, in pixels from the image's top.

Edit this page Report an issue Ask in Discord