Embeddings

Beta API

The Embeddings API is in beta. Its interface may change in future releases without a deprecation period.

LightlyStudio embeds your data automatically on ingestion. To supply your own embeddings — either computed on the fly or loaded from a precomputed store — subclass the capability interfaces for the inputs you can embed and register the embedder with register_default_embedder. The registration must happen before you load a dataset or before the GUI is started.

See the Embeddings page for more details.

register_default_embedder

Public API for configuring embedders.

register_default_embedder

register_default_embedder(
    embedder: Embedder, for_capabilities: Set[Capability] | None = None
) -> None

Register an embedder as the default for the capabilities it implements.

Beta

Call this in either of two cases:

• Before creating a dataset, so that ingestion embeds with embedder. This draws on the capabilities for the data being added, such as embedding images, image crops, or videos.
• Before starting the GUI, so that search embeds queries with embedder. This draws on the capabilities for the query kinds, such as embedding text, or images for reverse-image search. Search resolves the embedder by the collection's stored embedding space, so embedder must share that space to take effect.

An embedder that calls an embedding server, such as one that RemoteEmbedder.connect builds, needs the first case only. When ingestion first embeds a collection with it, the dataset stores the URL and the API key of the server in plain text. A later process, such as lightly-studio gui, then embeds search queries on that server with no registration. A server that the dataset already stores for the space is kept. To change the URL or the key, use register_remote_embedder.

Parameters:

Name	Type	Description	Default
embedder	Embedder	The embedder to register. Its embedding space is read from embedder.embedding_space_spec().	required
for_capabilities	Set[Capability] | None	Capabilities for which this embedder becomes the default choice. By default, all implemented capabilities are updated. An empty set registers the embedder without changing the defaults. Requested capabilities the embedder does not implement are ignored.	None

Raises:

Type	Description
ValueError	If the embedder implements no capability, or if it shares a space with an embedder the registry already holds but does not match its spec.

register_remote_embedder

Public API for configuring embedders.

register_remote_embedder

register_remote_embedder(dataset: Dataset[Any], url: str, api_key: str | None = None) -> None

Embed search queries of a dataset on a remote embedding server.

Beta

The server must produce an embedding space that the dataset already holds embeddings in, such as one filled by an embedder registered with register_default_embedder. Text and image search in that space then embed the query on the server, starting with the next query. If the server embeds image bytes, images that are added to the dataset later are embedded on the server too. An embedder registered in this process for the same space still serves the capabilities it implements. A server that embeds no images leaves the space without image search. The dataset stores url and api_key in plain text.

A remote embedder that register_default_embedder registered before the import stores its server without this call. Call this function for a space that was filled with no stored server, or to change the URL or the key of a server.

See examples/example_remote_embedder.py for a runnable example.

Parameters:

Name	Type	Description	Default
dataset	Dataset[Any]	The dataset whose embedding space the server produces.	required
url	str	The base URL of a server that lightly_studio_serve.serve runs, or of another server that speaks the same protocol.	required
api_key	str | None	The bearer token of the server, or None if the server needs none.	None

Raises:

Type	Description
RemoteEmbedderError	If the server cannot be reached or used, or produces an embedding space that the dataset does not hold with the same dimension. Nothing is stored then.

Capability interfaces

An embedder subclasses Embedder through one interface per input it can embed. Subclass only the capabilities your model provides.

Embedder

Capability-split embedder interfaces.

Defines the Embedder base class and one abstract subclass per input a model can embed: images and crops by path, videos, PIL images, text, and images by bytes. A concrete model implements only the capabilities it supports, and callers pick an embedder by the capability they need.

An embedder subclasses only the interfaces that it implements. The class list is the advertisement. An embedder cannot advertise a capability and then not serve it.

Embedder

Bases: ABC

Base class for every embedder.

Beta

Subclasses add one abstract method per input they support. A concrete model implements the subclasses for the capabilities it provides.

An embedder must accept calls from more than one thread, because the server runs a request in a worker thread. Hold a lock in the methods when the model cannot.

ready property

ready: bool

Whether the model is loaded and can answer requests.

The default suits a model that loads when the object exists. Override the property when the weights load in the background. While the value is False, /v1/describe reports ready: false and the embed endpoints answer 503. The property covers the weights only. embedding_space_spec answers either way.

embedding_space_spec abstractmethod

embedding_space_spec() -> EmbeddingSpaceSpec

Describe the embedding space this embedder produces.

The method must answer as soon as the object exists, also while ready is False. /v1/describe reports the space to a client that waits for the model. The identity of a space comes from the architecture, so a model knows it before the weights arrive.

Returns:

Type	Description
EmbeddingSpaceSpec	Metadata identifying the embedding space, stored so the same space can be recognized across LightlyStudio runs.

ImagePathEmbedder

Capability-split embedder interfaces.

Defines the Embedder base class and one abstract subclass per input a model can embed: images and crops by path, videos, PIL images, text, and images by bytes. A concrete model implements only the capabilities it supports, and callers pick an embedder by the capability they need.

An embedder subclasses only the interfaces that it implements. The class list is the advertisement. An embedder cannot advertise a capability and then not serve it.

ImagePathEmbedder

Bases: Embedder

Embeds images read from an fsspec path (local, s3://, ...).

Beta

ready property

ready: bool

Whether the model is loaded and can answer requests.

The default suits a model that loads when the object exists. Override the property when the weights load in the background. While the value is False, /v1/describe reports ready: false and the embed endpoints answer 503. The property covers the weights only. embedding_space_spec answers either way.

embed_images abstractmethod

embed_images(paths: list[str]) -> EmbeddingResult

Embed a batch of images given by path.

Parameters:

Name	Type	Description	Default
paths	list[str]	fsspec paths of the images to embed.	required

Returns:

Type	Description
EmbeddingResult	The embeddings and the indices of the inputs they cover.

embedding_space_spec abstractmethod

embedding_space_spec() -> EmbeddingSpaceSpec

Describe the embedding space this embedder produces.

The method must answer as soon as the object exists, also while ready is False. /v1/describe reports the space to a client that waits for the model. The identity of a space comes from the architecture, so a model knows it before the weights arrive.

Returns:

Type	Description
EmbeddingSpaceSpec	Metadata identifying the embedding space, stored so the same space can be recognized across LightlyStudio runs.

ImageCropPathEmbedder

Capability-split embedder interfaces.

Defines the Embedder base class and one abstract subclass per input a model can embed: images and crops by path, videos, PIL images, text, and images by bytes. A concrete model implements only the capabilities it supports, and callers pick an embedder by the capability they need.

An embedder subclasses only the interfaces that it implements. The class list is the advertisement. An embedder cannot advertise a capability and then not serve it.

ImageCropPathEmbedder

Bases: Embedder

Embeds crops of stored images, each an image path plus a pixel box.

Beta

ready property

ready: bool

Whether the model is loaded and can answer requests.

The default suits a model that loads when the object exists. Override the property when the weights load in the background. While the value is False, /v1/describe reports ready: false and the embed endpoints answer 503. The property covers the weights only. embedding_space_spec answers either way.

embed_image_crops abstractmethod

embed_image_crops(crops: list[ImageCrop]) -> EmbeddingResult

Embed a batch of image crops given by path and pixel box.

Parameters:

Name	Type	Description	Default
crops	list[ImageCrop]	The crops to embed.	required

Returns:

Type	Description
EmbeddingResult	The embeddings and the indices of the inputs they cover.

embedding_space_spec abstractmethod

embedding_space_spec() -> EmbeddingSpaceSpec

Describe the embedding space this embedder produces.

The method must answer as soon as the object exists, also while ready is False. /v1/describe reports the space to a client that waits for the model. The identity of a space comes from the architecture, so a model knows it before the weights arrive.

Returns:

Type	Description
EmbeddingSpaceSpec	Metadata identifying the embedding space, stored so the same space can be recognized across LightlyStudio runs.

VideoPathEmbedder

Capability-split embedder interfaces.

Defines the Embedder base class and one abstract subclass per input a model can embed: images and crops by path, videos, PIL images, text, and images by bytes. A concrete model implements only the capabilities it supports, and callers pick an embedder by the capability they need.

An embedder subclasses only the interfaces that it implements. The class list is the advertisement. An embedder cannot advertise a capability and then not serve it.

VideoPathEmbedder

Bases: Embedder

Embeds videos read from an fsspec path.

Beta

ready property

ready: bool

Whether the model is loaded and can answer requests.

The default suits a model that loads when the object exists. Override the property when the weights load in the background. While the value is False, /v1/describe reports ready: false and the embed endpoints answer 503. The property covers the weights only. embedding_space_spec answers either way.

embed_videos abstractmethod

embed_videos(paths: list[str]) -> EmbeddingResult

Embed a batch of videos given by path.

Parameters:

Name	Type	Description	Default
paths	list[str]	fsspec paths of the videos to embed.	required

Returns:

Type	Description
EmbeddingResult	The embeddings and the indices of the inputs they cover.

embedding_space_spec abstractmethod

embedding_space_spec() -> EmbeddingSpaceSpec

Describe the embedding space this embedder produces.

The method must answer as soon as the object exists, also while ready is False. /v1/describe reports the space to a client that waits for the model. The identity of a space comes from the architecture, so a model knows it before the weights arrive.

Returns:

Type	Description
EmbeddingSpaceSpec	Metadata identifying the embedding space, stored so the same space can be recognized across LightlyStudio runs.

ImagePILEmbedder

Capability-split embedder interfaces.

Defines the Embedder base class and one abstract subclass per input a model can embed: images and crops by path, videos, PIL images, text, and images by bytes. A concrete model implements only the capabilities it supports, and callers pick an embedder by the capability they need.

An embedder subclasses only the interfaces that it implements. The class list is the advertisement. An embedder cannot advertise a capability and then not serve it.

ImagePILEmbedder

Bases: Embedder

Embeds images represented as PIL images.

Beta

ready property

ready: bool

Whether the model is loaded and can answer requests.

The default suits a model that loads when the object exists. Override the property when the weights load in the background. While the value is False, /v1/describe reports ready: false and the embed endpoints answer 503. The property covers the weights only. embedding_space_spec answers either way.

embed_images_pil abstractmethod

embed_images_pil(images: list[Image]) -> EmbeddingResult

Embed a batch of PIL images.

Parameters:

Name	Type	Description	Default
images	list[Image]	The PIL images to embed.	required

Returns:

Type	Description
EmbeddingResult	The embeddings and the indices of the inputs they cover.

embedding_space_spec abstractmethod

embedding_space_spec() -> EmbeddingSpaceSpec

Describe the embedding space this embedder produces.

The method must answer as soon as the object exists, also while ready is False. /v1/describe reports the space to a client that waits for the model. The identity of a space comes from the architecture, so a model knows it before the weights arrive.

Returns:

Type	Description
EmbeddingSpaceSpec	Metadata identifying the embedding space, stored so the same space can be recognized across LightlyStudio runs.

TextEmbedder

Capability-split embedder interfaces.

Defines the Embedder base class and one abstract subclass per input a model can embed: images and crops by path, videos, PIL images, text, and images by bytes. A concrete model implements only the capabilities it supports, and callers pick an embedder by the capability they need.

An embedder subclasses only the interfaces that it implements. The class list is the advertisement. An embedder cannot advertise a capability and then not serve it.

TextEmbedder

Bases: Embedder

Embeds text queries.

Beta

ready property

ready: bool

Whether the model is loaded and can answer requests.

The default suits a model that loads when the object exists. Override the property when the weights load in the background. While the value is False, /v1/describe reports ready: false and the embed endpoints answer 503. The property covers the weights only. embedding_space_spec answers either way.

embed_text abstractmethod

embed_text(texts: list[str]) -> EmbeddingResult

Embed a batch of text strings.

Parameters:

Name	Type	Description	Default
texts	list[str]	The strings to embed.	required

Returns:

Type	Description
EmbeddingResult	The embeddings and the indices of the inputs they cover.

embedding_space_spec abstractmethod

embedding_space_spec() -> EmbeddingSpaceSpec

Describe the embedding space this embedder produces.

The method must answer as soon as the object exists, also while ready is False. /v1/describe reports the space to a client that waits for the model. The identity of a space comes from the architecture, so a model knows it before the weights arrive.

Returns:

Type	Description
EmbeddingSpaceSpec	Metadata identifying the embedding space, stored so the same space can be recognized across LightlyStudio runs.

ImageBytesEmbedder

Capability-split embedder interfaces.

Defines the Embedder base class and one abstract subclass per input a model can embed: images and crops by path, videos, PIL images, text, and images by bytes. A concrete model implements only the capabilities it supports, and callers pick an embedder by the capability they need.

An embedder subclasses only the interfaces that it implements. The class list is the advertisement. An embedder cannot advertise a capability and then not serve it.

ImageBytesEmbedder

Bases: Embedder

Embeds images passed as raw bytes.

Beta

For callers that have no path to read, such as an upload. JPEG, PNG and WebP only.

ready property

ready: bool

Whether the model is loaded and can answer requests.

The default suits a model that loads when the object exists. Override the property when the weights load in the background. While the value is False, /v1/describe reports ready: false and the embed endpoints answer 503. The property covers the weights only. embedding_space_spec answers either way.

embed_image_bytes abstractmethod

embed_image_bytes(images: list[bytes]) -> EmbeddingResult

Embed a batch of images given as encoded bytes.

Read the format from the header of the data. Do not trust the caller.

Parameters:

Name	Type	Description	Default
images	list[bytes]	Encoded image bytes (JPEG, PNG or WebP).	required

Returns:

Type	Description
EmbeddingResult	The embeddings and the indices of the inputs they cover.

embedding_space_spec abstractmethod

embedding_space_spec() -> EmbeddingSpaceSpec

Describe the embedding space this embedder produces.

The method must answer as soon as the object exists, also while ready is False. /v1/describe reports the space to a client that waits for the model. The identity of a space comes from the architecture, so a model knows it before the weights arrive.

Returns:

Type	Description
EmbeddingSpaceSpec	Metadata identifying the embedding space, stored so the same space can be recognized across LightlyStudio runs.

Supporting types

EmbeddingSpaceSpec

Shared value types for the embedding paths.

Holds the model-agnostic types passed to and returned by embedders: EmbeddingSpaceSpec, EmbeddingResult and ImageCrop. They live in their own module so any caller can depend on them without importing the Embedder classes.

EmbeddingSpaceSpec dataclass

EmbeddingSpaceSpec(space_key: str, dimension: int)

Identity and shape of the embedding space an embedder produces.

Beta

Stored in the database so the same embedding space can be recognized across LightlyStudio runs.

dimension instance-attribute

dimension: int

Length of each embedding vector this embedder produces.

space_key instance-attribute

space_key: str

Stable identifier for the embedding space.

Two embedders that share a space_key are treated as producing the same embedding space, so their vectors are comparable. Change it whenever the produced vectors become incomparable, e.g. for a model version change. Can be any string, e.g. your-company/model-family@version.

EmbeddingResult

Shared value types for the embedding paths.

Holds the model-agnostic types passed to and returned by embedders: EmbeddingSpaceSpec, EmbeddingResult and ImageCrop. They live in their own module so any caller can depend on them without importing the Embedder classes.

EmbeddingResult dataclass

EmbeddingResult(embeddings: NDArray[float32], kept_indices: list[int])

Embeddings for the inputs that could be read, plus which inputs they cover.

Beta

An embedder skips broken inputs (files it cannot read or decode) instead of failing the whole batch, so embeddings can have fewer rows than the input list. kept_indices gives the position of each row in the input list, in input order. Use it to line up the embeddings with any per-input data you keep on the side, such as sample IDs.

embeddings instance-attribute

embeddings: NDArray[float32]

Float32 array of shape (len(kept_indices), embedding_dimension).

kept_indices instance-attribute

kept_indices: list[int]

Indices into the input list of the inputs that were embedded, in input order.

ImageCrop

Shared value types for the embedding paths.

Holds the model-agnostic types passed to and returned by embedders: EmbeddingSpaceSpec, EmbeddingResult and ImageCrop. They live in their own module so any caller can depend on them without importing the Embedder classes.

ImageCrop dataclass

ImageCrop(filepath: str, x: int, y: int, width: int, height: int)

A rectangular region of an image to embed, given in pixel coordinates.

Beta

filepath instance-attribute

filepath: str

Path to the image the crop is taken from.

height instance-attribute

height: int

Crop height in pixels.

width instance-attribute

width: int

Crop width in pixels.

x instance-attribute

x: int

Left edge of the crop, in pixels from the image's left.

y instance-attribute

y: int

Top edge of the crop, in pixels from the image's top.

Serving

lightly_studio_serve serves an embedder over HTTP. An import can embed through it with register_default_embedder, and register_remote_embedder can connect a filled dataset to it. See Serving an embedder from a remote server.

serve

Makes the HTTP endpoints of the protocol from an embedder.

create_app reads the capability classes that the embedder implements. It mounts one endpoint for each class, and it always mounts /v1/describe. serve runs that application with uvicorn. Version 1 serves the texts, images/bytes and videos/bytes transports.

The ASGI middleware in lightly_studio_serve.middleware applies the bearer token, the readiness of the model and the request size limit. That middleware runs before the router.

serve

serve(
    embedder: Embedder,
    host: str = "127.0.0.1",
    port: int = 8080,
    api_key: str | None = None,
    limits: ServerLimits | None = None,
    ssl_certfile: str | Path | None = None,
    ssl_keyfile: str | Path | None = None,
) -> None

Serve an embedder over HTTP until the process stops.

Parameters:

Name	Type	Description	Default
embedder	Embedder	The model to serve. Its capability classes set which endpoints exist. A text-only embedder has no image route and no video route. It must accept calls from more than one thread.	required
host	str	The interface to bind. The default binds loopback only. Give "0.0.0.0" to accept requests from other hosts.	'127.0.0.1'
port	int	The TCP port to bind.	8080
api_key	str | None	The token that a client must send as Authorization: Bearer <api_key>. None leaves the server open. That is safe only on a loopback address or inside a network that you trust. The server trims the whitespace around the token, because a client cannot send it, and it gives a warning when it does.	None
limits	ServerLimits | None	The limits to report and to apply. The default is 999 items and 32 MiB for each request.	None
ssl_certfile	str | Path | None	The PEM certificate chain for HTTPS. Give this file for any address that is not loopback. You can also end TLS at a proxy in front of the server. Without one of these, the bearer token crosses the network in clear text. serve gives a warning when it has no certificate.	None
ssl_keyfile	str | Path | None	The private key for ssl_certfile. Omit it if the certificate file already holds the key.	None

Raises:

Type	Description
ValueError	If api_key is an empty or blank string, if you give ssl_keyfile without ssl_certfile, if limits names a max_batch_size that the multipart parser cannot deliver, or if the embedder implements no capability that version 1 serves.

create_app

Makes the HTTP endpoints of the protocol from an embedder.

create_app reads the capability classes that the embedder implements. It mounts one endpoint for each class, and it always mounts /v1/describe. serve runs that application with uvicorn. Version 1 serves the texts, images/bytes and videos/bytes transports.

The ASGI middleware in lightly_studio_serve.middleware applies the bearer token, the readiness of the model and the request size limit. That middleware runs before the router.

create_app

create_app(
    embedder: Embedder, api_key: str | None = None, limits: ServerLimits | None = None
) -> FastAPI

Build the application serve runs.

Use this function to mount the protocol in an application of your own. You can also use it to test an embedder with a test client and no open port.

Parameters:

Name	Type	Description	Default
embedder	Embedder	The model to serve. Its capability classes set which endpoints exist. A text-only embedder has no image route and no video route. It must accept calls from more than one thread.	required
api_key	str | None	The token that a client must send as Authorization: Bearer <api_key>. The server trims the whitespace around the token, because a client cannot send it, and it gives a warning when it does.	None
limits	ServerLimits | None	The limits to report and to apply. The default is 999 items and 32 MiB for each request.	None

Returns:

Type	Description
FastAPI	An application that serves /v1/describe and the capabilities of the embedder.

Raises:

Type	Description
ValueError	If api_key is an empty or blank string, because such a server looks authenticated but accepts an empty token, if limits names a max_batch_size that the multipart parser cannot deliver, or if the embedder implements no capability that version 1 serves.

ServerLimits

Wire models of the LightlyStudio embedding protocol, version 1.

The module holds the request bodies, the response bodies, the route paths and the multipart field name. The server in this package sends these models. lightly-studio reads the same models. One definition keeps the two halves equal.

A server serves the capability {subject}_{transport} at POST /v1/embed/{subjects}/{transport}. The capability text has no transport segment. Text is already a payload.

EmbeddingsResponse holds every rule that a response body shows. lightly_studio_serve.validation adds only the rules that the body cannot show.

ServerLimits

Bases: BaseModel

The limits that the server applies and reports. A client does not have to guess them.

max_batch_size class-attribute instance-attribute

max_batch_size: int = Field(default=DEFAULT_MAX_BATCH_SIZE, gt=0)

The largest number of items in one request.

The model holds no ceiling. A client reads this field from a server that it does not control, and that server can read its batch with another parser. create_app applies MAX_BATCH_SIZE_CEILING to the server of this package.

max_request_bytes class-attribute instance-attribute

max_request_bytes: int = Field(default=DEFAULT_MAX_REQUEST_BYTES, gt=0)

The largest request body that the server accepts, in bytes.

Protocol

Any server that speaks the protocol works, in any language. Version 1 has these endpoints. lightly_studio_serve.protocol holds the paths and the request and response models.

Endpoint	Body
GET /v1/describe	Response: DescribeResponse. The space key, dimension, capabilities and limits.
POST /v1/embed/texts	Request: EmbedTextsRequest. Response: EmbeddingsResponse.
POST /v1/embed/images/bytes	Request: multipart, one files part per image. Response: EmbeddingsResponse.
POST /v1/embed/videos/bytes	Request: multipart, one files part per video. Response: EmbeddingsResponse.

A server mounts only the embed endpoints of the capabilities it reports. If the server has an API key, each request must carry Authorization: Bearer <api_key>.

Conformance check

lightly-studio-serve conformance <url> [--api-key KEY] [--probe-timeout SECONDS]

The command reads /v1/describe, then sends one probe to each capability that the server reports.

Argument	Description
url	The address of the server, for example http://127.0.0.1:8080.
--api-key	The bearer token. Default: the LIGHTLY_STUDIO_SERVE_API_KEY variable.
--probe-timeout	The seconds one probe can take. Default: 60.

Exit code	Meaning
0	The server passes.
1	The server fails, or the URL is not valid.
2	The arguments are not valid.

Edit this page Report an issue Ask in Discord