lightly_edge_sdk

`lightly_edge_sdk`#

Package Contents#

Classes#

`LightlyEdge`	Provides access to the main functionalities of the LightlyEdge SDK.
`SelectInfo`	Selection information about a processed frame.
`DiversitySelectInfo`	The result of a diversity strategy.
`AdaptiveDiversitySelectInfo`	The result of an adaptive diversity strategy.
`SimilaritySelectInfo`	The result of a similarity strategy.
`DetectionSelectInfo`	The result of a detection strategy.
`InferenceDeviceType`	Enumeration used to select the inference device.
`LightlyEdgeDetectorConfig`	Configuration options for the Object detector.
`LightlyEdgeConfig`	Configuration options for the LightlyEdge SDK.
`ObjectDetection`	Object detection result.

class lightly_edge_sdk.LightlyEdge(tar_path: str, config: LightlyEdgeConfig | None = None)#

Provides access to the main functionalities of the LightlyEdge SDK.

Offers methods to embed images and texts and make decisions about which frames to select.

property num_diversity_strategies: int#: The number of registered diversity strategies.

property num_adaptive_diversity_strategies: int#: The number of registered adaptive diversity strategies.

property num_similarity_strategies: int#: The number of registered similarity strategies.

property num_detection_strategies: int#: The number of registered detection strategies.

embed_texts(texts: List[str]) → List[List[float]]#

Embeds a list of text strings.

Processes a list of text strings, generating an embedding for each text. The embeddings are returned as a list of lists, where each inner list represents the embedding of a single text.

Parameters:: texts – A list of text strings to embed.
Returns:: A list of lists of floats, where each inner list represents the embedding of a single text.
Raises:: LightlyEdgeError – If an error occurs during the embedding process.

Example

embeddings = lightly_edge.embed_texts(["cat", "dog"])

embed_frame(frame: PIL.Image.Image) → List[float]#

Embeds a PIL image.

Parameters:: frame – A PIL image to embed.
Returns:: An embedding (list of floats). The dimension of the embedding is determined by the model.
Raises:: LightlyEdgeError – If an error occurs during the embedding process.

Example

from PIL.Image import Image

with Image.open("cat.jpg") as frame:
    embedding = lightly_edge.embed_frame(frame=frame)

embed_frame_rgb_bytes(rgb_bytes: bytes, width: int, height: int) → List[float]#

Embeds an RGB image from a byte array.

Parameters:

rgb_bytes – A byte array representing the image data. The data must be in row-major RGB format, with each pixel consisting of three 8-bit integer values (0-255). The length of the byte array must equal width * height * 3.
width – The width of the image.
height – The height of the image.

Returns:

An embedding (list of floats). The dimension of the embedding is determined by the model.

Raises:

LightlyEdgeError – If an error occurs during the embedding process.

Example

rgb_bytes = b'ÿÿÿÿÿÿ'
embedding = lightly_edge.embed_frame_rgb_bytes(rgb_bytes=rgb_bytes, width=2, height=2)

detect(frame: PIL.Image.Image) → List[ObjectDetection]#

Detects objects in a PIL image.

Detection is disabled by default. It has to be enabled by setting the detector_config field of type LightlyEdgeDetectorConfig on the LightlyEdgeConfig object passed to the LightlyEdge constructor. If detection is disabled, this method returns an empty list.

If classifiers are enabled in the LightlyEdgeDetectorConfig, detections are classified for every detection class for which a classifier exists. To bound computation, the number of classifications per detection class is limited by the max_classifications field of the LightlyEdgeDetectorConfig. The most confident detections are classified first.

Parameters:: frame – A PIL image to detect objects in.
Returns:: A list of object detection results as List[ObjectDetection].
Raises:: LightlyEdgeError – If an error occurs during object detection.

Example

from lightly_edge_sdk import LightlyEdge, LightlyEdgeConfig

# Initialize LightlyEdge with object detector enabled.
config = LightlyEdgeConfig.default()
config.detector_config = LightlyEdgeDetectorConfig(
    object_detector_enable=True,
    classifiers_enable=True,
    max_classifications=5,
)
lightly_edge = LightlyEdge(tar_path="lightly_model.tar", config=config)

# Detect objects in the frame.
with Image.open("cat.jpg") as frame:
    detections = lightly_edge.detect(frame=frame)

# Print the detected objects.
class_labels = lightly_edge.detection_class_labels()
for det in detections:
    # Prints e.g. "cat 0.9 10 20 100 200"
    print(class_labels[det.class_id], det.confidence, det.x, det.y, det.w, det.h)

detect_rgb_bytes(rgb_bytes: bytes, width: int, height: int) → List[ObjectDetection]#

Detects objects in an RGB image.

Parameters:

rgb_bytes – A byte array representing the image data. The data must be in row-major RGB format, with each pixel consisting of three 8-bit integer values (0-255). The length of the byte array must equal width * height * 3.
width – The width of the image.
height – The height of the image.

Returns:

A list of object detection results as List[ObjectDetection].

Raises:

LightlyEdgeError – If an error occurs during object detection.

Example

from lightly_edge_sdk import LightlyEdge, LightlyEdgeConfig

# Initialize LightlyEdge with object detector enabled.
config = LightlyEdgeConfig.default()
config.detector_config = LightlyEdgeDetectorConfig(
    object_detector_enable=True,
    classifiers_enable=True,
    max_classifications=5,
)
lightly_edge = LightlyEdge(tar_path="lightly_model.tar", config=config)

# Detect objects in the frame.
rgb_bytes = b'ÿÿÿÿÿÿ'
detections = lightly_edge.detect_rgb_bytes(rgb_bytes=rgb_bytes, width=2, height=2)

# Print the detected objects.
class_labels = lightly_edge.detection_class_labels()
for det in detections:
    # Prints e.g. "cat 0.9 10 20 100 200"
    print(class_labels[det.class_id], det.confidence, det.x, det.y, det.w, det.h)

detection_class_labels() → List[str]#

Returns the class labels of the object detector.

Returns:: A list of class labels as strings. The zero-based index of the class label in the list corresponds to the class ID in the object detection results.

Example

class_labels = lightly_edge.detection_class_labels()

detection_subclass_labels(class_id: int) → List[str] | None#

Returns subclass labels for a given detection class.

Subclass labels are generated by a classifier specific to the class.

Parameters:: class_id – The class ID of the object detector.
Returns:: A list of subclass labels as strings. The zero-based index of the subclass label in the list corresponds to the subclass ID in the object detection results. Returns None if a classifier is not available for the given class.

Example

subclass_labels = lightly_edge.detection_subclass_labels(class_id=5)

insert_into_embedding_database(embedding: List[float]) → None#

Inserts an embedding into the embedding database.

Embedding database is used by the diversity strategy. If an embedding is inserted into the database the strategy will not anymore select frames with embeddings similar to it. You should call insert_into_embedding_database for frames marked by should_select with diversity strategy.

Parameters:: embedding – Embedding of a frame. Can be obtained using embed_frame.
Raises:: LightlyEdgeError – If the database is full or the embedding has a different dimension than other embeddings in the database.

clear_embedding_database() → None#

Clears the embedding database.

Removes all embeddings from the database.

register_diversity_strategy(min_distance: float) → None#

Registers a diversity strategy.

A diversity strategy selects frames that are different from previously selected frames. The min_distance parameter must be between 0 and 1 and controls how different the new frame must be from the previously selected frames: Only frames with a distance greater than the min_distance will be selected. To remember selected frames across multiple calls, the user must fill the database with embeddings by calling insert_into_embedding_database.

Parameters:: min_distance – Only frame embeddings with a distance greater than min_distance to all embeddings in the database will be selected. Must be between 0 and 1. A higher value will result in fewer frames being selected.
Raises:: LightlyEdgeError – If min_distance is not between 0 and 1.

Example

# Register a diversity strategy.
lightly_edge.register_diversity_strategy(min_distance=0.5)

# Embed a frame.
embedding = lightly_edge.embed_frame(frame=frame)

# Check if the frame should be selected.
select_info = lightly_edge.should_select(embedding)

# Add to the embedding database if the image is selected.
if select_info.diversity[0].should_select:
    lightly_edge.insert_into_embedding_database(embedding)

register_adaptive_diversity_strategy(target_ratio: float, buffer_max_length: int | None = None) → None#

Registers an adaptive diversity strategy.

An adaptive diversity strategy selects frames that are different from previously selected frames given a target_ratio. With every new frame, the strategy calculates the distance to the closest frame in the embedding database. It also keeps track of the past buffer_max_length observed distances.

Intuitively, the strategy selects a frame if its distance is in the top target_ratio percentile of the observed distances.

More precisely, the strategy computes an “adjusted target ratio” which is larger than the target_ratio if the overall selection ratio is too low, and vice versa. The “adjusted target ratio” is then used to compute the percentile of the observed distances. This assures the strategy compensates for long-term under- or oversampling.

To initialise the algorithm, the first image is always selected, and the second image is never selected. Convergence to the target ratio is expected after a few thousand images.

Parameters:

target_ratio – The target ratio of selected images. Must be between 0 and 1.
buffer_max_length –
The maximum number of past distances to store. Must be at least 2. If set to None, default value of 3000 is used for the the buffer length.

Lower values result in faster adaptation to changes in the selection ratio, while larger values select samples diverse over a longer time span. Recommended values are in thousands.

Raises:

OverflowError – If buffer_max_length is negative.
LightlyEdgeError – If target_ratio is not between 0 and 1 or buffer_max_length is less than 2.

register_similarity_strategy(query_embedding: List[float], max_distance: float) → None#

Registers a similarity strategy.

A similarity strategy selects frames that are similar to a query embedding. The max_distance parameter must be between 0 and 1 and controls how similar the embeddings must be for a frame to be selected: Only frames with a distance less than the max_distance will be selected.

Parameters:

query_embedding – Query embedding. Can be obtained using embed_texts.
max_distance – Only frame embeddings with a distance less than max_distance to the query embedding will be selected. Must be between 0 and 1. A higher value will result in more frames being selected.

Raises:

LightlyEdgeError – If max_distance is not between 0 and 1.

Example

# Embed query texts.
query_embeddings = lightly_edge.embed_texts(["cat", "dog"])

lightly_edge.register_similarity_strategy(query_embeddings[0], 0.3)
lightly_edge.register_similarity_strategy(query_embeddings[1], 0.4)

# Embed a frame.
embedding = lightly_edge.embed_frame(frame=frame)

# Check if the frame should be selected.
select_info = lightly_edge.should_select(embedding)
is_cat = select_info.similarity[0].should_select
is_dog = select_info.similarity[1].should_select

register_detection_strategy(class_id: int, threshold: float, subclass_id: int | None = None) → None#

Registers a detection strategy.

A detection strategy selects frames which contain an object detection of a given class. The class_id must match the ID in ObjectDetection returned by the detect function and the detection confidence value must be larger than threshold.

If subclass_id is specified, moreover the detection subclass ID must match.

Parameters:

class_id – The class ID of the object detector.
threshold – The confidence threshold for the detection. Images with detections with higher confidence will be selected.
subclass_id – The subclass ID of the object class. If set to None subclass ID is not considered.

Raises:

LightlyEdgeError – If threshold is not between 0 and 1.
OverflowError – If class_id or subclass_id is negative.

Example

# Register a detection strategy.
lightly_edge.register_detection_strategy(class_id=0, threshold=0.5)
lightly_edge.register_detection_strategy(class_id=1, threshold=0.6, subclass_id=0)

# Embed and run detections on a frame.
embedding = lightly_edge.embed_frame(frame=frame)
detections = lightly_edge.detect(frame=frame)

# Check if the frame should be selected.
select_info = lightly_edge.should_select(embedding, detections)
has_class_0 = select_info.detection[0].should_select
has_class_2_subclass_0 = select_info.detection[1].should_select

should_select(embedding: List[float], detections: List[ObjectDetection] | None = None) → SelectInfo#

Checks if a frame should be selected.

Checks if a frame should be selected based on its embedding, detections and registered strategies. In particular, it iterates over all registered strategies and creates a selection info object of for each strategy. Each strategy selection info object contains the selection result (true or false) and metadata about the selection.

Parameters:

embedding – Embedding of a frame. Can be obtained using embed_frame.
detections – A list of object detections. Can be obtained using detect. If set to None the frame is considered to have no detections.

Returns:

A selection info object containing the selection result and metadata for each registered strategy.

Raises:

LightlyEdgeError – If an error occurs during the selection process.

Example

# Register a diversity strategy.
lightly_edge.register_diversity_strategy(min_distance=0.5)

# Embed a frame and run detections.
embedding = lightly_edge.embed_frame(frame=frame)
detections = lightly_edge.detect(frame=frame)

# Check if the frame should be selected.
select_info = lightly_edge.should_select(embedding, detections)

assert len(select_info.diversity) == 1 # One diversity strategy is registered.
assert len(select_info.similarity) == 0 # No similarity strategy is registered.
assert len(select_info.detection) == 0 # No detection strategy is registered.
is_selected = select_info.diversity[0].selected

exception lightly_edge_sdk.LightlyEdgeError(message: str)#

Custom error type raised by LightlyEdge SDK functions.

__str__() → str#: Return str(self).

class lightly_edge_sdk.SelectInfo#

Selection information about a processed frame.

diversity: List[DiversitySelectInfo]#: A list of diversity strategy selection information.

adaptive_diversity: List[AdaptiveDiversitySelectInfo]#: A list of adaptive diversity strategy selection information.

similarity: List[SimilaritySelectInfo]#: A list of similarity strategy selection information.

detection: List[DetectionSelectInfo]#: A list of detection strategy selection information.

class lightly_edge_sdk.DiversitySelectInfo#

The result of a diversity strategy.

min_distance: float#: The minimum distance to the embeddings in the database for the frame to be selected.

should_select: bool#: A boolean indicating whether the frame should be selected.

class lightly_edge_sdk.AdaptiveDiversitySelectInfo#

The result of an adaptive diversity strategy.

min_distance: float#: The minimum distance to the embeddings in the database for the frame to be selected.

should_select: bool#: A boolean indicating whether the frame should be selected.

class lightly_edge_sdk.SimilaritySelectInfo#

The result of a similarity strategy.

distance: float#: The distance to the query embedding for the frame to be selected.

should_select: bool#: A boolean indicating whether the frame should be selected.

class lightly_edge_sdk.DetectionSelectInfo#

The result of a detection strategy.

class_id: int#: The class ID set by the detection strategy.

subclass_id: int | None#: The subclass ID set by the detection strategy.

threshold: float#: The confidence threshold set by the detection strategy.

should_select: bool#: A boolean indicating whether the frame should be selected.

class lightly_edge_sdk.InferenceDeviceType#

Enumeration used to select the inference device.

GPUFallbackCPU = 0#

CPU = 1#

GPU = 2#

class lightly_edge_sdk.LightlyEdgeDetectorConfig(object_detector_enable: bool, classifiers_enable: bool, max_classifications: int)#

Configuration options for the Object detector.

object_detector_enable: bool#: If set to true, object detector is initialized and LightlyEdge.detect method can be used.

classifiers_enable: bool#: If set to true, classifier is initialized and LightlyEdge.detect method will provide also classification if classifiers are available.

max_classifications: int#: Sets the maximum number of classifications done per frame per detection class.

static default() → LightlyEdgeDetectorConfig#: Returns the default configuration with object_detector_enable=False, classifiers_enable=False and max_classifications=5.

class lightly_edge_sdk.LightlyEdgeConfig(detector_config: LightlyEdgeDetectorConfig, inference_device_type: InferenceDeviceType)#

Configuration options for the LightlyEdge SDK.

detector_config: LightlyEdgeDetectorConfig#: Object detector options.

inference_device_type: InferenceDeviceType#: Sets the device used for the inference.

static default() → LightlyEdgeConfig#: Returns the default configuration with detector_config=LightlyEdgeDetectorConfig.default() and inference_device_type=GPUFallbackCPU.

class lightly_edge_sdk.ObjectDetection(class_id: int, subclass_id: int | None, confidence: float, x: int, y: int, w: int, h: int)#

Object detection result.

The bounding box is quaranteed to fit within the image. More precisely:

0 <= x <= x + w <= image_width
0 <= y <= y + h <= image_height

class_id: int#: The class ID of the detected object. Use LightlyEdge.detection_class_labels() to convert it to a human-readable label.

subclass_id: int | None#: The subclass ID of the object class. Set to None if this object has not been classified. Use LightlyEdge.classification_class_labels(class_id) to convert it to a human-readable label.

confidence: float#: The confidence score between 0.0 and 1.0 of the detected object.

x: int#: The x-coordinate of the top-left corner of the bounding box in pixel coordinates.

y: int#: The y-coordinate of the top-left corner of the bounding box in pixel coordinates.

w: int#: The width of the bounding box in pixel coordinates.

h: int#: The height of the bounding box in pixel coordinates.

lightly_edge_sdk

Contents

lightly_edge_sdk#

Package Contents#

Classes#

`lightly_edge_sdk`#