03 Diversity Selection

03 Diversity Selection#

In this guide, we will use the “diversity” strategy. It selects frames which are visually different from those observed before. This is useful to collect novel scenarios on your edge device.

We showcase it on the same images we used in 02 Similarity Search. This time, we expect only one image selected from each location.

matterhorn1.jpg matterhorn2.jpg london1.jpg london2.jpg

Project Setup#

The setup is identical to 02 Similarity Search guide.

  • Inside of the getting_started folder create a new folder 03_diversity_selection with images subfolder.

    mkdir -p getting_started/03_diversity_selection/images

  • Right-click and download the four images above into 03_diversity_selection/images

After completing this guide, the folder structure will be as follows:

getting_started
├── 02_similarity_search
│   └── ...
├── 03_diversity_selection
│   ├── images/
│   └── main.py
└── lightly_model.tar

Note

In this example we use the model version lightly_model_14.tar. You might need to adjust the thresholds in this tutorial if your model version differs.

Run a Complete Example#

Create 03_diversity_selection/main.py and copy the contents below into it. We will first run the example, and explain it right after.

from lightly_edge_sdk import LightlyEdge
from PIL import Image


# Initialize the LightlyEdge SDK.
print("Initializing LightlyEdge...\n")
lightly_edge = LightlyEdge(path="../lightly_model.tar")

# Register a diversity strategy for the text "mountains".
threshold = 0.1
lightly_edge.register_diversity_strategy(min_distance=threshold)

# Iterate over the images
image_paths = [
    "images/matterhorn1.jpg",
    "images/matterhorn2.jpg",
    "images/london1.jpg",
    "images/london2.jpg",
]
for image_path in image_paths:
    # Embed the image and check if it should be selected.
    with Image.open(image_path) as frame:
        print(f"Loading image: {image_path}")
        image_embedding = lightly_edge.embed_frame(frame=frame)

    select_info = lightly_edge.should_select(embedding=image_embedding)
    diversity_select_info = select_info.diversity[0]

    # Add to the embedding database if the image is selected.
    if diversity_select_info.should_select:
        lightly_edge.insert_into_embedding_database(embedding=image_embedding)

    # Print whether the image is selected.
    print(f"Should select: {diversity_select_info.should_select}")
    print(f"Min. distance: {diversity_select_info.min_distance:.2f}\n")

print("Program successfully finished.")

Run it:

# Enter the project folder.
cd 03_diversity_selection

# Run the Python script
python main.py

The output should be similar to the following, the distances might slightly differ on your machine architecture:

Initializing LightlyEdge...

Loading image: images/matterhorn1.jpg
Should select: True
Min. distance: 1.00

Loading image: images/matterhorn2.jpg
Should select: False
Min. distance: 0.04

Loading image: images/london1.jpg
Should select: True
Min. distance: 0.24

Loading image: images/london2.jpg
Should select: False
Min. distance: 0.08

Program successfully finished.

We see that only the first Matterhorn image and the first London image are selected, which is what we expected. We explain the rest of the output below.

Diversity Selection#

There are several steps needed to set up diversity selection. Let’s start with the initialization code:

# Initialize the LightlyEdge SDK.
print("Initializing LightlyEdge...\n")
lightly_edge = LightlyEdge(path="../lightly_model.tar")

# Register a diversity strategy for the text "mountains".
threshold = 0.1
lightly_edge.register_diversity_strategy(min_distance=threshold)

LightlyEdge is first initialized from a TAR archive. Recall that we can register any number of selection strategies with LightlyEdge. We register a single diversity strategy with lightly_edge_sdk.LightlyEdge.register_diversity_strategy. The function accepts a single argument: min_distance. An observed image is selected only if it is further than min_distance from all previously observed images in the embedding space. The distances are based on cosine similarity and range from 0 (closest) to 1 (furthest).

Next, images are processed in a for loop:

for image_path in image_paths:
    # Embed the image and check if it should be selected.
    with Image.open(image_path) as frame:
        print(f"Loading image: {image_path}")
        image_embedding = lightly_edge.embed_frame(frame=frame)

    select_info = lightly_edge.should_select(embedding=image_embedding)
    diversity_select_info = select_info.diversity[0]

    # Add to the embedding database if the image is selected.
    if diversity_select_info.should_select:
        lightly_edge.insert_into_embedding_database(embedding=image_embedding)

    # Print whether the image is selected.
    print(f"Should select: {diversity_select_info.should_select}")
    print(f"Min. distance: {diversity_select_info.min_distance:.2f}\n")

The code first embeds the image and calls lightly_edge_sdk.LightlyEdge.should_select which returns lightly_edge_sdk.SelectInfo. The structure contains the decision of each registered strategy whether a frame should be selected.

We load the decision for the single strategy we registered into diversity_select_info. LightlyEdge does not by default remember the images that are observed, we have to manually insert the embeddings into the embedding database by calling lightly_edge_sdk.LightlyEdge.insert_into_embedding_database.

Finally, we print whether the image was selected and its distance to the closest embedding in the database.

How To Choose Diversity Parameters#

Consecutive images in a video stream are often very similar. Set the minDistance parameter to a small value, in the range 0.005 to 0.1. The value should be adjusted based on the desired level of diversity in the selected images. A higher value selects more diverse images, leading to a lower selection rate.

With lightly_model_14.tar model, a value in the range 0.008 to 0.015 leads to a selection rate of about 0.5 fps. This is approximate, low values should be used if there is low variation in input video, e.g. a car moving slowly during a rush hour or driving in a rural area in the night.

Next Steps#

In the next section you will learn how to run object detection with LightlyEdge.