05 Adaptive Diversity Selection

In this guide, we introduce the "adaptive diversity" strategy.

A disadvantage of Diversity Selection is that we have to specify a threshold to control the sensitivity of the selection. With adaptive diversity, instead, we specify a target ratio of selected images. The strategy then optimizes two goals - select diverse images, and keep the ratio of selected images close to the target.

For example, if we have a dataset of 1000 images and want to select the 20% most diverse images, we set target_ratio to 0.2.

We showcase the strategy on the same images as in the previous guides. Note however that this strategy is designed for input datasets with thousands of images, like e.g. a camera stream.

Project Setup

Code for this tutorial is provided in examples/05_adaptive_diversity_selection directory. Before starting this tutorial, copy the model file to examples/lightly_model.tar and verify that your project layout is as follows:

lightly_edge_sdk_cpp
lightly_edge_sdk_cpp
├── ...
└── examples
    ├── ...
    ├── 05_adaptive_diversity_selection
    │   ├── CMakeLists.txt
    │   ├── images
    │   │   ├── london1.jpg
    │   │   ├── london2.jpg
    │   │   ├── matterhorn1.jpg
    │   │   └── matterhorn2.jpg
    │   ├── main.cpp
    │   └── stb_image.h
    └── lightly_model.tar

Build and Run a Complete Example

See below the content of the main.cpp file. We will first run the example, and explain it right after.

// main.cpp
#include <iostream>
#define STB_IMAGE_IMPLEMENTATION
#include "stb_image.h"
#include "lightly_edge_sdk.h"
using namespace lightly_edge_sdk;
 
// Loads an image using stb_image and returns it as a lightly_edge_sdk::Frame struct.
Frame load_image(std::string image_path) {
  std::cout << "Loading image: " << image_path << std::endl;
  int width, height, channels;
  unsigned char *data = stbi_load(image_path.c_str(), &width, &height, &channels, 0);
  if (data == nullptr) {
    throw std::runtime_error("Failed to load image.");
  }
  // Create a Frame struct.
  return Frame(width, height, data);
}
 
int main() {
  // Initialize the LightlyEdge SDK.
  std::cout << "Initializing LightlyEdge..." << std::endl << std::endl;
  lightly_edge_sdk::LightlyEdgeConfig config = lightly_edge_sdk::default_config();
  LightlyEdge lightly_edge =
      LightlyEdge::new_from_tar("../lightly_model.tar", config);
 
  // Register an adaptive diversity strategy.
  float target_ratio = 0.7;
  lightly_edge.register_adaptive_diversity_strategy(target_ratio);
 
  // Iterate over the images.
  std::vector<std::string> image_paths = {
    "images/matterhorn1.jpg",
    "images/matterhorn2.jpg",
    "images/london1.jpg",
    "images/london2.jpg",
  };
  for (const auto &path : image_paths) {
    // Load the image.
    Frame frame = load_image(path);
 
    // Embed the image and check if it should be selected.
    std::vector<float> image_embedding = lightly_edge.embed_frame(frame);
    SelectInfo select_info = lightly_edge.should_select(image_embedding, {});
 
    // Add to the embedding database if the image is selected.
    AdaptiveDiversitySelectInfo adaptive_diversity_select_info = select_info.adaptive_diversity[0];
    if (adaptive_diversity_select_info.should_select) {
      lightly_edge.insert_into_embedding_database(image_embedding);
    }
 
    // Print the information about the image.
    std::cout << "  should_select: " << adaptive_diversity_select_info.should_select << std::endl;
    std::cout << "  min_distance: " << adaptive_diversity_select_info.min_distance << std::endl << std::endl;
 
    // The image is no longer needed, free the memory.
    stbi_image_free(frame.rgbImageData_);
  }
 
  std::cout << "Program successfully finished." << std::endl;
  return 0;
}

Build and run:

# Enter the project folder.
cd 05_adaptive_diversity_selection
 
# Configure CMake. This will create a `build` subfolder.
cmake -B build
 
# Build using configuration from the `build` subfolder.
cmake --build build
 
# Run (Linux variant)
./build/main
# Or run (Windows variant)
.\build\[build_type]\main.exe
# where [build_type] is either Release or Debug.

The output should be similar to the following, the distances might slightly differ on your machine architecture:

Initializing LightlyEdge...
 
Loading image: images/matterhorn1.jpg
  should_select: 1
  min_distance: 1
 
Loading image: images/matterhorn2.jpg
  should_select: 0
  min_distance: 0.0369358
 
Loading image: images/london1.jpg
  should_select: 1
  min_distance: 0.24345
 
Loading image: images/london2.jpg
  should_select: 0
  min_distance: 0.084776
 
Program successfully finished.

With the chosen parameters, the output is identical to the output of Diversity Selection example.

Note

In this example we use the model version lightly_model_14.tar. You might need to adjust the thresholds in this tutorial if your model version differs.

Adaptive Diversity Selection

There are several steps needed to set up adaptive diversity selection. Note that in a real application the exceptions should be handled in a try-catch block. For clarity, we annotate the return types.

Image loading is identical to the previous guides. What differs is how LightlyEdge is the set up. Let's start with the initialization code at the beginning of the main function:

// Initialize the LightlyEdge SDK.
std::cout << "Initializing LightlyEdge..." << std::endl << std::endl;
LightlyEdge lightly_edge = LightlyEdge::new_from_tar("lightly_model.tar");
 
// Register an adaptive diversity strategy.
float target_ratio = 0.7;
lightly_edge.register_adaptive_diversity_strategy(target_ratio);

LightlyEdge is first initialized from a TAR archive. Recall that we can register any number of selection strategies with LightlyEdge. We register a single adaptive diversity strategy with lightly_edge_sdk::LightlyEdge::register_adaptive_diversity_strategy.

The function accepts two arguments: target_ratio and optional buffer_max_length.

The target_ratio is a float between 0 and 1 indicating what proportion of images should be selected. Because the strategy simultaneously tries to select diverse images, the ratio of actual selected images will fluctuate. Over long datasets of thousands of images the actual selected ratio should converge to the target ratio.

The optional buffer_max_length argument is set to 3000 by default. The strategy internally keeps track of information from buffer_max_length previously observed images. It determines whether the next image should be selected based on this buffer and the current selection ratio.

Note that for initialisation, the first image is always marked as selected, and the second one as not selected.

We set the target ratio to 70% and keep the second argument default. The images are then processed in the loop in the main function analogously as in the previous guide:

for (const auto &path : image_paths) {
  // Load the image.
  Frame frame = load_image(path);
 
  // Embed the image and check if it should be selected.
  std::vector<float> image_embedding = lightly_edge.embed_frame(frame);
  SelectInfo select_info = lightly_edge.should_select(image_embedding);
 
  // Add to the embedding database if the image is selected.
  AdaptiveDiversitySelectInfo adaptive_diversity_select_info = select_info.adaptive_diversity[0];
  if (adaptive_diversity_select_info.should_select) {
    lightly_edge.insert_into_embedding_database(image_embedding);
  }
 
  // Print the information about the image.
  std::cout << "  should_select: " << adaptive_diversity_select_info.should_select << std::endl;
  std::cout << "  min_distance: " << adaptive_diversity_select_info.min_distance << std::endl << std::endl;
 
  // The image is no longer needed, free the memory.
  stbi_image_free(frame.rgbImageData_);
}

The code first embeds the image and calls lightly_edge_sdk::LightlyEdge::should_select which returns lightly_edge_sdk::SelectInfo. The structure contains the decision of each registered strategy whether a frame should be selected.

We load the decision for the single strategy we registered into adaptive_diversity_select_info. LightlyEdge does not by default remember the images that are observed, we have to manually insert the embeddings into the embedding database by calling lightly_edge_sdk::LightlyEdge::insert_into_embedding_database.

Finally, we print whether the image was selected and its distance to the closest embedding in the database.

How To Choose Adaptive Diversity Parameters

Set target_ratio to the proportion of data you want to select. Note that the actual selection ratio might fluctuate, however over stretches of thousands of images it should converge to the specified value.

You can trade off between selecting diverse images versus accurately matching the target ratio by tweaking the buffer_max_length parameter. Setting it to a large value will give the algorithm large memory, and therefore prioritise diversity. Smaller values will on the other hand more dynamically adapt to current data distribution. We however recommend to set the value to at least ~100 to capture at least some statistics of the observed data.

Next Steps

In the next section you will learn how to run Object Detection with LightlyEdge.

Previous	Next
Diversity Selection	Object Detection