Active Learning Scorers

Lightly provides scorers for common computer vision tasks such as image classification, detection, and others.

Active learning scores are scalar values (per sample). High values indicate important samples. For example, for an image classification model, a high score indicates that the sample is hard to classify.

Classification Scores

For classification predictions, Lightly provides three uncertainty scores. They are based on the Active Learning Literature Survey, Section 3.1, page 12f and also explained in this blog post. They are computed out of the probabilities attribute of each prediction.
For all of them, the score is the highest if all classes have equal probability, and the score is 0.0 if the model assigns 100% probability to a single class. Thus the scores are always in the range of [0, 1]. They differ in the number of class confidences they take into account.

• uncertainty_least_confidence
  This score is 1.0 minus the highest confidence prediction. It is high (close to 1.0) when the confidence about the most probable class is low.

• uncertainty_margin
  This score is 1.0 minus the difference between the highest and second-highest confidence prediction. It is high (close to 1.0) when the model cannot decide between the two most probable classes.

• uncertainty_entropy
  This scorer computes the entropy of the prediction. Confidences of all classes are used to compute the entropy of a sample. It is high (close to 1.0) when the model cannot decide between all classes.

Object Detection Scores

For object detection predictions, Lightly computes all three classification scores. They are first computed out of the probabilities attribute of each single object prediction. Then the maximum over all predictions of each sample is taken. E.g. if an object detection prediction has one object with an uncertainty_entropy of 0.9 and the other object has an uncertainty_entropy of 0.5, then the uncertainty_entropy of the whole detection prediction is 0.9.

As the probabilities attribute is optional, the class probabilities are approximated if they are not given: The class given by the category_id attribute is set to the probability given by the score attribute. The probability of the other classes is set uniformly such that it sums up to 1.

Additionally, Lightly provides two object-detection-specific scores:

• object_frequency
  This score is the number of objects in a sample, but counting all objects of the same class
  after the first only with 0.25 instead of 1. It is very useful to select samples with many objects. For example, a sample with 2 bikes and 1 pedestrian has an object_frequency score of 2.25.

• objectness_least_confidence
  This score is 1 - the mean objectness score for each prediction. The objectness score is the score attribute of each object detection prediction. Thus the objectness_least_confidence score is always in the range of [0, 1].

If a sample has no objects, all its object detection scores are set to 0.

Keypoint Detection Scores

All object detection scores are available, as each keypoint detection is simultaneously treated as an object detection by Lightly.

Semantic Segmentation Scores

For semantic segmentation predictions, Lightly computes all three classification scores. They are first computed out of the classification prediction for each segment and then the mean over all segments is taken.