Active Learning Scorers
Lightly provides scorers for common computer vision tasks such as image classification, detection, and others.
Active learning scores are scalar values (per sample). High values indicate important samples. For example, for an image classification model, a high score indicates that the sample is hard to classify.
Classification Scores
For classification predictions, Lightly provides three uncertainty scores. They are based on the Active Learning Literature Survey, Section 3.1, page 12f and also explained in this blog post. They are computed out of the probabilities
attribute of each prediction.
For all of them, the score is the highest if all classes have equal probability, and the score is 0.0 if the model assigns 100% probability to a single class. Thus the scores are always in the range of [0, 1]
. They differ in the number of class confidences they take into account.
-
uncertainty_least_confidence
This score is 1.0 minus the highest confidence prediction. It is high (close to 1.0) when the confidence about the most probable class is low. -
uncertainty_margin
This score is 1.0 minus the difference between the highest and second-highest confidence prediction. It is high (close to 1.0) when the model cannot decide between the two most probable classes. -
uncertainty_entropy
This scorer computes the entropy of the prediction. Confidences of all classes are used to compute the entropy of a sample. It is high (close to 1.0) when the model cannot decide between all classes.
Object Detection Scores
For object detection predictions, Lightly computes all three classification scores. They are first computed out of the probabilities
attribute of each single object prediction. Then the maximum over all predictions of each sample is taken. E.g. if an object detection prediction has one object with an uncertainty_entropy
of 0.9 and the other object has an uncertainty_entropy
of 0.5, then the uncertainty_entropy
of the whole detection prediction is 0.9.
As the probabilities
attribute is optional, the class probabilities are approximated if they are not given: The class given by the category_id
attribute is set to the probability given by the score
attribute. The probability of the other classes is set uniformly such that it sums up to 1.
Additionally, Lightly provides two object-detection-specific scores:
-
object_frequency
This score is the number of objects in a sample, but counting all objects of the same class
after the first only with 0.25 instead of 1. It is very useful to select samples with many objects. For example, a sample with 2 bikes and 1 pedestrian has anobject_frequency
score of 2.25. -
objectness_least_confidence
This score is 1 - the mean objectness score for each prediction. The objectness score is thescore
attribute of each object detection prediction. Thus theobjectness_least_confidence
score is always in the range of[0, 1]
.
If a sample has no objects, all its object detection scores are set to 0.
Keypoint Detection Scores
All object detection scores are available, as each keypoint detection is simultaneously treated as an object detection by Lightly.
Semantic Segmentation Scores
For semantic segmentation predictions, Lightly computes all three classification scores. They are first computed out of the classification prediction for each segment and then the mean over all segments is taken.
Updated 3 months ago