Active Learning Scorers
LightlyOne provides scorers for common computer vision tasks such as image classification, detection, and others.
Active learning scores are scalar values (per sample). High values indicate important samples. For example, for an image classification model, a high score indicates that the sample is hard to classify.
Classification Scores
For classification predictions, LightlyOne provides three uncertainty scores. They are based on the Active Learning Literature Survey, Section 3.1, page 12f, and are also explained in this blog post. They are computed out of the probabilities
attribute of each prediction.
For all of them, the score is the highest if all classes have equal probability, and the score is 0.0 if the model assigns 100% probability to a single class. Thus the scores are always in the range of [0, 1]
. They differ in the number of class confidences they take into account.
-
uncertainty_least_confidence
This score is 1.0 minus the highest confidence prediction. It is high (close to 1.0) when the confidence about the most probable class is low. -
uncertainty_margin
This score is 1.0 minus the difference between the highest and second-highest confidence prediction. It is high (close to 1.0) when the model cannot decide between the two most probable classes. -
uncertainty_entropy
This scorer computes the entropy of the prediction. Confidences of all classes are used to compute the entropy of a sample. It is high (close to 1.0) when the model cannot decide between all classes.
Object Detection Scores
For object detection predictions, LightlyOne computes all three classification scores. They are first computed out of the probabilities
attribute of each single object prediction. Then the maximum overall predictions of each sample is taken. E.g. if an object detection prediction has one object with an uncertainty_entropy
of 0.9 and the other object has an uncertainty_entropy
of 0.5, then the uncertainty_entropy
of the whole detection prediction is 0.9.
As the probabilities
attribute is optional, the class probabilities are approximated if they are not given: The class given by the category_id
attribute is set to the probability given by the score
attribute. The probability of the other classes is set uniformly such that it sums up to 1.
Additionally, LightlyOne provides two object-detection-specific scores:
- objectness_least_confidence
This score is 1 - the mean objectness score for each prediction. The objectness score is thescore
attribute of each object detection prediction. Thus theobjectness_least_confidence
score is always in the range of[0, 1]
. - object_frequency
Pending deprecation: Please use a prediction selection configuration withCATEGORY_COUNT
for a more flexible approach instead.
This score is the number of objects in a sample, but counting all objects of the same class
after the first only with 0.25 instead of 1. For example, a sample with 2 bikes and 1 pedestrian has anobject_frequency
score of 2.25.
If a sample has no objects, all its object detection scores are set to 0.
Keypoint Detection Scores
All object detection scores work for keypoint detection as well. This is because each keypoint detection is simultaneously treated as an object detection by Lightly.
Instance Segmentation Scores
All object detection scores work for instance segmentation as well. This is because each instance segmentation is simultaneously treated as an object detection by Lightly.
Semantic Segmentation Scores
All classification scores work for semantic segmentation as well. The classification scores are first computed for each segment out of its classification prediction and then the mean over all segments is taken.
Updated 4 months ago