Model Evaluation

EvaluationResult

Evaluation interface for image datasets.

EvaluationResult

Bases: BaseModel

Summary of the inputs used by an evaluation run.

Returned by every task method on ImageDatasetEvaluate. The field set is shared across tasks (object detection, classification, segmentation).

Attributes:

Name	Type	Description
evaluation_run_id	UUID	ID of the persisted evaluation run.
sample_count	int	Number of samples included in the evaluation.
gt_annotation_count	int	Number of ground truth annotations used.
pred_annotation_count	int	Number of prediction annotations used.

from_evaluation_data classmethod

from_evaluation_data(data: EvaluationData) -> EvaluationResult

Build a result from the prepared evaluation data.

ObjectDetectionEvaluationConfig

Evaluation interface for image datasets.

ObjectDetectionEvaluationConfig

Bases: BaseModel

Configuration for object-detection evaluation runs.

Attributes:

Name	Type	Description
iou_threshold	float	IoU threshold used by object-detection evaluators. Stored in the run config for reproducibility.
classwise	bool	If True, match predictions and ground truths only within the same annotation class. If False, match globally across all annotation classes.

ClassificationEvaluationConfig

Evaluation interface for image datasets.

ClassificationEvaluationConfig

Bases: BaseModel

Configuration for classification evaluation runs.

Currently has no fields. Placeholder for future task-specific options.

InstanceSegmentationEvaluationConfig

Evaluation interface for image datasets.

InstanceSegmentationEvaluationConfig

Bases: BaseModel

Configuration for instance-segmentation evaluation runs.

Attributes:

Name	Type	Description
iou_threshold	float	Mask IoU threshold used to match predictions to ground truths. Stored in the run config for reproducibility.
classwise	bool	If True, match predictions and ground truths only within the same annotation class. If False, match across all annotation classes.

SemanticSegmentationEvaluationConfig

Evaluation interface for image datasets.

SemanticSegmentationEvaluationConfig

Bases: BaseModel

Configuration for semantic-segmentation evaluation runs.

Currently has no fields. Placeholder for future task-specific options.

ImageDatasetEvaluate

Evaluation interface for image datasets.

ImageDatasetEvaluate

ImageDatasetEvaluate(session: Session, collection_id: UUID, sample_ids: Iterable[UUID])

Task-specific evaluation entry points for image datasets.

This facade groups evaluation methods by task (e.g. object detection) and keeps evaluation-specific logic separate from ImageDataset.

Parameters:

Name	Type	Description	Default
session	Session	Database session used by resolver calls.	required
collection_id	UUID	ID of the collection being evaluated.	required
sample_ids	Iterable[UUID]	IDs of the samples selected for evaluation.	required

sample_ids cached property

sample_ids: set[UUID]

The IDs of the samples selected for evaluation, materialized once on first read.

classification

classification(
    name: str,
    gt_annotation_source: str,
    pred_annotation_source: str,
    config: ClassificationEvaluationConfig | None = None,
) -> EvaluationResult

Create a classification evaluation run and persist per-image metrics.

Parameters:

Name	Type	Description	Default
name	str	Display name of the evaluation run.	required
gt_annotation_source	str	Name of the annotation source containing ground truth annotations.	required
pred_annotation_source	str	Name of the annotation source containing predictions.	required
config	ClassificationEvaluationConfig | None	Optional classification evaluation config. If omitted, defaults are used.	None

Returns:

Type	Description
EvaluationResult	Summary of the samples and annotations used by the evaluation.

confusion_matrix

confusion_matrix(run_id: UUID) -> ConfusionMatrix

Return the confusion matrix of an evaluation run.

The matrix aggregates the run's persisted ground-truth/prediction annotation pairings by label. Object detection, classification, and instance segmentation are supported; semantic segmentation does not produce a confusion matrix.

Parameters:

Name	Type	Description	Default
run_id	UUID	ID of the evaluation run.	required

Returns:

Type	Description
ConfusionMatrix	The confusion matrix, with a shared class axis and synthetic false-positive and false-negative axes.

Raises:

Type	Description
ValueError	If no run with run_id exists in this dataset.
NotImplementedError	If the run's task type has no confusion matrix.

instance_segmentation

instance_segmentation(
    name: str,
    gt_annotation_source: str,
    pred_annotation_source: str,
    config: InstanceSegmentationEvaluationConfig | None = None,
) -> EvaluationResult

Create an instance-segmentation evaluation run and persist per-image metrics.

Parameters:

Name	Type	Description	Default
name	str	Display name of the evaluation run.	required
gt_annotation_source	str	Name of the annotation source containing ground truth masks.	required
pred_annotation_source	str	Name of the annotation source containing predictions.	required
config	InstanceSegmentationEvaluationConfig | None	Optional instance segmentation evaluation config. If omitted, defaults are used.	None

Returns:

Type	Description
EvaluationResult	Summary of the samples and annotations used by the evaluation.

list_runs

list_runs() -> list[EvaluationRunView]

List the evaluation runs stored for this dataset, newest first.

Returns:

Type	Description
list[EvaluationRunView]	A view per run, with its id, name, configuration, creation time, and resolved ground-truth and prediction source names.

metrics

metrics(run_id: UUID) -> EvaluationMetrics

Return aggregate metrics for an evaluation run.

Derives per-class and micro-averaged precision, recall, and F1 from the run's confusion matrix, plus accuracy for classification runs. Object detection and classification are supported; segmentation tasks have no confusion matrix and so no confusion-derived metrics.

Parameters:

Name	Type	Description	Default
run_id	UUID	ID of the evaluation run.	required

Returns:

Type	Description
EvaluationMetrics	The aggregate metrics for the run.

Raises:

Type	Description
ValueError	If no run with run_id exists in this dataset.
NotImplementedError	If the run's task type has no confusion matrix.

object_detection

object_detection(
    name: str,
    gt_annotation_source: str,
    pred_annotation_source: str,
    config: ObjectDetectionEvaluationConfig | None = None,
) -> EvaluationResult

Create an object-detection evaluation run and persist per-image metrics.

Parameters:

Name	Type	Description	Default
name	str	Display name of the evaluation run.	required
gt_annotation_source	str	Name of the annotation source containing ground truth annotations.	required
pred_annotation_source	str	Name of the annotation source containing predictions.	required
config	ObjectDetectionEvaluationConfig | None	Optional object-detection evaluation config. If omitted, defaults are used.	None

Returns:

Type	Description
EvaluationResult	Summary of the samples and annotations used by the evaluation.

semantic_segmentation

semantic_segmentation(
    name: str,
    gt_annotation_source: str,
    pred_annotation_source: str,
    config: SemanticSegmentationEvaluationConfig | None = None,
) -> EvaluationResult

Create a semantic segmentation evaluation run and persist per-image metrics.

Parameters:

Name	Type	Description	Default
name	str	Display name of the evaluation run.	required
gt_annotation_source	str	Name of the annotation source containing ground truth labels.	required
pred_annotation_source	str	Name of the annotation source containing predictions.	required
config	SemanticSegmentationEvaluationConfig | None	Optional semantic segmentation evaluation config. If omitted, defaults are used.	None

Returns:

Type	Description
EvaluationResult	Summary of the samples and annotations used by the evaluation.

EvaluationRunView

EvaluationRun model — persisted output of one evaluation run.

EvaluationRunView

Bases: BaseModel

API view of an evaluation run.

ConfusionMatrix

Dense confusion-matrix payloads for evaluation runs.

ConfusionMatrix

Bases: BaseModel

Confusion matrix built from evaluation_annotation_metric rows.

Each persisted pairing outcome (TP, FP, or FN) increments one cell. Rows follow ground-truth context (class labels, plus the synthetic FP row). Columns follow prediction context (class labels, plus the synthetic FN column). The synthetic axes are always present (with zero counts when unused).

Attributes:

Name	Type	Description
row_labels	list[str]	Ground-truth axis labels (sorted classes, then the FP row).
col_labels	list[str]	Prediction axis labels (sorted classes, then the FN column).
counts	list[list[int]]	Integer cell counts; counts[i][j] is row_labels[i] vs col_labels[j].

EvaluationMetrics

Aggregate evaluation metrics derived from a confusion matrix.

EvaluationMetrics

Bases: BaseModel

Aggregate metrics for an evaluation run, derived from its confusion matrix.

Attributes:

Name	Type	Description
per_class	list[ClassMetrics]	Per-class precision, recall, F1, and support.
precision	float	Micro-averaged precision over the pooled per-class counts.
recall	float	Micro-averaged recall over the pooled per-class counts.
f1	float	Harmonic mean of the micro-averaged precision and recall.
accuracy	float | None	Fraction of correctly classified samples. Set for classification runs; None for object detection, where predictions and ground truths are matched rather than compared one to one.

ClassMetrics

Bases: BaseModel

Per-class precision, recall, and F1 for one evaluation run.

Attributes:

Name	Type	Description
label	str	Annotation label name.
precision	float	True positives over predictions of this class. 0 when the class is never predicted.
recall	float	True positives over ground truths of this class. 0 when the class has no ground truth.
f1	float	Harmonic mean of precision and recall. 0 when both are 0.
support	int	Number of ground-truth annotations of this class.

Edit this page Report an issue Ask in Discord