Model Evaluation

EvaluationResult

Evaluation interface for image datasets.

EvaluationResult

Bases: BaseModel

Summary of the inputs used by an evaluation run.

Returned by every task method on ImageDatasetEvaluate. The field set is shared across tasks (object detection, classification, segmentation).

Attributes:

Name	Type	Description
evaluation_run_id	UUID	ID of the persisted evaluation run.
sample_count	int	Number of samples included in the evaluation.
gt_annotation_count	int	Number of ground truth annotations used.
pred_annotation_count	int	Number of prediction annotations used.

from_evaluation_data classmethod

from_evaluation_data(data: EvaluationData) -> EvaluationResult

Build a result from the prepared evaluation data.

ObjectDetectionEvaluationConfig

Evaluation interface for image datasets.

ObjectDetectionEvaluationConfig

Bases: BaseModel

Configuration for object-detection evaluation runs.

Attributes:

Name	Type	Description
iou_threshold	float	IoU threshold used by object-detection evaluators. Stored in the run config for reproducibility.
classwise	bool	If True, match predictions and ground truths only within the same annotation class. If False, match globally across all annotation classes.

ClassificationEvaluationConfig

Evaluation interface for image datasets.

ClassificationEvaluationConfig

Bases: BaseModel

Configuration for classification evaluation runs.

Currently has no fields. Placeholder for future task-specific options.

SemanticSegmentationEvaluationConfig

Evaluation interface for image datasets.

SemanticSegmentationEvaluationConfig

Bases: BaseModel

Configuration for semantic-segmentation evaluation runs.

Currently has no fields. Placeholder for future task-specific options.

ImageDatasetEvaluate

Evaluation interface for image datasets.

ImageDatasetEvaluate

ImageDatasetEvaluate(session: Session, collection_id: UUID, sample_ids: Iterable[UUID])

Task-specific evaluation entry points for image datasets.

This facade groups evaluation methods by task (e.g. object detection) and keeps evaluation-specific logic separate from ImageDataset.

Parameters:

Name	Type	Description	Default
session	Session	Database session used by resolver calls.	required
collection_id	UUID	ID of the collection being evaluated.	required
sample_ids	Iterable[UUID]	IDs of the samples selected for evaluation.	required

sample_ids cached property

sample_ids: set[UUID]

The IDs of the samples selected for evaluation, materialized once on first read.

classification

classification(
    name: str,
    gt_annotation_source: str,
    pred_annotation_source: str,
    config: ClassificationEvaluationConfig | None = None,
) -> EvaluationResult

Create a classification evaluation run and persist per-image metrics.

Parameters:

Name	Type	Description	Default
name	str	Display name of the evaluation run.	required
gt_annotation_source	str	Name of the annotation source containing ground truth annotations.	required
pred_annotation_source	str	Name of the annotation source containing predictions.	required
config	ClassificationEvaluationConfig | None	Optional classification evaluation config. If omitted, defaults are used.	None

Returns:

Type	Description
EvaluationResult	Summary of the samples and annotations used by the evaluation.

confusion_matrix

confusion_matrix(run_id: UUID) -> ConfusionMatrix

Return the confusion matrix of an evaluation run.

The matrix aggregates the run's persisted ground-truth/prediction annotation pairings by label. Object detection and classification are supported; segmentation tasks do not produce a confusion matrix.

Parameters:

Name	Type	Description	Default
run_id	UUID	ID of the evaluation run.	required

Returns:

Type	Description
ConfusionMatrix	The confusion matrix, with a shared class axis and synthetic false-positive and false-negative axes.

Raises:

Type	Description
ValueError	If no run with run_id exists in this dataset.
NotImplementedError	If the run's task type has no confusion matrix.

list_runs

list_runs() -> list[EvaluationRunView]

List the evaluation runs stored for this dataset, newest first.

Returns:

Type	Description
list[EvaluationRunView]	A view per run, with its id, name, configuration, creation time, and resolved ground-truth and prediction source names.

object_detection

object_detection(
    name: str,
    gt_annotation_source: str,
    pred_annotation_source: str,
    config: ObjectDetectionEvaluationConfig | None = None,
) -> EvaluationResult

Create an object-detection evaluation run and persist per-image metrics.

Parameters:

Name	Type	Description	Default
name	str	Display name of the evaluation run.	required
gt_annotation_source	str	Name of the annotation source containing ground truth annotations.	required
pred_annotation_source	str	Name of the annotation source containing predictions.	required
config	ObjectDetectionEvaluationConfig | None	Optional object-detection evaluation config. If omitted, defaults are used.	None

Returns:

Type	Description
EvaluationResult	Summary of the samples and annotations used by the evaluation.

semantic_segmentation

semantic_segmentation(
    name: str,
    gt_annotation_source: str,
    pred_annotation_source: str,
    config: SemanticSegmentationEvaluationConfig | None = None,
) -> EvaluationResult

Create a semantic segmentation evaluation run and persist per-image metrics.

Parameters:

Name	Type	Description	Default
name	str	Display name of the evaluation run.	required
gt_annotation_source	str	Name of the annotation source containing ground truth labels.	required
pred_annotation_source	str	Name of the annotation source containing predictions.	required
config	SemanticSegmentationEvaluationConfig | None	Optional semantic segmentation evaluation config. If omitted, defaults are used.	None

Returns:

Type	Description
EvaluationResult	Summary of the samples and annotations used by the evaluation.

EvaluationRunView

EvaluationRun model — persisted output of one evaluation run.

EvaluationRunView

Bases: BaseModel

API view of an evaluation run.

ConfusionMatrix

Dense confusion-matrix payloads for evaluation runs.

ConfusionMatrix

Bases: BaseModel

Confusion matrix built from evaluation_annotation_metric rows.

Each persisted pairing outcome (TP, FP, or FN) increments one cell. Rows follow ground-truth context (class labels, plus the synthetic FP row). Columns follow prediction context (class labels, plus the synthetic FN column). The synthetic axes are always present (with zero counts when unused).

Attributes:

Name	Type	Description
row_labels	list[str]	Ground-truth axis labels (sorted classes, then the FP row).
col_labels	list[str]	Prediction axis labels (sorted classes, then the FN column).
counts	list[list[int]]	Integer cell counts; counts[i][j] is row_labels[i] vs col_labels[j].

Edit this page Report an issue Ask in Discord