DatasetQuery¶
DatasetQuery¶
Dataset query utilities for filtering, ordering, and slicing samples.
DatasetQuery ¶
DatasetQuery(dataset: CollectionTable, session: Session, sample_class: type[T] | None = None)
Bases: Generic[T]
Class for executing a query on a dataset.
Filtering, ordering, and slicing samples in a dataset¶
Allows filtering, ordering, and slicing of samples in a dataset.
This class can be accessed via calling .query() on a Dataset instance.
dataset : Dataset = ...
query = dataset.query()
match(), order_by(), and slice() methods can be chained in this order.
You can also access the methods directly on the Dataset instance:
dataset.match(...) # shorthand for dataset.query().match(...)
The object is converted to a SQL query that is lazily evaluated when iterating over it or converting it to a list.
match() - Filtering samples¶
Filtering is done via the match() method.
from lightly_studio.core.dataset_query import ImageSampleField
query_1 = dataset.query().match(ImageSampleField.width > 100)
query_2 = dataset.query().match(ImageSampleField.tags.contains('cat'))
from lightly_studio.core.dataset_query import ImageSampleField, AND, OR
query = dataset.query().match(
AND(
ImageSampleField.height < 200,
OR(
ImageSampleField.file_name == 'image.png',
ImageSampleField.file_name == 'image2.png',
)
)
)
order_by() - Ordering samples¶
The results can be ordered by using order_by(). For tie-breaking, multiple fields
can be provided. The first field has the highest priority. The default is
ascending order. To order in descending order, use OrderByField(...).desc().
from lightly_studio.core.dataset_query import OrderByField, ImageSampleField
query = query.order_by(
OrderByField(ImageSampleField.width),
OrderByField(ImageSampleField.file_name).desc()
)
slice() - Slicing samples¶
Slicing can be applied via slice() or bracket notation.
query = query.slice(offset=10, limit=20)
query = query[10:30] # equivalent to slice(offset=10, limit=20)
Usage of the filtered, ordered and sliced query¶
Iterating and converting to list¶
Finally, the query can be executed by iterating over it or converting to a list.
for sample in query:
print(sample.file_name)
samples = query.to_list()
Sample class. They are writable, and
changes to them will be persisted to the database.
Adding tags to matching samples¶
The filtered set can also be used to add a tag to all matching samples.
query.add_tag('my_tag')
Selecting a subset of samples using sampling¶
A Sampling interface can be created from the current query results. It will only select the samples matching the current query at the time of calling sampling().
# Choosing 100 diverse samples from the 'cat' tag.
# Save them under the tag name "diverse_cats".
sampling = dataset.query().match(
ImageSampleField.tags.contains('cat')
).sampling()
sampling.diverse(100, "diverse_cats")
Exporting the query results¶
An export interface can be created from the current query results.
query = dataset.query().match(...)
export = dataset.export(query)
export.to_coco_object_detections('/path/to/coco.json')
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset
|
CollectionTable
|
The dataset to query. |
required |
session
|
Session
|
Database session for executing queries. |
required |
sample_class
|
type[T] | None
|
The class of type |
None
|
__getitem__ ¶
__getitem__(key: _SliceType) -> Self
Enable bracket notation for slicing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
_SliceType
|
A slice object (e.g., [10:20], [:50], [100:]). |
required |
Returns:
| Type | Description |
|---|---|
Self
|
Self with slice applied. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If key is not a slice object. |
ValueError
|
If slice contains unsupported features or conflicts with existing slice. |
__iter__ ¶
__iter__() -> Iterator[T]
Iterate over the query results.
Returns:
| Type | Description |
|---|---|
Iterator[T]
|
Iterator of Sample objects from the database. |
add_tag ¶
add_tag(tag_name: str) -> None
Add a tag to all samples returned by this query.
First, creates the tag if it doesn't exist. Then applies the tag to all samples that match the current query filters. Samples already having that tag are unchanged, as the database prevents duplicates.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
tag_name
|
str
|
Name of the tag to add to matching samples. |
required |
match ¶
match(match_expression: MatchExpression) -> Self
Store a field condition for filtering.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
match_expression
|
MatchExpression
|
Defines the filter. |
required |
Returns:
| Type | Description |
|---|---|
Self
|
Self for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If match() has already been called on this instance. |
order_by ¶
order_by(*order_by: OrderByExpression) -> Self
Store ordering expressions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
order_by
|
OrderByExpression
|
One or more ordering expressions. They are applied in order. E.g. first ordering by sample width and then by sample file_name will only order the samples with the same sample width by file_name. |
()
|
Returns:
| Type | Description |
|---|---|
Self
|
Self for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If order_by() has already been called on this instance. |
sampling ¶
sampling() -> Sampling
Sampling interface for this query.
The returned Sampling snapshots the current query results immediately. Mutating the query after calling this method will therefore not affect the samples used by that Sampling instance.
Returns:
| Type | Description |
|---|---|
Sampling
|
Sampling interface operating on the current query result snapshot. |
slice ¶
slice(offset: int = 0, limit: int | None = None) -> Self
Apply offset and limit to results.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
offset
|
int
|
Number of items to skip from beginning (default: 0). |
0
|
limit
|
int | None
|
Maximum number of items to return (None = no limit). |
None
|
Returns:
| Type | Description |
|---|---|
Self
|
Self for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If slice() has already been called on this instance. |
to_list ¶
to_list() -> list[T]
Execute the query and return the results as a list.
Returns:
| Type | Description |
|---|---|
list[T]
|
List of Sample objects from the database. |
ImageSampleField¶
Fields for querying sample properties in the dataset query system.
ImageSampleField ¶
Providing access to predefined sample fields for queries.
It is used for the query.match(...) and query.order_by(...) methods of the
DatasetQuery class.
from lightly_studio.core.dataset_query import ImageSampleField, OrderByField
query = dataset.query()
query.match(ImageSampleField.tags.contains("cat"))
query.order_by(OrderByField(ImageSampleField.file_path_abs))
samples = query.to_list()
file_path_abs
class-attribute
instance-attribute
¶
file_path_abs = ComparableField(col(file_path_abs))
VideoSampleField¶
Fields for querying video sample properties in the dataset query system.
VideoSampleField ¶
Providing access to predefined sample fields for queries.
It is used for the query.match(...) and query.order_by(...) methods of the
DatasetQuery class.
from lightly_studio.core.dataset_query import VideoSampleField, OrderByField
query = dataset.query()
query.match(VideoSampleField.tags.contains("cat"))
query.order_by(OrderByField(VideoSampleField.file_path_abs))
samples = query.to_list()
file_path_abs
class-attribute
instance-attribute
¶
file_path_abs = ComparableField(col(file_path_abs))
ClassificationField¶
Classes and functions for building complex queries against classification annotations.
ClassificationQuery¶
Classes and functions for building complex queries against classification annotations.
ClassificationQuery ¶
ClassificationQuery(*criteria: MatchExpression)
Bases: MatchExpression
Query if a sample has a classification matching a criterion.
Example
ClassificationQuery(ClassificationField.class_name == "cat")
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
criteria
|
MatchExpression
|
The classification criteria to combine. |
()
|
ObjectDetectionField¶
Classes and functions for building complex queries against object detection annotations.
ObjectDetectionField ¶
Providing access to predefined object detection fields for queries.
class_name
class-attribute
instance-attribute
¶
class_name = ForeignComparableField(
column=col(annotation_label_name), relationship=annotation_label
)
height
class-attribute
instance-attribute
¶
height = ForeignNumericalField(column=col(height), relationship=object_detection_details)
width
class-attribute
instance-attribute
¶
width = ForeignNumericalField(column=col(width), relationship=object_detection_details)
x
class-attribute
instance-attribute
¶
x = ForeignNumericalField(column=col(x), relationship=object_detection_details)
y
class-attribute
instance-attribute
¶
y = ForeignNumericalField(column=col(y), relationship=object_detection_details)
ObjectDetectionQuery¶
Classes and functions for building complex queries against object detection annotations.
ObjectDetectionQuery ¶
ObjectDetectionQuery(*criteria: MatchExpression)
Bases: MatchExpression
Query if a sample has an object detection matching a criterion.
Example
ObjectDetectionQuery(ObjectDetectionField.width <= 100)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
criteria
|
MatchExpression
|
The object detection criteria to combine. |
()
|
SegmentationMaskField¶
Classes and functions for building complex queries against segmentation mask annotations.
SegmentationMaskField ¶
Providing access to predefined segmentation mask fields for queries.
class_name
class-attribute
instance-attribute
¶
class_name = ForeignComparableField(
column=col(annotation_label_name), relationship=annotation_label
)
height
class-attribute
instance-attribute
¶
height = ForeignNumericalField(column=col(height), relationship=segmentation_details)
width
class-attribute
instance-attribute
¶
width = ForeignNumericalField(column=col(width), relationship=segmentation_details)
x
class-attribute
instance-attribute
¶
x = ForeignNumericalField(column=col(x), relationship=segmentation_details)
y
class-attribute
instance-attribute
¶
y = ForeignNumericalField(column=col(y), relationship=segmentation_details)
SegmentationMaskQuery¶
Classes and functions for building complex queries against segmentation mask annotations.
SegmentationMaskQuery ¶
SegmentationMaskQuery(*criteria: MatchExpression)
Bases: MatchExpression
Query if a sample has a segmentation mask matching a criterion.
Example
SegmentationMaskQuery(SegmentationMaskField.width <= 100)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
criteria
|
MatchExpression
|
The segmentation mask criteria to combine. |
()
|
SampleEvaluationQuery¶
Classes and functions for building queries against sample evaluation metrics.
SampleEvaluationQuery
dataclass
¶
SampleEvaluationQuery(run_name: str, *criteria: MatchExpression)
Bases: MatchExpression
Query if a sample has evaluation metrics matching a criterion.
Example
SampleEvaluationQuery("run1", EvaluationMetricField("miou") < 0.3)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
run_name
|
str
|
The evaluation run name to match metrics against. |
required |
criteria
|
MatchExpression
|
The evaluation metric criteria to combine. |
()
|
AnnotationMetricQuery¶
Classes and functions for building queries against annotation evaluation metrics.
AnnotationMetricQuery
dataclass
¶
AnnotationMetricQuery(
match_kind: AnnotationMetricMatchKind,
run_name: str,
gt_label_name: str,
pred_label_name: str,
criteria: list[AnnotationEvaluationMetricMatchExpression],
)
Bases: MatchExpression
Query samples by annotation-level evaluation results.
This query matches samples that belong to an evaluation run and contain annotation pairs in a selected confusion-matrix cell, optionally constrained by persisted annotation metrics.
Example
AnnotationMetricQuery.confusion(
"run1",
"cat",
"dog",
AnnotationEvaluationMetricField("iou") > 0.3,
)
confusion
classmethod
¶
confusion(
run_name: str,
ground_truth: str,
prediction: str,
*criteria: AnnotationEvaluationMetricMatchExpression
) -> AnnotationMetricQuery
Match samples by confusion-matrix cell within an evaluation run.
Example
AnnotationMetricQuery.confusion(
"run1",
"cat",
"dog",
AnnotationEvaluationMetricField("iou") < 0.3,
)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
run_name
|
str
|
The evaluation run name to match metrics against. |
required |
ground_truth
|
str
|
Ground-truth annotation class name. |
required |
prediction
|
str
|
Predicted annotation class name. |
required |
criteria
|
AnnotationEvaluationMetricMatchExpression
|
Zero or more metric comparisons that must all match the same annotation pair. |
()
|
AnnotationEvaluationMetricField¶
Classes for filtering samples by persisted annotation evaluation metrics.
AnnotationEvaluationMetricField ¶
AnnotationEvaluationMetricField(metric_name: str)
Queryable annotation metric field for annotation-level evaluation results.
Use this field inside :meth:AnnotationMetricQuery.confusion to filter samples by
persisted annotation metrics such as IoU.
Example
AnnotationMetricQuery.confusion(
"run1",
"cat",
"dog",
AnnotationEvaluationMetricField("iou") > 0.3,
)
__eq__ ¶
__eq__(other: float | int) -> AnnotationEvaluationMetricMatchExpression
Create an equality filter.
__ge__ ¶
__ge__(other: float | int) -> AnnotationEvaluationMetricMatchExpression
Create a greater-than-or-equal filter.
__gt__ ¶
__gt__(other: float | int) -> AnnotationEvaluationMetricMatchExpression
Create a greater-than filter.
__le__ ¶
__le__(other: float | int) -> AnnotationEvaluationMetricMatchExpression
Create a less-than-or-equal filter.
__lt__ ¶
__lt__(other: float | int) -> AnnotationEvaluationMetricMatchExpression
Create a less-than filter.
__ne__ ¶
__ne__(other: float | int) -> AnnotationEvaluationMetricMatchExpression
Create an inequality filter.
EvaluationMetricField¶
Classes for filtering samples by persisted evaluation metrics.
EvaluationMetricField ¶
EvaluationMetricField(metric_name: str)
Queryable per-sample metric field from an evaluation run.
Example
SampleEvaluationQuery("run1", EvaluationMetricField("miou") < 0.3)
OrderByEvaluationMetricField¶
Classes for order by expressions in dataset queries.
OrderByEvaluationMetricField ¶
OrderByEvaluationMetricField(evaluation_run_name: str, metric_name: str)
Bases: OrderByExpression
Order by an evaluation metric value from EvaluationSampleMetricTable.
Two LEFT OUTER JOINs are added automatically: first to EvaluationRunTable (filtering by name) to resolve the run UUID, then to EvaluationSampleMetricTable (filtering by run ID, sample ID, and metric name) to get at most one row per sample. Samples without a metric value still appear in results (sorted last when ascending).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
evaluation_run_name
|
str
|
The name of the evaluation run to sort by. |
required |
metric_name
|
str
|
The metric name to sort by. |
required |
apply ¶
apply(query: SelectOfScalar[T]) -> SelectOfScalar[T]
Apply joins for this sort and append the ORDER BY.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
query
|
SelectOfScalar[T]
|
The SQLModel Select query to modify. |
required |
Returns:
| Type | Description |
|---|---|
SelectOfScalar[T]
|
The modified query after joining and ordering. |
apply_joins ¶
apply_joins(query: SelectT) -> SelectT
Left-outer-join evaluation run and sample-metric tables.
apply_with_order_value ¶
apply_with_order_value(query: SelectOfScalar[T]) -> Select[tuple[T, Any]]
Apply this sort and append its value to the SELECT list.
Behaves like apply but also appends the sort value to the SELECT list as
ORDER_VALUE_LABEL. Read values back with get_order_value. Returns a
multi-column Select (read rows with session.execute) because the sort
value is added to the row.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
query
|
SelectOfScalar[T]
|
The SQLModel Select query to modify. |
required |
Returns:
| Type | Description |
|---|---|
Select[tuple[T, Any]]
|
The query after joining, appending the labeled sort value, and ordering. |
asc ¶
asc() -> Self
Set the ordering to ascending.
Returns:
| Type | Description |
|---|---|
Self
|
Self for method chaining. |
desc ¶
desc() -> Self
Set the ordering to descending.
Returns:
| Type | Description |
|---|---|
Self
|
Self for method chaining. |
to_column_element ¶
to_column_element() -> ColumnElement[Any]
Return the SQLAlchemy column element with direction applied.
For use in query.order_by() or window over(order_by=...). Does not
apply joins; call apply first when joins are required.