DINOv31

DINOv31 post-trains (“continues”) a DINOv2-pretrained backbone with the full DINOv2 objective (DINO + iBOT + KoLeo) plus an auxiliary Patch Kernel Alignment (PaKA / CKA) loss that aligns the relational structure of student and teacher dense patch tokens, aimed at improving dense (patch-level) features for downstream segmentation and detection. See the PaKA paper for details.

See also

DINOv31 is a DINOv2 post-training method: start from a DINOv2 checkpoint (see DINOv2) and continue training with PaKA.

Use DINOv31 in LightlyTrain

Python

import lightly_train

if __name__ == "__main__":
    lightly_train.pretrain(
        out="out/my_experiment",
        data="my_data_dir",
        model="dinov2/vits14",
        method="dinov31",
        checkpoint="dinov2_vits14.ckpt",  # full DINOv2 checkpoint (incl. optimizer); only model weights are loaded
    )

Command Line

lightly-train pretrain \
    out=out/my_experiment \
    data=my_data_dir \
    model="dinov2/vits14" \
    method="dinov31" \
    checkpoint="dinov2_vits14.ckpt"

Default Method Arguments

The following are the default method arguments for DINOv31. To learn how you can override these settings, see Method Arguments.

Default Method Arguments

{
    "batch_norm": false,
    "center_method": "softmax",
    "center_momentum": 0.9,
    "dino_bottleneck_dim": 256,
    "dino_loss_weight": 1.0,
    "gradient_clip_val": 3.0,
    "hidden_dim": 2048,
    "ibot_bottleneck_dim": 256,
    "ibot_loss_weight": 1.0,
    "ibot_separate_head": false,
    "koleo_loss_weight": 0.1,
    "layerwise_decay": 0.9,
    "lr_scale_method": "sqrt",
    "mask_probability": 0.5,
    "mask_ratio_max": 0.5,
    "mask_ratio_min": 0.1,
    "min_lr": 1e-06,
    "momentum_end": 1.0,
    "momentum_start": 0.992,
    "output_dim": 65536,
    "paka_max_tokens": 512,
    "paka_num_local": 8,
    "paka_start_step": 0,
    "paka_weight": 1.0,
    "patch_embed_lr_multiplier": 0.2,
    "reference_batch_size": 1024,
    "student_freeze_backbone_steps": 0,
    "student_freeze_last_layer_steps": 1250,
    "student_temp": 0.1,
    "teacher_temp_end": 0.07,
    "teacher_temp_start": 0.04,
    "teacher_temp_warmup_steps": 37500,
    "warmup_steps": 12500,
    "weight_decay_end": 0.4,
    "weight_decay_start": "auto"
}

Default Image Transform Arguments

The following are the default transform arguments for DINOv31. To learn how you can override these settings, see Transforms.

Default Image Transforms

{
    "channel_drop": null,
    "color_jitter": {
        "brightness": 0.8,
        "contrast": 0.8,
        "hue": 0.2,
        "prob": 0.8,
        "saturation": 0.4,
        "strength": 0.5
    },
    "gaussian_blur": {
        "blur_limit": 0,
        "prob": 1.0,
        "sigmas": [
            0.1,
            2.0
        ]
    },
    "global_view_1": {
        "gaussian_blur": {
            "blur_limit": 0,
            "prob": 0.1,
            "sigmas": [
                0.1,
                2.0
            ]
        },
        "solarize": {
            "prob": 0.2,
            "threshold": 0.5
        }
    },
    "image_size": [
        224,
        224
    ],
    "local_view": {
        "gaussian_blur": {
            "blur_limit": 0,
            "prob": 0.5,
            "sigmas": [
                0.1,
                2.0
            ]
        },
        "num_views": 8,
        "random_resize": {
            "max_scale": 0.32,
            "min_scale": 0.05
        },
        "view_size": [
            98,
            98
        ]
    },
    "normalize": {
        "mean": [
            0.485,
            0.456,
            0.406
        ],
        "std": [
            0.229,
            0.224,
            0.225
        ]
    },
    "num_channels": 3,
    "paka_high_overlap_iou": 0.9,
    "paka_num_local": 8,
    "random_flip": {
        "horizontal_prob": 0.5,
        "vertical_prob": 0.0
    },
    "random_gray_scale": 0.2,
    "random_resize": {
        "max_scale": 1.0,
        "min_scale": 0.32
    },
    "random_rotation": null,
    "record_geometry": true,
    "solarize": null
}