Models

Self-supervised methods with runnable examples in PyTorch, PyTorch Lightning and distributed PyTorch Lightning. Each one is built from the same losses, heads and transforms, so you can start from a method and swap the parts.

Tip

Not sure where to start? With a ResNet, try SimCLR or BYOL. With a ViT, try DINOv2 or MAE.

SimCLR

Contrastive · 2020

Pulls two augmented views of an image together and pushes other images away.

ResNetTutorial

SimCLR
MoCo

Contrastive · 2020

Contrastive learning with a momentum encoder and a memory bank of negatives.

ResNetTutorial

MoCo
NNCLR

Contrastive · 2021

Uses a view’s nearest neighbour from a support set as its positive.

ResNet

NNCLR
DCL & DCLW

Contrastive · 2021

SimCLR with a decoupled loss that drops the positive pair from the denominator.

ResNet

DCL & DCLW
TiCo

Contrastive · 2022

Agreement between distorted views plus a covariance contrast term.

ResNet

TiCo
BYOL

Self-distillation · 2020

An online network predicts the output of a momentum target network, with no negatives.

ResNet

BYOL
SimSiam

Self-distillation · 2020

A Siamese network with a stop-gradient: no negatives, large batches or momentum encoder.

ResNetTutorial

SimSiam
DINO

Self-distillation · 2021

A student network matches the outputs of a momentum-averaged teacher.

ResNet

DINO
FastSiam

Self-distillation · 2022

SimSiam that averages several target predictions to train with small batches.

ResNet

FastSiam
MSN

Self-distillation · 2022

Compares the cluster assignments of masked and unmasked views.

ViT

MSN
PMSN

Self-distillation · 2022

MSN with a custom clustering prior, a power law by default, for imbalanced data.

ViT

PMSN
DINOv2

Self-distillation · 2023

DINO with self-distillation at both the image and the patch level.

ViT

DINOv2
MAE

Masked image modeling · 2021

Masks most patches and reconstructs their pixels with a light decoder.

ViT

MAE
SimMIM

Masked image modeling · 2021

Like MAE, but the encoder sees both masked and visible patches.

ViT

SimMIM
iBOT

Masked image modeling · 2021

Masked image modeling with an online tokenizer and DINO-style self-distillation.

ViT

iBOT
AIM

Masked image modeling · 2024

Predicts the pixels of image patches autoregressively.

ViT

AIM
CAPI

Masked image modeling · 2025

Predicts the cluster assignments of masked patches instead of their pixels.

ViT

CAPI
Pixio

Masked image modeling · 2025

MAE with a much deeper decoder and block-wise masking.

ViT

Pixio
Barlow Twins

Redundancy reduction · 2021

Drives the cross-correlation of two views’ embeddings toward the identity.

ResNet

Barlow Twins
VICReg

Redundancy reduction · 2022

Variance, invariance and covariance terms prevent collapse without negatives.

ResNet

VICReg
VICRegL

Redundancy reduction · 2022

VICReg on global and local features, for detection and segmentation.

ResNet

VICRegL
MMCR

Redundancy reduction · 2023

Maximizes manifold capacity: the spread of each image’s mean view embedding.

ResNet

MMCR
FroSSL

Redundancy reduction · 2024

Avoids collapse by maximizing embedding entropy through Frobenius norms.

ResNet

FroSSL
LeJEPA

Redundancy reduction · 2025

Invariance across views, with embeddings regularized by SIGReg.

ViT

LeJEPA
SwaV

Clustering and dense · 2020

Clusters the data and swaps cluster assignments between views.

ResNet

SwaV
DenseCL

Clustering and dense · 2021

MoCo with a dense, pixel-level contrastive loss for detection and segmentation.

ResNet

DenseCL
DetConS

Clustering and dense · 2021

SimCLR on per-region embeddings from unsupervised segmentation masks.

ResNet

DetConS
SMoG

Clustering and dense · 2022

Contrastive learning between groups of images instead of single images.

ResNet

SMoG
LeWM

World models · 2026

A latent world model: predicts the next frame’s embedding from past frames and actions.

ViTTutorial

LeWM