{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Tutorial 2: Train MoCo on CIFAR-10\n\nIn this tutorial, we will train a model based on the MoCo Paper\n[Momentum Contrast for Unsupervised Visual Representation Learning](https://arxiv.org/abs/1911.05722).\n\nWhen training self-supervised models using contrastive loss we\nusually face one big problem. To get good results, we need\nmany negative examples for the contrastive loss to work. Therefore,\nwe need a large batch size. However, not everyone has access to a cluster\nfull of GPUs or TPUs. To solve this problem, alternative approaches have been developed.\nSome of them use a memory bank to store old negative examples we can query\nto compensate for the smaller batch size. MoCo takes this approach\none step further by including a momentum encoder.\n\nWe use the **CIFAR-10** dataset for this tutorial.\n\nIn this tutorial you will learn:\n\n- How to use lightly to load a dataset and train a model\n\n- How to create a MoCo model with a memory bank\n\n- How to use the pre-trained model after self-supervised learning for a\n  transfer learning task\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Imports\n\nImport the Python frameworks we need for this tutorial.\nMake sure you have lightly installed.\n\n```console\npip install lightly\n```\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import copy\n\nimport pytorch_lightning as pl\nimport torch\nimport torch.nn as nn\nimport torchvision\n\nfrom lightly.data import LightlyDataset\nfrom lightly.loss import NTXentLoss\nfrom lightly.models import ResNetGenerator\nfrom lightly.models.modules.heads import MoCoProjectionHead\nfrom lightly.models.utils import (\n    batch_shuffle,\n    batch_unshuffle,\n    deactivate_requires_grad,\n    update_momentum,\n)\nfrom lightly.transforms import MoCoV2Transform, utils"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Configuration\n\nWe set some configuration parameters for our experiment.\nFeel free to change them and analyze the effect.\n\nThe default configuration uses a batch size of 512. This requires around 6.4GB\nof GPU memory.\nWhen training for 100 epochs you should achieve around 73% test set accuracy.\nWhen training for 200 epochs accuracy increases to about 80%.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "num_workers = 8\nbatch_size = 512\nmemory_bank_size = 4096\nseed = 1\nmax_epochs = 100"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Replace the path with the location of your CIFAR-10 dataset.\nWe assume we have a train folder with subfolders\nfor each class and .png images inside.\n\nYou can download [CIFAR-10 in folders from Kaggle](https://www.kaggle.com/swaroopkml/cifar10-pngs-in-folders).\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# The dataset structure should be like this:\n# cifar10/train/\n#  L airplane/\n#    L 10008_airplane.png\n#    L ...\n#  L automobile/\n#  L bird/\n#  L cat/\n#  L deer/\n#  L dog/\n#  L frog/\n#  L horse/\n#  L ship/\n#  L truck/\npath_to_train = \"/datasets/cifar10/train/\"\npath_to_test = \"/datasets/cifar10/test/\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's set the seed to ensure reproducibility of the experiments\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "pl.seed_everything(seed)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setup data augmentations and loaders\n\nWe start with our data preprocessing pipeline. We can implement augmentations\nfrom the MoCo paper using the transforms provided by lightly.\nImages from the CIFAR-10 dataset have a resolution of 32x32 pixels. Let's use\nthis resolution to train our model.\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>We could use a higher input resolution to train our model. However,\n  since the original resolution of CIFAR-10 images is low there is no real value\n  in increasing the resolution. A higher resolution results in higher memory\n  consumption and to compensate for that we would need to reduce the batch size.</p></div>\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# disable blur because we're working with tiny images\ntransform = MoCoV2Transform(\n    input_size=32,\n    gaussian_blur=0.0,\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We don't want any augmentation for our test data. Therefore,\nwe create custom, torchvision based data transformations.\nLet's ensure the size is correct and we normalize the data in\nthe same way as we do with the training data.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Augmentations typically used to train on cifar-10\ntrain_classifier_transforms = torchvision.transforms.Compose(\n    [\n        torchvision.transforms.RandomCrop(32, padding=4),\n        torchvision.transforms.RandomHorizontalFlip(),\n        torchvision.transforms.ToTensor(),\n        torchvision.transforms.Normalize(\n            mean=utils.IMAGENET_NORMALIZE[\"mean\"],\n            std=utils.IMAGENET_NORMALIZE[\"std\"],\n        ),\n    ]\n)\n\n# No additional augmentations for the test set\ntest_transforms = torchvision.transforms.Compose(\n    [\n        torchvision.transforms.Resize((32, 32)),\n        torchvision.transforms.ToTensor(),\n        torchvision.transforms.Normalize(\n            mean=utils.IMAGENET_NORMALIZE[\"mean\"],\n            std=utils.IMAGENET_NORMALIZE[\"std\"],\n        ),\n    ]\n)\n\n# We use the moco augmentations for training moco\ndataset_train_moco = LightlyDataset(input_dir=path_to_train, transform=transform)\n\n# Since we also train a linear classifier on the pre-trained moco model we\n# reuse the test augmentations here (MoCo augmentations are very strong and\n# usually reduce accuracy of models which are not used for contrastive learning.\n# Our linear layer will be trained using cross entropy loss and labels provided\n# by the dataset. Therefore we chose light augmentations.)\ndataset_train_classifier = LightlyDataset(\n    input_dir=path_to_train, transform=train_classifier_transforms\n)\n\ndataset_test = LightlyDataset(input_dir=path_to_test, transform=test_transforms)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create the dataloaders to load and preprocess the data\nin the background.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "dataloader_train_moco = torch.utils.data.DataLoader(\n    dataset_train_moco,\n    batch_size=batch_size,\n    shuffle=True,\n    drop_last=True,\n    num_workers=num_workers,\n)\n\ndataloader_train_classifier = torch.utils.data.DataLoader(\n    dataset_train_classifier,\n    batch_size=batch_size,\n    shuffle=True,\n    drop_last=True,\n    num_workers=num_workers,\n)\n\ndataloader_test = torch.utils.data.DataLoader(\n    dataset_test,\n    batch_size=batch_size,\n    shuffle=False,\n    drop_last=False,\n    num_workers=num_workers,\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Create the MoCo Lightning Module\nNow we create our MoCo model. We use PyTorch Lightning to train\nour model. We follow the specification of the lightning module.\nIn this example we set the number of features for the hidden dimension to 512.\nThe momentum for the Momentum Encoder is set to 0.99 (default is 0.999) since\nother reports show that this works better for Cifar-10.\n\nFor the backbone we use the lightly variant of a resnet-18. You can use another model following\nour [playground to use custom backbones](https://colab.research.google.com/drive/1ubepXnpANiWOSmq80e-mqAxjLx53m-zu?usp=sharing).\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>We use a split batch norm to simulate multi-gpu behaviour. Combined\n  with the use of batch shuffling, this prevents the model from communicating\n  through the batch norm layers.</p></div>\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "class MocoModel(pl.LightningModule):\n    def __init__(self):\n        super().__init__()\n\n        # create a ResNet backbone and remove the classification head\n        resnet = ResNetGenerator(\"resnet-18\", 1, num_splits=8)\n        self.backbone = nn.Sequential(\n            *list(resnet.children())[:-1],\n            nn.AdaptiveAvgPool2d(1),\n        )\n\n        # create a moco model based on ResNet\n        self.projection_head = MoCoProjectionHead(512, 512, 128)\n        self.backbone_momentum = copy.deepcopy(self.backbone)\n        self.projection_head_momentum = copy.deepcopy(self.projection_head)\n        deactivate_requires_grad(self.backbone_momentum)\n        deactivate_requires_grad(self.projection_head_momentum)\n\n        # create our loss with the optional memory bank\n        self.criterion = NTXentLoss(\n            temperature=0.1, memory_bank_size=(memory_bank_size, 128)\n        )\n\n    def training_step(self, batch, batch_idx):\n        (x_q, x_k), _, _ = batch\n\n        # update momentum\n        update_momentum(self.backbone, self.backbone_momentum, 0.99)\n        update_momentum(self.projection_head, self.projection_head_momentum, 0.99)\n\n        # get queries\n        q = self.backbone(x_q).flatten(start_dim=1)\n        q = self.projection_head(q)\n\n        # get keys\n        k, shuffle = batch_shuffle(x_k)\n        k = self.backbone_momentum(k).flatten(start_dim=1)\n        k = self.projection_head_momentum(k)\n        k = batch_unshuffle(k, shuffle)\n\n        loss = self.criterion(q, k)\n        self.log(\"train_loss_ssl\", loss)\n        return loss\n\n    def on_train_epoch_end(self):\n        self.custom_histogram_weights()\n\n    # We provide a helper method to log weights in tensorboard\n    # which is useful for debugging.\n    def custom_histogram_weights(self):\n        for name, params in self.named_parameters():\n            self.logger.experiment.add_histogram(name, params, self.current_epoch)\n\n    def configure_optimizers(self):\n        optim = torch.optim.SGD(\n            self.parameters(),\n            lr=6e-2,\n            momentum=0.9,\n            weight_decay=5e-4,\n        )\n        scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim, max_epochs)\n        return [optim], [scheduler]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Create the Classifier Lightning Module\nWe create a linear classifier using the features we extract using MoCo\nand train it on the dataset\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "class Classifier(pl.LightningModule):\n    def __init__(self, backbone):\n        super().__init__()\n        # use the pretrained ResNet backbone\n        self.backbone = backbone\n\n        # freeze the backbone\n        deactivate_requires_grad(backbone)\n\n        # create a linear layer for our downstream classification model\n        self.fc = nn.Linear(512, 10)\n\n        self.criterion = nn.CrossEntropyLoss()\n        self.validation_step_outputs = []\n\n    def forward(self, x):\n        y_hat = self.backbone(x).flatten(start_dim=1)\n        y_hat = self.fc(y_hat)\n        return y_hat\n\n    def training_step(self, batch, batch_idx):\n        x, y, _ = batch\n        y_hat = self.forward(x)\n        loss = self.criterion(y_hat, y)\n        self.log(\"train_loss_fc\", loss)\n        return loss\n\n    def on_train_epoch_end(self):\n        self.custom_histogram_weights()\n\n    # We provide a helper method to log weights in tensorboard\n    # which is useful for debugging.\n    def custom_histogram_weights(self):\n        for name, params in self.named_parameters():\n            self.logger.experiment.add_histogram(name, params, self.current_epoch)\n\n    def validation_step(self, batch, batch_idx):\n        x, y, _ = batch\n        y_hat = self.forward(x)\n        y_hat = torch.nn.functional.softmax(y_hat, dim=1)\n\n        # calculate number of correct predictions\n        _, predicted = torch.max(y_hat, 1)\n        num = predicted.shape[0]\n        correct = (predicted == y).float().sum()\n        self.validation_step_outputs.append((num, correct))\n        return num, correct\n\n    def on_validation_epoch_end(self):\n        # calculate and log top1 accuracy\n        if self.validation_step_outputs:\n            total_num = 0\n            total_correct = 0\n            for num, correct in self.validation_step_outputs:\n                total_num += num\n                total_correct += correct\n            acc = total_correct / total_num\n            self.log(\"val_acc\", acc, on_epoch=True, prog_bar=True)\n            self.validation_step_outputs.clear()\n\n    def configure_optimizers(self):\n        optim = torch.optim.SGD(self.fc.parameters(), lr=30.0)\n        scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim, max_epochs)\n        return [optim], [scheduler]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Train the MoCo model\n\nWe can instantiate the model and train it using the\nlightning trainer.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "model = MocoModel()\ntrainer = pl.Trainer(max_epochs=max_epochs, devices=1, accelerator=\"gpu\")\ntrainer.fit(model, dataloader_train_moco)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Train the Classifier\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "model.eval()\nclassifier = Classifier(model.backbone)\ntrainer = pl.Trainer(max_epochs=max_epochs, devices=1, accelerator=\"gpu\")\ntrainer.fit(classifier, dataloader_train_classifier, dataloader_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Checkout the tensorboard logs while the model is training.\n\nRun `tensorboard --logdir lightning_logs/` to start tensorboard\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>If you run the code on a remote machine you can't just\n  access the tensorboard logs. You need to forward the port.\n  You can do this by using an editor such as Visual Studio Code\n  which has a port forwarding functionality (make sure\n  the remote extensions are installed and are connected with your machine).\n\n  Or you can use a shell command similar to this one to forward port\n  6006 from your remote machine to your local machine:\n\n  `ssh username:host -N -L localhost:6006:localhost:6006`</p></div>\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Next Steps\n\nInterested in exploring other self-supervised models? Check out our other\ntutorials:\n\n- `lightly-simclr-tutorial-3`\n- `lightly-simsiam-tutorial-4`\n- `lightly-custom-augmentation-5`\n- `lightly-detectron-tutorial-6`\n\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.13"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}