Tutorial 3: Train SimCLR on Clothing

In this tutorial, we will train a SimCLR model using lightly. The model, augmentations and training procedure is from A Simple Framework for Contrastive Learning of Visual Representations.

The paper explores a rather simple training procedure for contrastive learning. Since we use the typical contrastive learning loss based on NCE the method greatly benefits from having larger batch sizes. In this example, we use a batch size of 256 and paired with the input resolution per image of 64x64 pixels and a resnet-18 model this example requires 16GB of GPU memory.

We use the clothing dataset from Alex Grigorev for this tutorial.

In this tutorial you will learn:

  • How to create a SimCLR model

  • How to generate image representations

  • How different augmentations impact the learned representations

Imports

Import the Python frameworks we need for this tutorial.

import os

import matplotlib.pyplot as plt
import numpy as np
import pytorch_lightning as pl
import torch
import torch.nn as nn
import torchvision
from PIL import Image
from sklearn.neighbors import NearestNeighbors
from sklearn.preprocessing import normalize

from lightly.data import LightlyDataset
from lightly.transforms import SimCLRTransform, utils

Configuration

We set some configuration parameters for our experiment. Feel free to change them and analyze the effect.

The default configuration with a batch size of 256 and input resolution of 128 requires 6GB of GPU memory.

num_workers = 8
batch_size = 256
seed = 1
max_epochs = 20
input_size = 128
num_ftrs = 32

Let’s set the seed for our experiments

pl.seed_everything(seed)
1

Make sure path_to_data points to the downloaded clothing dataset. You can download it using git clone https://github.com/alexeygrigorev/clothing-dataset.git

path_to_data = "/datasets/clothing-dataset/images"

Setup data augmentations and loaders

The images from the dataset have been taken from above when the clothing was on a table, bed or floor. Therefore, we can make use of additional augmentations such as vertical flip or random rotation (90 degrees). By adding these augmentations we learn our model invariance regarding the orientation of the clothing piece. E.g. we don’t care if a shirt is upside down but more about the structure which make it a shirt.

You can learn more about the different augmentations and learned invariances here: Advanced Concepts in Self-Supervised Learning.

transform = SimCLRTransform(input_size=input_size, vf_prob=0.5, rr_prob=0.5)

# We create a torchvision transformation for embedding the dataset after
# training
test_transform = torchvision.transforms.Compose(
    [
        torchvision.transforms.Resize((input_size, input_size)),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize(
            mean=utils.IMAGENET_NORMALIZE["mean"],
            std=utils.IMAGENET_NORMALIZE["std"],
        ),
    ]
)

dataset_train_simclr = LightlyDataset(input_dir=path_to_data, transform=transform)

dataset_test = LightlyDataset(input_dir=path_to_data, transform=test_transform)

dataloader_train_simclr = torch.utils.data.DataLoader(
    dataset_train_simclr,
    batch_size=batch_size,
    shuffle=True,
    drop_last=True,
    num_workers=num_workers,
)

dataloader_test = torch.utils.data.DataLoader(
    dataset_test,
    batch_size=batch_size,
    shuffle=False,
    drop_last=False,
    num_workers=num_workers,
)

Create the SimCLR Model

Now we create the SimCLR model. We implement it as a PyTorch Lightning Module and use a ResNet-18 backbone from Torchvision. Lightly provides implementations of the SimCLR projection head and loss function in the SimCLRProjectionHead and NTXentLoss classes. We can simply import them and combine the building blocks in the module.

from lightly.loss import NTXentLoss
from lightly.models.modules.heads import SimCLRProjectionHead


class SimCLRModel(pl.LightningModule):
    def __init__(self):
        super().__init__()

        # create a ResNet backbone and remove the classification head
        resnet = torchvision.models.resnet18()
        self.backbone = nn.Sequential(*list(resnet.children())[:-1])

        hidden_dim = resnet.fc.in_features
        self.projection_head = SimCLRProjectionHead(hidden_dim, hidden_dim, 128)

        self.criterion = NTXentLoss()

    def forward(self, x):
        h = self.backbone(x).flatten(start_dim=1)
        z = self.projection_head(h)
        return z

    def training_step(self, batch, batch_idx):
        (x0, x1), _, _ = batch
        z0 = self.forward(x0)
        z1 = self.forward(x1)
        loss = self.criterion(z0, z1)
        self.log("train_loss_ssl", loss)
        return loss

    def configure_optimizers(self):
        optim = torch.optim.SGD(
            self.parameters(), lr=6e-2, momentum=0.9, weight_decay=5e-4
        )
        scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim, max_epochs)
        return [optim], [scheduler]

Train the module using the PyTorch Lightning Trainer on a single GPU.

model = SimCLRModel()
trainer = pl.Trainer(max_epochs=max_epochs, devices=1, accelerator="gpu")
trainer.fit(model, dataloader_train_simclr)
/datasets/actions-runner/core_gpu_runner_01/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/lightning_fabric/plugins/environments/slurm.py:165: PossibleUserWarning: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python /datasets/actions-runner/core_gpu_runner_01/_work/li ...
  rank_zero_warn(
/datasets/actions-runner/core_gpu_runner_01/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1609: PossibleUserWarning: The number of training batches (22) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
  rank_zero_warn(

Training: 0it [00:00, ?it/s]
Training:   0%|          | 0/22 [00:00<?, ?it/s]
Epoch 0:   0%|          | 0/22 [00:00<?, ?it/s]
Epoch 0:   5%|▍         | 1/22 [00:04<01:44,  4.99s/it]
Epoch 0:   5%|▍         | 1/22 [00:05<01:45,  5.04s/it, loss=6.21, v_num=0]
Epoch 0:   9%|▉         | 2/22 [00:05<00:51,  2.57s/it, loss=6.21, v_num=0]
Epoch 0:   9%|▉         | 2/22 [00:05<00:51,  2.60s/it, loss=6.2, v_num=0]
Epoch 0:  14%|█▎        | 3/22 [00:05<00:33,  1.76s/it, loss=6.2, v_num=0]
Epoch 0:  14%|█▎        | 3/22 [00:05<00:33,  1.78s/it, loss=6.2, v_num=0]
Epoch 0:  18%|█▊        | 4/22 [00:05<00:24,  1.36s/it, loss=6.2, v_num=0]
Epoch 0:  18%|█▊        | 4/22 [00:05<00:24,  1.37s/it, loss=6.2, v_num=0]
Epoch 0:  23%|██▎       | 5/22 [00:05<00:19,  1.12s/it, loss=6.2, v_num=0]
Epoch 0:  23%|██▎       | 5/22 [00:05<00:19,  1.13s/it, loss=6.18, v_num=0]
Epoch 0:  27%|██▋       | 6/22 [00:05<00:15,  1.05it/s, loss=6.18, v_num=0]
Epoch 0:  27%|██▋       | 6/22 [00:05<00:15,  1.03it/s, loss=6.18, v_num=0]
Epoch 0:  32%|███▏      | 7/22 [00:05<00:12,  1.19it/s, loss=6.18, v_num=0]
Epoch 0:  32%|███▏      | 7/22 [00:05<00:12,  1.18it/s, loss=6.17, v_num=0]
Epoch 0:  36%|███▋      | 8/22 [00:06<00:10,  1.32it/s, loss=6.17, v_num=0]
Epoch 0:  36%|███▋      | 8/22 [00:06<00:10,  1.31it/s, loss=6.16, v_num=0]
Epoch 0:  41%|████      | 9/22 [00:06<00:08,  1.45it/s, loss=6.16, v_num=0]
Epoch 0:  41%|████      | 9/22 [00:06<00:09,  1.44it/s, loss=6.16, v_num=0]
Epoch 0:  45%|████▌     | 10/22 [00:06<00:07,  1.58it/s, loss=6.16, v_num=0]
Epoch 0:  45%|████▌     | 10/22 [00:06<00:07,  1.56it/s, loss=6.15, v_num=0]
Epoch 0:  50%|█████     | 11/22 [00:06<00:06,  1.69it/s, loss=6.15, v_num=0]
Epoch 0:  50%|█████     | 11/22 [00:06<00:06,  1.68it/s, loss=6.15, v_num=0]
Epoch 0:  55%|█████▍    | 12/22 [00:06<00:05,  1.81it/s, loss=6.15, v_num=0]
Epoch 0:  55%|█████▍    | 12/22 [00:06<00:05,  1.79it/s, loss=6.14, v_num=0]
Epoch 0:  59%|█████▉    | 13/22 [00:06<00:04,  1.91it/s, loss=6.14, v_num=0]
Epoch 0:  59%|█████▉    | 13/22 [00:06<00:04,  1.90it/s, loss=6.14, v_num=0]
Epoch 0:  64%|██████▎   | 14/22 [00:06<00:03,  2.02it/s, loss=6.14, v_num=0]
Epoch 0:  64%|██████▎   | 14/22 [00:07<00:04,  2.00it/s, loss=6.13, v_num=0]
Epoch 0:  68%|██████▊   | 15/22 [00:07<00:03,  2.12it/s, loss=6.13, v_num=0]
Epoch 0:  68%|██████▊   | 15/22 [00:07<00:03,  2.10it/s, loss=6.13, v_num=0]
Epoch 0:  73%|███████▎  | 16/22 [00:07<00:02,  2.21it/s, loss=6.13, v_num=0]
Epoch 0:  73%|███████▎  | 16/22 [00:07<00:02,  2.19it/s, loss=6.12, v_num=0]
Epoch 0:  77%|███████▋  | 17/22 [00:07<00:02,  2.30it/s, loss=6.12, v_num=0]
Epoch 0:  77%|███████▋  | 17/22 [00:07<00:02,  2.28it/s, loss=6.11, v_num=0]
Epoch 0:  82%|████████▏ | 18/22 [00:07<00:01,  2.39it/s, loss=6.11, v_num=0]
Epoch 0:  82%|████████▏ | 18/22 [00:07<00:01,  2.37it/s, loss=6.1, v_num=0]
Epoch 0:  86%|████████▋ | 19/22 [00:07<00:01,  2.47it/s, loss=6.1, v_num=0]
Epoch 0:  86%|████████▋ | 19/22 [00:07<00:01,  2.45it/s, loss=6.09, v_num=0]
Epoch 0:  91%|█████████ | 20/22 [00:07<00:00,  2.55it/s, loss=6.09, v_num=0]
Epoch 0:  91%|█████████ | 20/22 [00:07<00:00,  2.53it/s, loss=6.09, v_num=0]
Epoch 0:  95%|█████████▌| 21/22 [00:07<00:00,  2.63it/s, loss=6.09, v_num=0]
Epoch 0:  95%|█████████▌| 21/22 [00:08<00:00,  2.61it/s, loss=6.08, v_num=0]
Epoch 0: 100%|██████████| 22/22 [00:08<00:00,  2.70it/s, loss=6.08, v_num=0]
Epoch 0: 100%|██████████| 22/22 [00:08<00:00,  2.68it/s, loss=6.07, v_num=0]
Epoch 0: 100%|██████████| 22/22 [00:08<00:00,  2.68it/s, loss=6.07, v_num=0]
Epoch 0:   0%|          | 0/22 [00:00<?, ?it/s, loss=6.07, v_num=0]
Epoch 1:   0%|          | 0/22 [00:00<?, ?it/s, loss=6.07, v_num=0]
Epoch 1:   5%|▍         | 1/22 [00:02<00:43,  2.06s/it, loss=6.07, v_num=0]
Epoch 1:   5%|▍         | 1/22 [00:02<00:44,  2.12s/it, loss=6.06, v_num=0]
Epoch 1:   9%|▉         | 2/22 [00:02<00:22,  1.10s/it, loss=6.06, v_num=0]
Epoch 1:   9%|▉         | 2/22 [00:02<00:22,  1.13s/it, loss=6.05, v_num=0]
Epoch 1:  14%|█▎        | 3/22 [00:02<00:14,  1.27it/s, loss=6.05, v_num=0]
Epoch 1:  14%|█▎        | 3/22 [00:02<00:15,  1.24it/s, loss=6.04, v_num=0]
Epoch 1:  18%|█▊        | 4/22 [00:02<00:11,  1.59it/s, loss=6.04, v_num=0]
Epoch 1:  18%|█▊        | 4/22 [00:02<00:11,  1.55it/s, loss=6.03, v_num=0]
Epoch 1:  23%|██▎       | 5/22 [00:02<00:09,  1.88it/s, loss=6.03, v_num=0]
Epoch 1:  23%|██▎       | 5/22 [00:02<00:09,  1.84it/s, loss=6.02, v_num=0]
Epoch 1:  27%|██▋       | 6/22 [00:02<00:07,  2.13it/s, loss=6.02, v_num=0]
Epoch 1:  27%|██▋       | 6/22 [00:02<00:07,  2.09it/s, loss=6.01, v_num=0]
Epoch 1:  32%|███▏      | 7/22 [00:02<00:06,  2.36it/s, loss=6.01, v_num=0]
Epoch 1:  32%|███▏      | 7/22 [00:03<00:06,  2.31it/s, loss=6, v_num=0]
Epoch 1:  36%|███▋      | 8/22 [00:03<00:05,  2.57it/s, loss=6, v_num=0]
Epoch 1:  36%|███▋      | 8/22 [00:03<00:05,  2.52it/s, loss=6, v_num=0]
Epoch 1:  41%|████      | 9/22 [00:03<00:04,  2.72it/s, loss=6, v_num=0]
Epoch 1:  41%|████      | 9/22 [00:03<00:04,  2.67it/s, loss=5.98, v_num=0]
Epoch 1:  45%|████▌     | 10/22 [00:03<00:04,  2.88it/s, loss=5.98, v_num=0]
Epoch 1:  45%|████▌     | 10/22 [00:03<00:04,  2.83it/s, loss=5.97, v_num=0]
Epoch 1:  50%|█████     | 11/22 [00:03<00:03,  3.04it/s, loss=5.97, v_num=0]
Epoch 1:  50%|█████     | 11/22 [00:03<00:03,  2.99it/s, loss=5.96, v_num=0]
Epoch 1:  55%|█████▍    | 12/22 [00:03<00:03,  3.19it/s, loss=5.96, v_num=0]
Epoch 1:  55%|█████▍    | 12/22 [00:03<00:03,  3.13it/s, loss=5.95, v_num=0]
Epoch 1:  59%|█████▉    | 13/22 [00:03<00:02,  3.31it/s, loss=5.95, v_num=0]
Epoch 1:  59%|█████▉    | 13/22 [00:03<00:02,  3.27it/s, loss=5.95, v_num=0]
Epoch 1:  64%|██████▎   | 14/22 [00:04<00:02,  3.44it/s, loss=5.95, v_num=0]
Epoch 1:  64%|██████▎   | 14/22 [00:04<00:02,  3.39it/s, loss=5.94, v_num=0]
Epoch 1:  68%|██████▊   | 15/22 [00:04<00:01,  3.56it/s, loss=5.94, v_num=0]
Epoch 1:  68%|██████▊   | 15/22 [00:04<00:01,  3.51it/s, loss=5.94, v_num=0]
Epoch 1:  73%|███████▎  | 16/22 [00:04<00:01,  3.67it/s, loss=5.94, v_num=0]
Epoch 1:  73%|███████▎  | 16/22 [00:04<00:01,  3.61it/s, loss=5.93, v_num=0]
Epoch 1:  77%|███████▋  | 17/22 [00:04<00:01,  3.72it/s, loss=5.93, v_num=0]
Epoch 1:  77%|███████▋  | 17/22 [00:04<00:01,  3.67it/s, loss=5.93, v_num=0]
Epoch 1:  82%|████████▏ | 18/22 [00:04<00:01,  3.81it/s, loss=5.93, v_num=0]
Epoch 1:  82%|████████▏ | 18/22 [00:04<00:01,  3.76it/s, loss=5.92, v_num=0]
Epoch 1:  86%|████████▋ | 19/22 [00:04<00:00,  3.90it/s, loss=5.92, v_num=0]
Epoch 1:  86%|████████▋ | 19/22 [00:04<00:00,  3.85it/s, loss=5.91, v_num=0]
Epoch 1:  91%|█████████ | 20/22 [00:05<00:00,  3.97it/s, loss=5.91, v_num=0]
Epoch 1:  91%|█████████ | 20/22 [00:05<00:00,  3.93it/s, loss=5.91, v_num=0]
Epoch 1:  95%|█████████▌| 21/22 [00:05<00:00,  4.05it/s, loss=5.91, v_num=0]
Epoch 1:  95%|█████████▌| 21/22 [00:05<00:00,  4.01it/s, loss=5.9, v_num=0]
Epoch 1: 100%|██████████| 22/22 [00:05<00:00,  4.12it/s, loss=5.9, v_num=0]
Epoch 1: 100%|██████████| 22/22 [00:05<00:00,  4.08it/s, loss=5.89, v_num=0]
Epoch 1: 100%|██████████| 22/22 [00:05<00:00,  4.08it/s, loss=5.89, v_num=0]
Epoch 1:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.89, v_num=0]
Epoch 2:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.89, v_num=0]
Epoch 2:   5%|▍         | 1/22 [00:02<00:42,  2.02s/it, loss=5.89, v_num=0]
Epoch 2:   5%|▍         | 1/22 [00:02<00:43,  2.08s/it, loss=5.89, v_num=0]
Epoch 2:   9%|▉         | 2/22 [00:02<00:21,  1.08s/it, loss=5.89, v_num=0]
Epoch 2:   9%|▉         | 2/22 [00:02<00:22,  1.12s/it, loss=5.88, v_num=0]
Epoch 2:  14%|█▎        | 3/22 [00:02<00:14,  1.29it/s, loss=5.88, v_num=0]
Epoch 2:  14%|█▎        | 3/22 [00:02<00:15,  1.26it/s, loss=5.88, v_num=0]
Epoch 2:  18%|█▊        | 4/22 [00:02<00:11,  1.61it/s, loss=5.88, v_num=0]
Epoch 2:  18%|█▊        | 4/22 [00:02<00:11,  1.58it/s, loss=5.88, v_num=0]
Epoch 2:  23%|██▎       | 5/22 [00:02<00:08,  1.90it/s, loss=5.88, v_num=0]
Epoch 2:  23%|██▎       | 5/22 [00:02<00:09,  1.86it/s, loss=5.88, v_num=0]
Epoch 2:  27%|██▋       | 6/22 [00:02<00:07,  2.16it/s, loss=5.88, v_num=0]
Epoch 2:  27%|██▋       | 6/22 [00:02<00:07,  2.11it/s, loss=5.88, v_num=0]
Epoch 2:  32%|███▏      | 7/22 [00:02<00:06,  2.38it/s, loss=5.88, v_num=0]
Epoch 2:  32%|███▏      | 7/22 [00:02<00:06,  2.34it/s, loss=5.88, v_num=0]
Epoch 2:  36%|███▋      | 8/22 [00:03<00:05,  2.59it/s, loss=5.88, v_num=0]
Epoch 2:  36%|███▋      | 8/22 [00:03<00:05,  2.54it/s, loss=5.88, v_num=0]
Epoch 2:  41%|████      | 9/22 [00:03<00:04,  2.74it/s, loss=5.88, v_num=0]
Epoch 2:  41%|████      | 9/22 [00:03<00:04,  2.69it/s, loss=5.88, v_num=0]
Epoch 2:  45%|████▌     | 10/22 [00:03<00:04,  2.91it/s, loss=5.88, v_num=0]
Epoch 2:  45%|████▌     | 10/22 [00:03<00:04,  2.86it/s, loss=5.87, v_num=0]
Epoch 2:  50%|█████     | 11/22 [00:03<00:03,  3.06it/s, loss=5.87, v_num=0]
Epoch 2:  50%|█████     | 11/22 [00:03<00:03,  3.01it/s, loss=5.87, v_num=0]
Epoch 2:  55%|█████▍    | 12/22 [00:03<00:03,  3.21it/s, loss=5.87, v_num=0]
Epoch 2:  55%|█████▍    | 12/22 [00:03<00:03,  3.16it/s, loss=5.87, v_num=0]
Epoch 2:  59%|█████▉    | 13/22 [00:03<00:02,  3.34it/s, loss=5.87, v_num=0]
Epoch 2:  59%|█████▉    | 13/22 [00:03<00:02,  3.29it/s, loss=5.87, v_num=0]
Epoch 2:  64%|██████▎   | 14/22 [00:04<00:02,  3.47it/s, loss=5.87, v_num=0]
Epoch 2:  64%|██████▎   | 14/22 [00:04<00:02,  3.41it/s, loss=5.87, v_num=0]
Epoch 2:  68%|██████▊   | 15/22 [00:04<00:01,  3.58it/s, loss=5.87, v_num=0]
Epoch 2:  68%|██████▊   | 15/22 [00:04<00:01,  3.53it/s, loss=5.87, v_num=0]
Epoch 2:  73%|███████▎  | 16/22 [00:04<00:01,  3.69it/s, loss=5.87, v_num=0]
Epoch 2:  73%|███████▎  | 16/22 [00:04<00:01,  3.64it/s, loss=5.86, v_num=0]
Epoch 2:  77%|███████▋  | 17/22 [00:04<00:01,  3.77it/s, loss=5.86, v_num=0]
Epoch 2:  77%|███████▋  | 17/22 [00:04<00:01,  3.72it/s, loss=5.86, v_num=0]
Epoch 2:  82%|████████▏ | 18/22 [00:04<00:01,  3.86it/s, loss=5.86, v_num=0]
Epoch 2:  82%|████████▏ | 18/22 [00:04<00:01,  3.81it/s, loss=5.86, v_num=0]
Epoch 2:  86%|████████▋ | 19/22 [00:04<00:00,  3.95it/s, loss=5.86, v_num=0]
Epoch 2:  86%|████████▋ | 19/22 [00:04<00:00,  3.90it/s, loss=5.85, v_num=0]
Epoch 2:  91%|█████████ | 20/22 [00:04<00:00,  4.02it/s, loss=5.85, v_num=0]
Epoch 2:  91%|█████████ | 20/22 [00:05<00:00,  3.98it/s, loss=5.85, v_num=0]
Epoch 2:  95%|█████████▌| 21/22 [00:05<00:00,  4.10it/s, loss=5.85, v_num=0]
Epoch 2:  95%|█████████▌| 21/22 [00:05<00:00,  4.05it/s, loss=5.85, v_num=0]
Epoch 2: 100%|██████████| 22/22 [00:05<00:00,  4.17it/s, loss=5.85, v_num=0]
Epoch 2: 100%|██████████| 22/22 [00:05<00:00,  4.13it/s, loss=5.84, v_num=0]
Epoch 2: 100%|██████████| 22/22 [00:05<00:00,  4.13it/s, loss=5.84, v_num=0]
Epoch 2:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.84, v_num=0]
Epoch 3:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.84, v_num=0]
Epoch 3:   5%|▍         | 1/22 [00:02<00:43,  2.09s/it, loss=5.84, v_num=0]
Epoch 3:   5%|▍         | 1/22 [00:02<00:45,  2.14s/it, loss=5.84, v_num=0]
Epoch 3:   9%|▉         | 2/22 [00:02<00:22,  1.12s/it, loss=5.84, v_num=0]
Epoch 3:   9%|▉         | 2/22 [00:02<00:22,  1.15s/it, loss=5.83, v_num=0]
Epoch 3:  14%|█▎        | 3/22 [00:02<00:15,  1.26it/s, loss=5.83, v_num=0]
Epoch 3:  14%|█▎        | 3/22 [00:02<00:15,  1.23it/s, loss=5.83, v_num=0]
Epoch 3:  18%|█▊        | 4/22 [00:02<00:11,  1.57it/s, loss=5.83, v_num=0]
Epoch 3:  18%|█▊        | 4/22 [00:02<00:11,  1.54it/s, loss=5.82, v_num=0]
Epoch 3:  23%|██▎       | 5/22 [00:02<00:09,  1.85it/s, loss=5.82, v_num=0]
Epoch 3:  23%|██▎       | 5/22 [00:02<00:09,  1.82it/s, loss=5.82, v_num=0]
Epoch 3:  27%|██▋       | 6/22 [00:02<00:07,  2.11it/s, loss=5.82, v_num=0]
Epoch 3:  27%|██▋       | 6/22 [00:02<00:07,  2.06it/s, loss=5.82, v_num=0]
Epoch 3:  32%|███▏      | 7/22 [00:02<00:06,  2.34it/s, loss=5.82, v_num=0]
Epoch 3:  32%|███▏      | 7/22 [00:03<00:06,  2.29it/s, loss=5.81, v_num=0]
Epoch 3:  36%|███▋      | 8/22 [00:03<00:05,  2.54it/s, loss=5.81, v_num=0]
Epoch 3:  36%|███▋      | 8/22 [00:03<00:05,  2.49it/s, loss=5.81, v_num=0]
Epoch 3:  41%|████      | 9/22 [00:03<00:04,  2.67it/s, loss=5.81, v_num=0]
Epoch 3:  41%|████      | 9/22 [00:03<00:04,  2.62it/s, loss=5.8, v_num=0]
Epoch 3:  45%|████▌     | 10/22 [00:03<00:04,  2.83it/s, loss=5.8, v_num=0]
Epoch 3:  45%|████▌     | 10/22 [00:03<00:04,  2.79it/s, loss=5.8, v_num=0]
Epoch 3:  50%|█████     | 11/22 [00:03<00:03,  2.99it/s, loss=5.8, v_num=0]
Epoch 3:  50%|█████     | 11/22 [00:03<00:03,  2.94it/s, loss=5.8, v_num=0]
Epoch 3:  55%|█████▍    | 12/22 [00:03<00:03,  3.14it/s, loss=5.8, v_num=0]
Epoch 3:  55%|█████▍    | 12/22 [00:03<00:03,  3.09it/s, loss=5.79, v_num=0]
Epoch 3:  59%|█████▉    | 13/22 [00:03<00:02,  3.27it/s, loss=5.79, v_num=0]
Epoch 3:  59%|█████▉    | 13/22 [00:04<00:02,  3.22it/s, loss=5.79, v_num=0]
Epoch 3:  64%|██████▎   | 14/22 [00:04<00:02,  3.39it/s, loss=5.79, v_num=0]
Epoch 3:  64%|██████▎   | 14/22 [00:04<00:02,  3.34it/s, loss=5.79, v_num=0]
Epoch 3:  68%|██████▊   | 15/22 [00:04<00:01,  3.51it/s, loss=5.79, v_num=0]
Epoch 3:  68%|██████▊   | 15/22 [00:04<00:02,  3.46it/s, loss=5.79, v_num=0]
Epoch 3:  73%|███████▎  | 16/22 [00:04<00:01,  3.61it/s, loss=5.79, v_num=0]
Epoch 3:  73%|███████▎  | 16/22 [00:04<00:01,  3.56it/s, loss=5.79, v_num=0]
Epoch 3:  77%|███████▋  | 17/22 [00:04<00:01,  3.67it/s, loss=5.79, v_num=0]
Epoch 3:  77%|███████▋  | 17/22 [00:04<00:01,  3.62it/s, loss=5.78, v_num=0]
Epoch 3:  82%|████████▏ | 18/22 [00:04<00:01,  3.76it/s, loss=5.78, v_num=0]
Epoch 3:  82%|████████▏ | 18/22 [00:04<00:01,  3.71it/s, loss=5.78, v_num=0]
Epoch 3:  86%|████████▋ | 19/22 [00:04<00:00,  3.85it/s, loss=5.78, v_num=0]
Epoch 3:  86%|████████▋ | 19/22 [00:04<00:00,  3.80it/s, loss=5.78, v_num=0]
Epoch 3:  91%|█████████ | 20/22 [00:05<00:00,  3.93it/s, loss=5.78, v_num=0]
Epoch 3:  91%|█████████ | 20/22 [00:05<00:00,  3.88it/s, loss=5.79, v_num=0]
Epoch 3:  95%|█████████▌| 21/22 [00:05<00:00,  4.01it/s, loss=5.79, v_num=0]
Epoch 3:  95%|█████████▌| 21/22 [00:05<00:00,  3.96it/s, loss=5.79, v_num=0]
Epoch 3: 100%|██████████| 22/22 [00:05<00:00,  4.08it/s, loss=5.79, v_num=0]
Epoch 3: 100%|██████████| 22/22 [00:05<00:00,  4.03it/s, loss=5.78, v_num=0]
Epoch 3: 100%|██████████| 22/22 [00:05<00:00,  4.03it/s, loss=5.78, v_num=0]
Epoch 3:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.78, v_num=0]
Epoch 4:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.78, v_num=0]
Epoch 4:   5%|▍         | 1/22 [00:02<00:43,  2.06s/it, loss=5.78, v_num=0]
Epoch 4:   5%|▍         | 1/22 [00:02<00:44,  2.11s/it, loss=5.78, v_num=0]
Epoch 4:   9%|▉         | 2/22 [00:02<00:22,  1.10s/it, loss=5.78, v_num=0]
Epoch 4:   9%|▉         | 2/22 [00:02<00:22,  1.13s/it, loss=5.78, v_num=0]
Epoch 4:  14%|█▎        | 3/22 [00:02<00:14,  1.28it/s, loss=5.78, v_num=0]
Epoch 4:  14%|█▎        | 3/22 [00:02<00:15,  1.24it/s, loss=5.77, v_num=0]
Epoch 4:  18%|█▊        | 4/22 [00:02<00:11,  1.59it/s, loss=5.77, v_num=0]
Epoch 4:  18%|█▊        | 4/22 [00:02<00:11,  1.56it/s, loss=5.77, v_num=0]
Epoch 4:  23%|██▎       | 5/22 [00:02<00:09,  1.88it/s, loss=5.77, v_num=0]
Epoch 4:  23%|██▎       | 5/22 [00:02<00:09,  1.84it/s, loss=5.77, v_num=0]
Epoch 4:  27%|██▋       | 6/22 [00:02<00:07,  2.13it/s, loss=5.77, v_num=0]
Epoch 4:  27%|██▋       | 6/22 [00:02<00:07,  2.09it/s, loss=5.77, v_num=0]
Epoch 4:  32%|███▏      | 7/22 [00:02<00:06,  2.36it/s, loss=5.77, v_num=0]
Epoch 4:  32%|███▏      | 7/22 [00:03<00:06,  2.31it/s, loss=5.76, v_num=0]
Epoch 4:  36%|███▋      | 8/22 [00:03<00:05,  2.57it/s, loss=5.76, v_num=0]
Epoch 4:  36%|███▋      | 8/22 [00:03<00:05,  2.52it/s, loss=5.76, v_num=0]
Epoch 4:  41%|████      | 9/22 [00:03<00:04,  2.71it/s, loss=5.76, v_num=0]
Epoch 4:  41%|████      | 9/22 [00:03<00:04,  2.66it/s, loss=5.75, v_num=0]
Epoch 4:  45%|████▌     | 10/22 [00:03<00:04,  2.88it/s, loss=5.75, v_num=0]
Epoch 4:  45%|████▌     | 10/22 [00:03<00:04,  2.83it/s, loss=5.75, v_num=0]
Epoch 4:  50%|█████     | 11/22 [00:03<00:03,  3.04it/s, loss=5.75, v_num=0]
Epoch 4:  50%|█████     | 11/22 [00:03<00:03,  2.99it/s, loss=5.75, v_num=0]
Epoch 4:  55%|█████▍    | 12/22 [00:03<00:03,  3.18it/s, loss=5.75, v_num=0]
Epoch 4:  55%|█████▍    | 12/22 [00:03<00:03,  3.13it/s, loss=5.75, v_num=0]
Epoch 4:  59%|█████▉    | 13/22 [00:03<00:02,  3.32it/s, loss=5.75, v_num=0]
Epoch 4:  59%|█████▉    | 13/22 [00:03<00:02,  3.27it/s, loss=5.74, v_num=0]
Epoch 4:  64%|██████▎   | 14/22 [00:04<00:02,  3.44it/s, loss=5.74, v_num=0]
Epoch 4:  64%|██████▎   | 14/22 [00:04<00:02,  3.39it/s, loss=5.74, v_num=0]
Epoch 4:  68%|██████▊   | 15/22 [00:04<00:01,  3.56it/s, loss=5.74, v_num=0]
Epoch 4:  68%|██████▊   | 15/22 [00:04<00:01,  3.50it/s, loss=5.74, v_num=0]
Epoch 4:  73%|███████▎  | 16/22 [00:04<00:01,  3.65it/s, loss=5.74, v_num=0]
Epoch 4:  73%|███████▎  | 16/22 [00:04<00:01,  3.61it/s, loss=5.73, v_num=0]
Epoch 4:  77%|███████▋  | 17/22 [00:04<00:01,  3.74it/s, loss=5.73, v_num=0]
Epoch 4:  77%|███████▋  | 17/22 [00:04<00:01,  3.69it/s, loss=5.73, v_num=0]
Epoch 4:  82%|████████▏ | 18/22 [00:04<00:01,  3.83it/s, loss=5.73, v_num=0]
Epoch 4:  82%|████████▏ | 18/22 [00:04<00:01,  3.78it/s, loss=5.72, v_num=0]
Epoch 4:  86%|████████▋ | 19/22 [00:04<00:00,  3.91it/s, loss=5.72, v_num=0]
Epoch 4:  86%|████████▋ | 19/22 [00:04<00:00,  3.87it/s, loss=5.72, v_num=0]
Epoch 4:  91%|█████████ | 20/22 [00:05<00:00,  4.00it/s, loss=5.72, v_num=0]
Epoch 4:  91%|█████████ | 20/22 [00:05<00:00,  3.95it/s, loss=5.71, v_num=0]
Epoch 4:  95%|█████████▌| 21/22 [00:05<00:00,  4.07it/s, loss=5.71, v_num=0]
Epoch 4:  95%|█████████▌| 21/22 [00:05<00:00,  4.03it/s, loss=5.71, v_num=0]
Epoch 4: 100%|██████████| 22/22 [00:05<00:00,  4.15it/s, loss=5.71, v_num=0]
Epoch 4: 100%|██████████| 22/22 [00:05<00:00,  4.10it/s, loss=5.71, v_num=0]
Epoch 4: 100%|██████████| 22/22 [00:05<00:00,  4.10it/s, loss=5.71, v_num=0]
Epoch 4:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.71, v_num=0]
Epoch 5:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.71, v_num=0]
Epoch 5:   5%|▍         | 1/22 [00:02<00:43,  2.06s/it, loss=5.71, v_num=0]
Epoch 5:   5%|▍         | 1/22 [00:02<00:44,  2.12s/it, loss=5.71, v_num=0]
Epoch 5:   9%|▉         | 2/22 [00:02<00:22,  1.10s/it, loss=5.71, v_num=0]
Epoch 5:   9%|▉         | 2/22 [00:02<00:22,  1.14s/it, loss=5.7, v_num=0]
Epoch 5:  14%|█▎        | 3/22 [00:02<00:14,  1.27it/s, loss=5.7, v_num=0]
Epoch 5:  14%|█▎        | 3/22 [00:02<00:15,  1.24it/s, loss=5.7, v_num=0]
Epoch 5:  18%|█▊        | 4/22 [00:02<00:11,  1.59it/s, loss=5.7, v_num=0]
Epoch 5:  18%|█▊        | 4/22 [00:02<00:11,  1.55it/s, loss=5.7, v_num=0]
Epoch 5:  23%|██▎       | 5/22 [00:02<00:09,  1.87it/s, loss=5.7, v_num=0]
Epoch 5:  23%|██▎       | 5/22 [00:02<00:09,  1.83it/s, loss=5.7, v_num=0]
Epoch 5:  27%|██▋       | 6/22 [00:02<00:07,  2.13it/s, loss=5.7, v_num=0]
Epoch 5:  27%|██▋       | 6/22 [00:02<00:07,  2.08it/s, loss=5.7, v_num=0]
Epoch 5:  32%|███▏      | 7/22 [00:02<00:06,  2.35it/s, loss=5.7, v_num=0]
Epoch 5:  32%|███▏      | 7/22 [00:03<00:06,  2.30it/s, loss=5.71, v_num=0]
Epoch 5:  36%|███▋      | 8/22 [00:03<00:05,  2.56it/s, loss=5.71, v_num=0]
Epoch 5:  36%|███▋      | 8/22 [00:03<00:05,  2.51it/s, loss=5.7, v_num=0]
Epoch 5:  41%|████      | 9/22 [00:03<00:04,  2.72it/s, loss=5.7, v_num=0]
Epoch 5:  41%|████      | 9/22 [00:03<00:04,  2.67it/s, loss=5.71, v_num=0]
Epoch 5:  45%|████▌     | 10/22 [00:03<00:04,  2.89it/s, loss=5.71, v_num=0]
Epoch 5:  45%|████▌     | 10/22 [00:03<00:04,  2.84it/s, loss=5.7, v_num=0]
Epoch 5:  50%|█████     | 11/22 [00:03<00:03,  3.05it/s, loss=5.7, v_num=0]
Epoch 5:  50%|█████     | 11/22 [00:03<00:03,  2.99it/s, loss=5.69, v_num=0]
Epoch 5:  55%|█████▍    | 12/22 [00:03<00:03,  3.18it/s, loss=5.69, v_num=0]
Epoch 5:  55%|█████▍    | 12/22 [00:03<00:03,  3.13it/s, loss=5.69, v_num=0]
Epoch 5:  59%|█████▉    | 13/22 [00:03<00:02,  3.31it/s, loss=5.69, v_num=0]
Epoch 5:  59%|█████▉    | 13/22 [00:03<00:02,  3.26it/s, loss=5.68, v_num=0]
Epoch 5:  64%|██████▎   | 14/22 [00:04<00:02,  3.44it/s, loss=5.68, v_num=0]
Epoch 5:  64%|██████▎   | 14/22 [00:04<00:02,  3.39it/s, loss=5.67, v_num=0]
Epoch 5:  68%|██████▊   | 15/22 [00:04<00:01,  3.55it/s, loss=5.67, v_num=0]
Epoch 5:  68%|██████▊   | 15/22 [00:04<00:01,  3.50it/s, loss=5.68, v_num=0]
Epoch 5:  73%|███████▎  | 16/22 [00:04<00:01,  3.65it/s, loss=5.68, v_num=0]
Epoch 5:  73%|███████▎  | 16/22 [00:04<00:01,  3.61it/s, loss=5.67, v_num=0]
Epoch 5:  77%|███████▋  | 17/22 [00:04<00:01,  3.73it/s, loss=5.67, v_num=0]
Epoch 5:  77%|███████▋  | 17/22 [00:04<00:01,  3.68it/s, loss=5.67, v_num=0]
Epoch 5:  82%|████████▏ | 18/22 [00:04<00:01,  3.82it/s, loss=5.67, v_num=0]
Epoch 5:  82%|████████▏ | 18/22 [00:04<00:01,  3.77it/s, loss=5.68, v_num=0]
Epoch 5:  86%|████████▋ | 19/22 [00:04<00:00,  3.90it/s, loss=5.68, v_num=0]
Epoch 5:  86%|████████▋ | 19/22 [00:04<00:00,  3.86it/s, loss=5.68, v_num=0]
Epoch 5:  91%|█████████ | 20/22 [00:05<00:00,  3.98it/s, loss=5.68, v_num=0]
Epoch 5:  91%|█████████ | 20/22 [00:05<00:00,  3.94it/s, loss=5.68, v_num=0]
Epoch 5:  95%|█████████▌| 21/22 [00:05<00:00,  4.06it/s, loss=5.68, v_num=0]
Epoch 5:  95%|█████████▌| 21/22 [00:05<00:00,  4.02it/s, loss=5.68, v_num=0]
Epoch 5: 100%|██████████| 22/22 [00:05<00:00,  4.14it/s, loss=5.68, v_num=0]
Epoch 5: 100%|██████████| 22/22 [00:05<00:00,  4.09it/s, loss=5.68, v_num=0]
Epoch 5: 100%|██████████| 22/22 [00:05<00:00,  4.09it/s, loss=5.68, v_num=0]
Epoch 5:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.68, v_num=0]
Epoch 6:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.68, v_num=0]
Epoch 6:   5%|▍         | 1/22 [00:02<00:43,  2.06s/it, loss=5.68, v_num=0]
Epoch 6:   5%|▍         | 1/22 [00:02<00:44,  2.11s/it, loss=5.68, v_num=0]
Epoch 6:   9%|▉         | 2/22 [00:02<00:22,  1.10s/it, loss=5.68, v_num=0]
Epoch 6:   9%|▉         | 2/22 [00:02<00:22,  1.13s/it, loss=5.68, v_num=0]
Epoch 6:  14%|█▎        | 3/22 [00:02<00:14,  1.27it/s, loss=5.68, v_num=0]
Epoch 6:  14%|█▎        | 3/22 [00:02<00:15,  1.24it/s, loss=5.68, v_num=0]
Epoch 6:  18%|█▊        | 4/22 [00:02<00:11,  1.59it/s, loss=5.68, v_num=0]
Epoch 6:  18%|█▊        | 4/22 [00:02<00:11,  1.55it/s, loss=5.67, v_num=0]
Epoch 6:  23%|██▎       | 5/22 [00:02<00:09,  1.87it/s, loss=5.67, v_num=0]
Epoch 6:  23%|██▎       | 5/22 [00:02<00:09,  1.83it/s, loss=5.67, v_num=0]
Epoch 6:  27%|██▋       | 6/22 [00:02<00:07,  2.13it/s, loss=5.67, v_num=0]
Epoch 6:  27%|██▋       | 6/22 [00:02<00:07,  2.08it/s, loss=5.67, v_num=0]
Epoch 6:  32%|███▏      | 7/22 [00:02<00:06,  2.36it/s, loss=5.67, v_num=0]
Epoch 6:  32%|███▏      | 7/22 [00:03<00:06,  2.31it/s, loss=5.66, v_num=0]
Epoch 6:  36%|███▋      | 8/22 [00:03<00:05,  2.56it/s, loss=5.66, v_num=0]
Epoch 6:  36%|███▋      | 8/22 [00:03<00:05,  2.51it/s, loss=5.67, v_num=0]
Epoch 6:  41%|████      | 9/22 [00:03<00:04,  2.72it/s, loss=5.67, v_num=0]
Epoch 6:  41%|████      | 9/22 [00:03<00:04,  2.67it/s, loss=5.67, v_num=0]
Epoch 6:  45%|████▌     | 10/22 [00:03<00:04,  2.89it/s, loss=5.67, v_num=0]
Epoch 6:  45%|████▌     | 10/22 [00:03<00:04,  2.84it/s, loss=5.67, v_num=0]
Epoch 6:  50%|█████     | 11/22 [00:03<00:03,  3.05it/s, loss=5.67, v_num=0]
Epoch 6:  50%|█████     | 11/22 [00:03<00:03,  2.99it/s, loss=5.68, v_num=0]
Epoch 6:  55%|█████▍    | 12/22 [00:03<00:03,  3.18it/s, loss=5.68, v_num=0]
Epoch 6:  55%|█████▍    | 12/22 [00:03<00:03,  3.13it/s, loss=5.68, v_num=0]
Epoch 6:  59%|█████▉    | 13/22 [00:03<00:02,  3.32it/s, loss=5.68, v_num=0]
Epoch 6:  59%|█████▉    | 13/22 [00:03<00:02,  3.27it/s, loss=5.68, v_num=0]
Epoch 6:  64%|██████▎   | 14/22 [00:04<00:02,  3.44it/s, loss=5.68, v_num=0]
Epoch 6:  64%|██████▎   | 14/22 [00:04<00:02,  3.39it/s, loss=5.68, v_num=0]
Epoch 6:  68%|██████▊   | 15/22 [00:04<00:01,  3.55it/s, loss=5.68, v_num=0]
Epoch 6:  68%|██████▊   | 15/22 [00:04<00:01,  3.50it/s, loss=5.67, v_num=0]
Epoch 6:  73%|███████▎  | 16/22 [00:04<00:01,  3.66it/s, loss=5.67, v_num=0]
Epoch 6:  73%|███████▎  | 16/22 [00:04<00:01,  3.61it/s, loss=5.66, v_num=0]
Epoch 6:  77%|███████▋  | 17/22 [00:04<00:01,  3.74it/s, loss=5.66, v_num=0]
Epoch 6:  77%|███████▋  | 17/22 [00:04<00:01,  3.69it/s, loss=5.67, v_num=0]
Epoch 6:  82%|████████▏ | 18/22 [00:04<00:01,  3.83it/s, loss=5.67, v_num=0]
Epoch 6:  82%|████████▏ | 18/22 [00:04<00:01,  3.78it/s, loss=5.66, v_num=0]
Epoch 6:  86%|████████▋ | 19/22 [00:04<00:00,  3.92it/s, loss=5.66, v_num=0]
Epoch 6:  86%|████████▋ | 19/22 [00:04<00:00,  3.87it/s, loss=5.66, v_num=0]
Epoch 6:  91%|█████████ | 20/22 [00:05<00:00,  4.00it/s, loss=5.66, v_num=0]
Epoch 6:  91%|█████████ | 20/22 [00:05<00:00,  3.95it/s, loss=5.66, v_num=0]
Epoch 6:  95%|█████████▌| 21/22 [00:05<00:00,  4.07it/s, loss=5.66, v_num=0]
Epoch 6:  95%|█████████▌| 21/22 [00:05<00:00,  4.03it/s, loss=5.65, v_num=0]
Epoch 6: 100%|██████████| 22/22 [00:05<00:00,  4.15it/s, loss=5.65, v_num=0]
Epoch 6: 100%|██████████| 22/22 [00:05<00:00,  4.10it/s, loss=5.65, v_num=0]
Epoch 6: 100%|██████████| 22/22 [00:05<00:00,  4.10it/s, loss=5.65, v_num=0]
Epoch 6:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.65, v_num=0]
Epoch 7:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.65, v_num=0]
Epoch 7:   5%|▍         | 1/22 [00:02<00:43,  2.07s/it, loss=5.65, v_num=0]
Epoch 7:   5%|▍         | 1/22 [00:02<00:44,  2.12s/it, loss=5.65, v_num=0]
Epoch 7:   9%|▉         | 2/22 [00:02<00:22,  1.11s/it, loss=5.65, v_num=0]
Epoch 7:   9%|▉         | 2/22 [00:02<00:22,  1.14s/it, loss=5.65, v_num=0]
Epoch 7:  14%|█▎        | 3/22 [00:02<00:15,  1.27it/s, loss=5.65, v_num=0]
Epoch 7:  14%|█▎        | 3/22 [00:02<00:15,  1.23it/s, loss=5.65, v_num=0]
Epoch 7:  18%|█▊        | 4/22 [00:02<00:11,  1.58it/s, loss=5.65, v_num=0]
Epoch 7:  18%|█▊        | 4/22 [00:02<00:11,  1.55it/s, loss=5.64, v_num=0]
Epoch 7:  23%|██▎       | 5/22 [00:02<00:09,  1.87it/s, loss=5.64, v_num=0]
Epoch 7:  23%|██▎       | 5/22 [00:02<00:09,  1.83it/s, loss=5.64, v_num=0]
Epoch 7:  27%|██▋       | 6/22 [00:02<00:07,  2.12it/s, loss=5.64, v_num=0]
Epoch 7:  27%|██▋       | 6/22 [00:02<00:07,  2.08it/s, loss=5.64, v_num=0]
Epoch 7:  32%|███▏      | 7/22 [00:02<00:06,  2.35it/s, loss=5.64, v_num=0]
Epoch 7:  32%|███▏      | 7/22 [00:03<00:06,  2.30it/s, loss=5.64, v_num=0]
Epoch 7:  36%|███▋      | 8/22 [00:03<00:05,  2.55it/s, loss=5.64, v_num=0]
Epoch 7:  36%|███▋      | 8/22 [00:03<00:05,  2.50it/s, loss=5.64, v_num=0]
Epoch 7:  41%|████      | 9/22 [00:03<00:04,  2.67it/s, loss=5.64, v_num=0]
Epoch 7:  41%|████      | 9/22 [00:03<00:04,  2.62it/s, loss=5.64, v_num=0]
Epoch 7:  45%|████▌     | 10/22 [00:03<00:04,  2.84it/s, loss=5.64, v_num=0]
Epoch 7:  45%|████▌     | 10/22 [00:03<00:04,  2.79it/s, loss=5.63, v_num=0]
Epoch 7:  50%|█████     | 11/22 [00:03<00:03,  2.99it/s, loss=5.63, v_num=0]
Epoch 7:  50%|█████     | 11/22 [00:03<00:03,  2.95it/s, loss=5.62, v_num=0]
Epoch 7:  55%|█████▍    | 12/22 [00:03<00:03,  3.14it/s, loss=5.62, v_num=0]
Epoch 7:  55%|█████▍    | 12/22 [00:03<00:03,  3.09it/s, loss=5.62, v_num=0]
Epoch 7:  59%|█████▉    | 13/22 [00:03<00:02,  3.27it/s, loss=5.62, v_num=0]
Epoch 7:  59%|█████▉    | 13/22 [00:04<00:02,  3.22it/s, loss=5.62, v_num=0]
Epoch 7:  64%|██████▎   | 14/22 [00:04<00:02,  3.39it/s, loss=5.62, v_num=0]
Epoch 7:  64%|██████▎   | 14/22 [00:04<00:02,  3.34it/s, loss=5.62, v_num=0]
Epoch 7:  68%|██████▊   | 15/22 [00:04<00:01,  3.50it/s, loss=5.62, v_num=0]
Epoch 7:  68%|██████▊   | 15/22 [00:04<00:02,  3.46it/s, loss=5.61, v_num=0]
Epoch 7:  73%|███████▎  | 16/22 [00:04<00:01,  3.61it/s, loss=5.61, v_num=0]
Epoch 7:  73%|███████▎  | 16/22 [00:04<00:01,  3.56it/s, loss=5.61, v_num=0]
Epoch 7:  77%|███████▋  | 17/22 [00:04<00:01,  3.68it/s, loss=5.61, v_num=0]
Epoch 7:  77%|███████▋  | 17/22 [00:04<00:01,  3.63it/s, loss=5.61, v_num=0]
Epoch 7:  82%|████████▏ | 18/22 [00:04<00:01,  3.77it/s, loss=5.61, v_num=0]
Epoch 7:  82%|████████▏ | 18/22 [00:04<00:01,  3.73it/s, loss=5.61, v_num=0]
Epoch 7:  86%|████████▋ | 19/22 [00:04<00:00,  3.86it/s, loss=5.61, v_num=0]
Epoch 7:  86%|████████▋ | 19/22 [00:04<00:00,  3.81it/s, loss=5.61, v_num=0]
Epoch 7:  91%|█████████ | 20/22 [00:05<00:00,  3.94it/s, loss=5.61, v_num=0]
Epoch 7:  91%|█████████ | 20/22 [00:05<00:00,  3.90it/s, loss=5.6, v_num=0]
Epoch 7:  95%|█████████▌| 21/22 [00:05<00:00,  4.02it/s, loss=5.6, v_num=0]
Epoch 7:  95%|█████████▌| 21/22 [00:05<00:00,  3.97it/s, loss=5.6, v_num=0]
Epoch 7: 100%|██████████| 22/22 [00:05<00:00,  4.09it/s, loss=5.6, v_num=0]
Epoch 7: 100%|██████████| 22/22 [00:05<00:00,  4.05it/s, loss=5.6, v_num=0]
Epoch 7: 100%|██████████| 22/22 [00:05<00:00,  4.05it/s, loss=5.6, v_num=0]
Epoch 7:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.6, v_num=0]
Epoch 8:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.6, v_num=0]
Epoch 8:   5%|▍         | 1/22 [00:02<00:43,  2.08s/it, loss=5.6, v_num=0]
Epoch 8:   5%|▍         | 1/22 [00:02<00:44,  2.14s/it, loss=5.59, v_num=0]
Epoch 8:   9%|▉         | 2/22 [00:02<00:22,  1.11s/it, loss=5.59, v_num=0]
Epoch 8:   9%|▉         | 2/22 [00:02<00:22,  1.14s/it, loss=5.59, v_num=0]
Epoch 8:  14%|█▎        | 3/22 [00:02<00:15,  1.26it/s, loss=5.59, v_num=0]
Epoch 8:  14%|█▎        | 3/22 [00:02<00:15,  1.23it/s, loss=5.59, v_num=0]
Epoch 8:  18%|█▊        | 4/22 [00:02<00:11,  1.58it/s, loss=5.59, v_num=0]
Epoch 8:  18%|█▊        | 4/22 [00:02<00:11,  1.54it/s, loss=5.59, v_num=0]
Epoch 8:  23%|██▎       | 5/22 [00:02<00:09,  1.86it/s, loss=5.59, v_num=0]
Epoch 8:  23%|██▎       | 5/22 [00:02<00:09,  1.82it/s, loss=5.58, v_num=0]
Epoch 8:  27%|██▋       | 6/22 [00:02<00:07,  2.11it/s, loss=5.58, v_num=0]
Epoch 8:  27%|██▋       | 6/22 [00:02<00:07,  2.07it/s, loss=5.57, v_num=0]
Epoch 8:  32%|███▏      | 7/22 [00:02<00:06,  2.34it/s, loss=5.57, v_num=0]
Epoch 8:  32%|███▏      | 7/22 [00:03<00:06,  2.29it/s, loss=5.57, v_num=0]
Epoch 8:  36%|███▋      | 8/22 [00:03<00:05,  2.54it/s, loss=5.57, v_num=0]
Epoch 8:  36%|███▋      | 8/22 [00:03<00:05,  2.50it/s, loss=5.58, v_num=0]
Epoch 8:  41%|████      | 9/22 [00:03<00:04,  2.69it/s, loss=5.58, v_num=0]
Epoch 8:  41%|████      | 9/22 [00:03<00:04,  2.64it/s, loss=5.58, v_num=0]
Epoch 8:  45%|████▌     | 10/22 [00:03<00:04,  2.86it/s, loss=5.58, v_num=0]
Epoch 8:  45%|████▌     | 10/22 [00:03<00:04,  2.81it/s, loss=5.57, v_num=0]
Epoch 8:  50%|█████     | 11/22 [00:03<00:03,  3.02it/s, loss=5.57, v_num=0]
Epoch 8:  50%|█████     | 11/22 [00:03<00:03,  2.96it/s, loss=5.57, v_num=0]
Epoch 8:  55%|█████▍    | 12/22 [00:03<00:03,  3.15it/s, loss=5.57, v_num=0]
Epoch 8:  55%|█████▍    | 12/22 [00:03<00:03,  3.11it/s, loss=5.57, v_num=0]
Epoch 8:  59%|█████▉    | 13/22 [00:03<00:02,  3.29it/s, loss=5.57, v_num=0]
Epoch 8:  59%|█████▉    | 13/22 [00:04<00:02,  3.24it/s, loss=5.56, v_num=0]
Epoch 8:  64%|██████▎   | 14/22 [00:04<00:02,  3.41it/s, loss=5.56, v_num=0]
Epoch 8:  64%|██████▎   | 14/22 [00:04<00:02,  3.36it/s, loss=5.55, v_num=0]
Epoch 8:  68%|██████▊   | 15/22 [00:04<00:01,  3.52it/s, loss=5.55, v_num=0]
Epoch 8:  68%|██████▊   | 15/22 [00:04<00:02,  3.47it/s, loss=5.56, v_num=0]
Epoch 8:  73%|███████▎  | 16/22 [00:04<00:01,  3.62it/s, loss=5.56, v_num=0]
Epoch 8:  73%|███████▎  | 16/22 [00:04<00:01,  3.58it/s, loss=5.55, v_num=0]
Epoch 8:  77%|███████▋  | 17/22 [00:04<00:01,  3.69it/s, loss=5.55, v_num=0]
Epoch 8:  77%|███████▋  | 17/22 [00:04<00:01,  3.64it/s, loss=5.55, v_num=0]
Epoch 8:  82%|████████▏ | 18/22 [00:04<00:01,  3.78it/s, loss=5.55, v_num=0]
Epoch 8:  82%|████████▏ | 18/22 [00:04<00:01,  3.73it/s, loss=5.55, v_num=0]
Epoch 8:  86%|████████▋ | 19/22 [00:04<00:00,  3.87it/s, loss=5.55, v_num=0]
Epoch 8:  86%|████████▋ | 19/22 [00:04<00:00,  3.82it/s, loss=5.54, v_num=0]
Epoch 8:  91%|█████████ | 20/22 [00:05<00:00,  3.95it/s, loss=5.54, v_num=0]
Epoch 8:  91%|█████████ | 20/22 [00:05<00:00,  3.90it/s, loss=5.54, v_num=0]
Epoch 8:  95%|█████████▌| 21/22 [00:05<00:00,  4.02it/s, loss=5.54, v_num=0]
Epoch 8:  95%|█████████▌| 21/22 [00:05<00:00,  3.98it/s, loss=5.54, v_num=0]
Epoch 8: 100%|██████████| 22/22 [00:05<00:00,  4.10it/s, loss=5.54, v_num=0]
Epoch 8: 100%|██████████| 22/22 [00:05<00:00,  4.05it/s, loss=5.54, v_num=0]
Epoch 8: 100%|██████████| 22/22 [00:05<00:00,  4.05it/s, loss=5.54, v_num=0]
Epoch 8:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.54, v_num=0]
Epoch 9:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.54, v_num=0]
Epoch 9:   5%|▍         | 1/22 [00:02<00:43,  2.08s/it, loss=5.54, v_num=0]
Epoch 9:   5%|▍         | 1/22 [00:02<00:44,  2.14s/it, loss=5.54, v_num=0]
Epoch 9:   9%|▉         | 2/22 [00:02<00:22,  1.12s/it, loss=5.54, v_num=0]
Epoch 9:   9%|▉         | 2/22 [00:02<00:22,  1.15s/it, loss=5.53, v_num=0]
Epoch 9:  14%|█▎        | 3/22 [00:02<00:15,  1.26it/s, loss=5.53, v_num=0]
Epoch 9:  14%|█▎        | 3/22 [00:02<00:15,  1.23it/s, loss=5.53, v_num=0]
Epoch 9:  18%|█▊        | 4/22 [00:02<00:11,  1.58it/s, loss=5.53, v_num=0]
Epoch 9:  18%|█▊        | 4/22 [00:02<00:11,  1.54it/s, loss=5.53, v_num=0]
Epoch 9:  23%|██▎       | 5/22 [00:02<00:09,  1.86it/s, loss=5.53, v_num=0]
Epoch 9:  23%|██▎       | 5/22 [00:02<00:09,  1.82it/s, loss=5.53, v_num=0]
Epoch 9:  27%|██▋       | 6/22 [00:02<00:07,  2.11it/s, loss=5.53, v_num=0]
Epoch 9:  27%|██▋       | 6/22 [00:02<00:07,  2.07it/s, loss=5.53, v_num=0]
Epoch 9:  32%|███▏      | 7/22 [00:02<00:06,  2.33it/s, loss=5.53, v_num=0]
Epoch 9:  32%|███▏      | 7/22 [00:03<00:06,  2.29it/s, loss=5.53, v_num=0]
Epoch 9:  36%|███▋      | 8/22 [00:03<00:05,  2.54it/s, loss=5.53, v_num=0]
Epoch 9:  36%|███▋      | 8/22 [00:03<00:05,  2.49it/s, loss=5.52, v_num=0]
Epoch 9:  41%|████      | 9/22 [00:03<00:04,  2.69it/s, loss=5.52, v_num=0]
Epoch 9:  41%|████      | 9/22 [00:03<00:04,  2.64it/s, loss=5.52, v_num=0]
Epoch 9:  45%|████▌     | 10/22 [00:03<00:04,  2.86it/s, loss=5.52, v_num=0]
Epoch 9:  45%|████▌     | 10/22 [00:03<00:04,  2.81it/s, loss=5.52, v_num=0]
Epoch 9:  50%|█████     | 11/22 [00:03<00:03,  3.01it/s, loss=5.52, v_num=0]
Epoch 9:  50%|█████     | 11/22 [00:03<00:03,  2.96it/s, loss=5.52, v_num=0]
Epoch 9:  55%|█████▍    | 12/22 [00:03<00:03,  3.15it/s, loss=5.52, v_num=0]
Epoch 9:  55%|█████▍    | 12/22 [00:03<00:03,  3.11it/s, loss=5.52, v_num=0]
Epoch 9:  59%|█████▉    | 13/22 [00:03<00:02,  3.29it/s, loss=5.52, v_num=0]
Epoch 9:  59%|█████▉    | 13/22 [00:04<00:02,  3.24it/s, loss=5.51, v_num=0]
Epoch 9:  64%|██████▎   | 14/22 [00:04<00:02,  3.41it/s, loss=5.51, v_num=0]
Epoch 9:  64%|██████▎   | 14/22 [00:04<00:02,  3.36it/s, loss=5.51, v_num=0]
Epoch 9:  68%|██████▊   | 15/22 [00:04<00:01,  3.52it/s, loss=5.51, v_num=0]
Epoch 9:  68%|██████▊   | 15/22 [00:04<00:02,  3.47it/s, loss=5.51, v_num=0]
Epoch 9:  73%|███████▎  | 16/22 [00:04<00:01,  3.63it/s, loss=5.51, v_num=0]
Epoch 9:  73%|███████▎  | 16/22 [00:04<00:01,  3.58it/s, loss=5.51, v_num=0]
Epoch 9:  77%|███████▋  | 17/22 [00:04<00:01,  3.68it/s, loss=5.51, v_num=0]
Epoch 9:  77%|███████▋  | 17/22 [00:04<00:01,  3.63it/s, loss=5.51, v_num=0]
Epoch 9:  82%|████████▏ | 18/22 [00:04<00:01,  3.78it/s, loss=5.51, v_num=0]
Epoch 9:  82%|████████▏ | 18/22 [00:04<00:01,  3.73it/s, loss=5.51, v_num=0]
Epoch 9:  86%|████████▋ | 19/22 [00:04<00:00,  3.86it/s, loss=5.51, v_num=0]
Epoch 9:  86%|████████▋ | 19/22 [00:04<00:00,  3.82it/s, loss=5.5, v_num=0]
Epoch 9:  91%|█████████ | 20/22 [00:05<00:00,  3.95it/s, loss=5.5, v_num=0]
Epoch 9:  91%|█████████ | 20/22 [00:05<00:00,  3.90it/s, loss=5.51, v_num=0]
Epoch 9:  95%|█████████▌| 21/22 [00:05<00:00,  4.02it/s, loss=5.51, v_num=0]
Epoch 9:  95%|█████████▌| 21/22 [00:05<00:00,  3.97it/s, loss=5.51, v_num=0]
Epoch 9: 100%|██████████| 22/22 [00:05<00:00,  4.09it/s, loss=5.51, v_num=0]
Epoch 9: 100%|██████████| 22/22 [00:05<00:00,  4.05it/s, loss=5.5, v_num=0]
Epoch 9: 100%|██████████| 22/22 [00:05<00:00,  4.05it/s, loss=5.5, v_num=0]
Epoch 9:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.5, v_num=0]
Epoch 10:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.5, v_num=0]
Epoch 10:   5%|▍         | 1/22 [00:02<00:44,  2.11s/it, loss=5.5, v_num=0]
Epoch 10:   5%|▍         | 1/22 [00:02<00:45,  2.17s/it, loss=5.5, v_num=0]
Epoch 10:   9%|▉         | 2/22 [00:02<00:22,  1.13s/it, loss=5.5, v_num=0]
Epoch 10:   9%|▉         | 2/22 [00:02<00:23,  1.16s/it, loss=5.49, v_num=0]
Epoch 10:  14%|█▎        | 3/22 [00:02<00:15,  1.24it/s, loss=5.49, v_num=0]
Epoch 10:  14%|█▎        | 3/22 [00:02<00:15,  1.21it/s, loss=5.49, v_num=0]
Epoch 10:  18%|█▊        | 4/22 [00:02<00:11,  1.56it/s, loss=5.49, v_num=0]
Epoch 10:  18%|█▊        | 4/22 [00:02<00:11,  1.52it/s, loss=5.49, v_num=0]
Epoch 10:  23%|██▎       | 5/22 [00:02<00:09,  1.84it/s, loss=5.49, v_num=0]
Epoch 10:  23%|██▎       | 5/22 [00:02<00:09,  1.80it/s, loss=5.48, v_num=0]
Epoch 10:  27%|██▋       | 6/22 [00:02<00:07,  2.09it/s, loss=5.48, v_num=0]
Epoch 10:  27%|██▋       | 6/22 [00:02<00:07,  2.05it/s, loss=5.49, v_num=0]
Epoch 10:  32%|███▏      | 7/22 [00:03<00:06,  2.32it/s, loss=5.49, v_num=0]
Epoch 10:  32%|███▏      | 7/22 [00:03<00:06,  2.27it/s, loss=5.49, v_num=0]
Epoch 10:  36%|███▋      | 8/22 [00:03<00:05,  2.52it/s, loss=5.49, v_num=0]
Epoch 10:  36%|███▋      | 8/22 [00:03<00:05,  2.47it/s, loss=5.49, v_num=0]
Epoch 10:  41%|████      | 9/22 [00:03<00:04,  2.66it/s, loss=5.49, v_num=0]
Epoch 10:  41%|████      | 9/22 [00:03<00:04,  2.61it/s, loss=5.48, v_num=0]
Epoch 10:  45%|████▌     | 10/22 [00:03<00:04,  2.83it/s, loss=5.48, v_num=0]
Epoch 10:  45%|████▌     | 10/22 [00:03<00:04,  2.78it/s, loss=5.48, v_num=0]
Epoch 10:  50%|█████     | 11/22 [00:03<00:03,  2.98it/s, loss=5.48, v_num=0]
Epoch 10:  50%|█████     | 11/22 [00:03<00:03,  2.93it/s, loss=5.48, v_num=0]
Epoch 10:  55%|█████▍    | 12/22 [00:03<00:03,  3.12it/s, loss=5.48, v_num=0]
Epoch 10:  55%|█████▍    | 12/22 [00:03<00:03,  3.08it/s, loss=5.48, v_num=0]
Epoch 10:  59%|█████▉    | 13/22 [00:03<00:02,  3.26it/s, loss=5.48, v_num=0]
Epoch 10:  59%|█████▉    | 13/22 [00:04<00:02,  3.21it/s, loss=5.48, v_num=0]
Epoch 10:  64%|██████▎   | 14/22 [00:04<00:02,  3.38it/s, loss=5.48, v_num=0]
Epoch 10:  64%|██████▎   | 14/22 [00:04<00:02,  3.33it/s, loss=5.48, v_num=0]
Epoch 10:  68%|██████▊   | 15/22 [00:04<00:02,  3.49it/s, loss=5.48, v_num=0]
Epoch 10:  68%|██████▊   | 15/22 [00:04<00:02,  3.44it/s, loss=5.48, v_num=0]
Epoch 10:  73%|███████▎  | 16/22 [00:04<00:01,  3.60it/s, loss=5.48, v_num=0]
Epoch 10:  73%|███████▎  | 16/22 [00:04<00:01,  3.55it/s, loss=5.48, v_num=0]
Epoch 10:  77%|███████▋  | 17/22 [00:04<00:01,  3.64it/s, loss=5.48, v_num=0]
Epoch 10:  77%|███████▋  | 17/22 [00:04<00:01,  3.59it/s, loss=5.48, v_num=0]
Epoch 10:  82%|████████▏ | 18/22 [00:04<00:01,  3.73it/s, loss=5.48, v_num=0]
Epoch 10:  82%|████████▏ | 18/22 [00:04<00:01,  3.69it/s, loss=5.47, v_num=0]
Epoch 10:  86%|████████▋ | 19/22 [00:04<00:00,  3.82it/s, loss=5.47, v_num=0]
Epoch 10:  86%|████████▋ | 19/22 [00:05<00:00,  3.77it/s, loss=5.46, v_num=0]
Epoch 10:  91%|█████████ | 20/22 [00:05<00:00,  3.90it/s, loss=5.46, v_num=0]
Epoch 10:  91%|█████████ | 20/22 [00:05<00:00,  3.86it/s, loss=5.47, v_num=0]
Epoch 10:  95%|█████████▌| 21/22 [00:05<00:00,  3.98it/s, loss=5.47, v_num=0]
Epoch 10:  95%|█████████▌| 21/22 [00:05<00:00,  3.93it/s, loss=5.47, v_num=0]
Epoch 10: 100%|██████████| 22/22 [00:05<00:00,  4.05it/s, loss=5.47, v_num=0]
Epoch 10: 100%|██████████| 22/22 [00:05<00:00,  4.01it/s, loss=5.47, v_num=0]
Epoch 10: 100%|██████████| 22/22 [00:05<00:00,  4.01it/s, loss=5.47, v_num=0]
Epoch 10:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.47, v_num=0]
Epoch 11:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.47, v_num=0]
Epoch 11:   5%|▍         | 1/22 [00:02<00:43,  2.08s/it, loss=5.47, v_num=0]
Epoch 11:   5%|▍         | 1/22 [00:02<00:44,  2.14s/it, loss=5.47, v_num=0]
Epoch 11:   9%|▉         | 2/22 [00:02<00:22,  1.12s/it, loss=5.47, v_num=0]
Epoch 11:   9%|▉         | 2/22 [00:02<00:22,  1.15s/it, loss=5.47, v_num=0]
Epoch 11:  14%|█▎        | 3/22 [00:02<00:15,  1.26it/s, loss=5.47, v_num=0]
Epoch 11:  14%|█▎        | 3/22 [00:02<00:15,  1.22it/s, loss=5.46, v_num=0]
Epoch 11:  18%|█▊        | 4/22 [00:02<00:11,  1.57it/s, loss=5.46, v_num=0]
Epoch 11:  18%|█▊        | 4/22 [00:02<00:11,  1.54it/s, loss=5.46, v_num=0]
Epoch 11:  23%|██▎       | 5/22 [00:02<00:09,  1.85it/s, loss=5.46, v_num=0]
Epoch 11:  23%|██▎       | 5/22 [00:02<00:09,  1.81it/s, loss=5.46, v_num=0]
Epoch 11:  27%|██▋       | 6/22 [00:02<00:07,  2.10it/s, loss=5.46, v_num=0]
Epoch 11:  27%|██▋       | 6/22 [00:02<00:07,  2.06it/s, loss=5.46, v_num=0]
Epoch 11:  32%|███▏      | 7/22 [00:03<00:06,  2.33it/s, loss=5.46, v_num=0]
Epoch 11:  32%|███▏      | 7/22 [00:03<00:06,  2.29it/s, loss=5.46, v_num=0]
Epoch 11:  36%|███▋      | 8/22 [00:03<00:05,  2.53it/s, loss=5.46, v_num=0]
Epoch 11:  36%|███▋      | 8/22 [00:03<00:05,  2.49it/s, loss=5.46, v_num=0]
Epoch 11:  41%|████      | 9/22 [00:03<00:04,  2.67it/s, loss=5.46, v_num=0]
Epoch 11:  41%|████      | 9/22 [00:03<00:04,  2.62it/s, loss=5.45, v_num=0]
Epoch 11:  45%|████▌     | 10/22 [00:03<00:04,  2.84it/s, loss=5.45, v_num=0]
Epoch 11:  45%|████▌     | 10/22 [00:03<00:04,  2.79it/s, loss=5.45, v_num=0]
Epoch 11:  50%|█████     | 11/22 [00:03<00:03,  2.99it/s, loss=5.45, v_num=0]
Epoch 11:  50%|█████     | 11/22 [00:03<00:03,  2.94it/s, loss=5.45, v_num=0]
Epoch 11:  55%|█████▍    | 12/22 [00:03<00:03,  3.13it/s, loss=5.45, v_num=0]
Epoch 11:  55%|█████▍    | 12/22 [00:03<00:03,  3.08it/s, loss=5.45, v_num=0]
Epoch 11:  59%|█████▉    | 13/22 [00:03<00:02,  3.27it/s, loss=5.45, v_num=0]
Epoch 11:  59%|█████▉    | 13/22 [00:04<00:02,  3.22it/s, loss=5.45, v_num=0]
Epoch 11:  64%|██████▎   | 14/22 [00:04<00:02,  3.39it/s, loss=5.45, v_num=0]
Epoch 11:  64%|██████▎   | 14/22 [00:04<00:02,  3.34it/s, loss=5.45, v_num=0]
Epoch 11:  68%|██████▊   | 15/22 [00:04<00:02,  3.50it/s, loss=5.45, v_num=0]
Epoch 11:  68%|██████▊   | 15/22 [00:04<00:02,  3.45it/s, loss=5.45, v_num=0]
Epoch 11:  73%|███████▎  | 16/22 [00:04<00:01,  3.61it/s, loss=5.45, v_num=0]
Epoch 11:  73%|███████▎  | 16/22 [00:04<00:01,  3.56it/s, loss=5.45, v_num=0]
Epoch 11:  77%|███████▋  | 17/22 [00:04<00:01,  3.66it/s, loss=5.45, v_num=0]
Epoch 11:  77%|███████▋  | 17/22 [00:04<00:01,  3.61it/s, loss=5.45, v_num=0]
Epoch 11:  82%|████████▏ | 18/22 [00:04<00:01,  3.75it/s, loss=5.45, v_num=0]
Epoch 11:  82%|████████▏ | 18/22 [00:04<00:01,  3.71it/s, loss=5.46, v_num=0]
Epoch 11:  86%|████████▋ | 19/22 [00:04<00:00,  3.84it/s, loss=5.46, v_num=0]
Epoch 11:  86%|████████▋ | 19/22 [00:05<00:00,  3.79it/s, loss=5.46, v_num=0]
Epoch 11:  91%|█████████ | 20/22 [00:05<00:00,  3.92it/s, loss=5.46, v_num=0]
Epoch 11:  91%|█████████ | 20/22 [00:05<00:00,  3.87it/s, loss=5.46, v_num=0]
Epoch 11:  95%|█████████▌| 21/22 [00:05<00:00,  4.00it/s, loss=5.46, v_num=0]
Epoch 11:  95%|█████████▌| 21/22 [00:05<00:00,  3.95it/s, loss=5.46, v_num=0]
Epoch 11: 100%|██████████| 22/22 [00:05<00:00,  4.07it/s, loss=5.46, v_num=0]
Epoch 11: 100%|██████████| 22/22 [00:05<00:00,  4.02it/s, loss=5.46, v_num=0]
Epoch 11: 100%|██████████| 22/22 [00:05<00:00,  4.02it/s, loss=5.46, v_num=0]
Epoch 11:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.46, v_num=0]
Epoch 12:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.46, v_num=0]
Epoch 12:   5%|▍         | 1/22 [00:02<00:42,  2.02s/it, loss=5.46, v_num=0]
Epoch 12:   5%|▍         | 1/22 [00:02<00:43,  2.08s/it, loss=5.46, v_num=0]
Epoch 12:   9%|▉         | 2/22 [00:02<00:21,  1.09s/it, loss=5.46, v_num=0]
Epoch 12:   9%|▉         | 2/22 [00:02<00:22,  1.12s/it, loss=5.46, v_num=0]
Epoch 12:  14%|█▎        | 3/22 [00:02<00:14,  1.29it/s, loss=5.46, v_num=0]
Epoch 12:  14%|█▎        | 3/22 [00:02<00:15,  1.26it/s, loss=5.45, v_num=0]
Epoch 12:  18%|█▊        | 4/22 [00:02<00:11,  1.61it/s, loss=5.45, v_num=0]
Epoch 12:  18%|█▊        | 4/22 [00:02<00:11,  1.57it/s, loss=5.45, v_num=0]
Epoch 12:  23%|██▎       | 5/22 [00:02<00:08,  1.89it/s, loss=5.45, v_num=0]
Epoch 12:  23%|██▎       | 5/22 [00:02<00:09,  1.85it/s, loss=5.45, v_num=0]
Epoch 12:  27%|██▋       | 6/22 [00:02<00:07,  2.15it/s, loss=5.45, v_num=0]
Epoch 12:  27%|██▋       | 6/22 [00:02<00:07,  2.10it/s, loss=5.45, v_num=0]
Epoch 12:  32%|███▏      | 7/22 [00:02<00:06,  2.38it/s, loss=5.45, v_num=0]
Epoch 12:  32%|███▏      | 7/22 [00:03<00:06,  2.33it/s, loss=5.45, v_num=0]
Epoch 12:  36%|███▋      | 8/22 [00:03<00:05,  2.58it/s, loss=5.45, v_num=0]
Epoch 12:  36%|███▋      | 8/22 [00:03<00:05,  2.53it/s, loss=5.45, v_num=0]
Epoch 12:  41%|████      | 9/22 [00:03<00:04,  2.74it/s, loss=5.45, v_num=0]
Epoch 12:  41%|████      | 9/22 [00:03<00:04,  2.69it/s, loss=5.44, v_num=0]
Epoch 12:  45%|████▌     | 10/22 [00:03<00:04,  2.91it/s, loss=5.44, v_num=0]
Epoch 12:  45%|████▌     | 10/22 [00:03<00:04,  2.85it/s, loss=5.45, v_num=0]
Epoch 12:  50%|█████     | 11/22 [00:03<00:03,  3.06it/s, loss=5.45, v_num=0]
Epoch 12:  50%|█████     | 11/22 [00:03<00:03,  3.01it/s, loss=5.44, v_num=0]
Epoch 12:  55%|█████▍    | 12/22 [00:03<00:03,  3.20it/s, loss=5.44, v_num=0]
Epoch 12:  55%|█████▍    | 12/22 [00:03<00:03,  3.15it/s, loss=5.44, v_num=0]
Epoch 12:  59%|█████▉    | 13/22 [00:03<00:02,  3.33it/s, loss=5.44, v_num=0]
Epoch 12:  59%|█████▉    | 13/22 [00:03<00:02,  3.28it/s, loss=5.44, v_num=0]
Epoch 12:  64%|██████▎   | 14/22 [00:04<00:02,  3.45it/s, loss=5.44, v_num=0]
Epoch 12:  64%|██████▎   | 14/22 [00:04<00:02,  3.40it/s, loss=5.44, v_num=0]
Epoch 12:  68%|██████▊   | 15/22 [00:04<00:01,  3.57it/s, loss=5.44, v_num=0]
Epoch 12:  68%|██████▊   | 15/22 [00:04<00:01,  3.52it/s, loss=5.44, v_num=0]
Epoch 12:  73%|███████▎  | 16/22 [00:04<00:01,  3.67it/s, loss=5.44, v_num=0]
Epoch 12:  73%|███████▎  | 16/22 [00:04<00:01,  3.62it/s, loss=5.43, v_num=0]
Epoch 12:  77%|███████▋  | 17/22 [00:04<00:01,  3.75it/s, loss=5.43, v_num=0]
Epoch 12:  77%|███████▋  | 17/22 [00:04<00:01,  3.70it/s, loss=5.43, v_num=0]
Epoch 12:  82%|████████▏ | 18/22 [00:04<00:01,  3.85it/s, loss=5.43, v_num=0]
Epoch 12:  82%|████████▏ | 18/22 [00:04<00:01,  3.80it/s, loss=5.43, v_num=0]
Epoch 12:  86%|████████▋ | 19/22 [00:04<00:00,  3.93it/s, loss=5.43, v_num=0]
Epoch 12:  86%|████████▋ | 19/22 [00:04<00:00,  3.88it/s, loss=5.43, v_num=0]
Epoch 12:  91%|█████████ | 20/22 [00:04<00:00,  4.01it/s, loss=5.43, v_num=0]
Epoch 12:  91%|█████████ | 20/22 [00:05<00:00,  3.96it/s, loss=5.43, v_num=0]
Epoch 12:  95%|█████████▌| 21/22 [00:05<00:00,  4.09it/s, loss=5.43, v_num=0]
Epoch 12:  95%|█████████▌| 21/22 [00:05<00:00,  4.04it/s, loss=5.43, v_num=0]
Epoch 12: 100%|██████████| 22/22 [00:05<00:00,  4.16it/s, loss=5.43, v_num=0]
Epoch 12: 100%|██████████| 22/22 [00:05<00:00,  4.11it/s, loss=5.44, v_num=0]
Epoch 12: 100%|██████████| 22/22 [00:05<00:00,  4.11it/s, loss=5.44, v_num=0]
Epoch 12:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.44, v_num=0]
Epoch 13:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.44, v_num=0]
Epoch 13:   5%|▍         | 1/22 [00:02<00:43,  2.07s/it, loss=5.44, v_num=0]
Epoch 13:   5%|▍         | 1/22 [00:02<00:44,  2.12s/it, loss=5.44, v_num=0]
Epoch 13:   9%|▉         | 2/22 [00:02<00:22,  1.11s/it, loss=5.44, v_num=0]
Epoch 13:   9%|▉         | 2/22 [00:02<00:22,  1.14s/it, loss=5.44, v_num=0]
Epoch 13:  14%|█▎        | 3/22 [00:02<00:15,  1.26it/s, loss=5.44, v_num=0]
Epoch 13:  14%|█▎        | 3/22 [00:02<00:15,  1.23it/s, loss=5.44, v_num=0]
Epoch 13:  18%|█▊        | 4/22 [00:02<00:11,  1.58it/s, loss=5.44, v_num=0]
Epoch 13:  18%|█▊        | 4/22 [00:02<00:11,  1.54it/s, loss=5.44, v_num=0]
Epoch 13:  23%|██▎       | 5/22 [00:02<00:09,  1.86it/s, loss=5.44, v_num=0]
Epoch 13:  23%|██▎       | 5/22 [00:02<00:09,  1.82it/s, loss=5.44, v_num=0]
Epoch 13:  27%|██▋       | 6/22 [00:02<00:07,  2.11it/s, loss=5.44, v_num=0]
Epoch 13:  27%|██▋       | 6/22 [00:02<00:07,  2.07it/s, loss=5.45, v_num=0]
Epoch 13:  32%|███▏      | 7/22 [00:02<00:06,  2.34it/s, loss=5.45, v_num=0]
Epoch 13:  32%|███▏      | 7/22 [00:03<00:06,  2.29it/s, loss=5.45, v_num=0]
Epoch 13:  36%|███▋      | 8/22 [00:03<00:05,  2.54it/s, loss=5.45, v_num=0]
Epoch 13:  36%|███▋      | 8/22 [00:03<00:05,  2.49it/s, loss=5.44, v_num=0]
Epoch 13:  41%|████      | 9/22 [00:03<00:04,  2.72it/s, loss=5.44, v_num=0]
Epoch 13:  41%|████      | 9/22 [00:03<00:04,  2.67it/s, loss=5.45, v_num=0]
Epoch 13:  45%|████▌     | 10/22 [00:03<00:04,  2.88it/s, loss=5.45, v_num=0]
Epoch 13:  45%|████▌     | 10/22 [00:03<00:04,  2.83it/s, loss=5.44, v_num=0]
Epoch 13:  50%|█████     | 11/22 [00:03<00:03,  3.04it/s, loss=5.44, v_num=0]
Epoch 13:  50%|█████     | 11/22 [00:03<00:03,  2.99it/s, loss=5.44, v_num=0]
Epoch 13:  55%|█████▍    | 12/22 [00:03<00:03,  3.18it/s, loss=5.44, v_num=0]
Epoch 13:  55%|█████▍    | 12/22 [00:03<00:03,  3.13it/s, loss=5.43, v_num=0]
Epoch 13:  59%|█████▉    | 13/22 [00:03<00:02,  3.31it/s, loss=5.43, v_num=0]
Epoch 13:  59%|█████▉    | 13/22 [00:03<00:02,  3.26it/s, loss=5.43, v_num=0]
Epoch 13:  64%|██████▎   | 14/22 [00:04<00:02,  3.43it/s, loss=5.43, v_num=0]
Epoch 13:  64%|██████▎   | 14/22 [00:04<00:02,  3.38it/s, loss=5.43, v_num=0]
Epoch 13:  68%|██████▊   | 15/22 [00:04<00:01,  3.55it/s, loss=5.43, v_num=0]
Epoch 13:  68%|██████▊   | 15/22 [00:04<00:02,  3.50it/s, loss=5.42, v_num=0]
Epoch 13:  73%|███████▎  | 16/22 [00:04<00:01,  3.65it/s, loss=5.42, v_num=0]
Epoch 13:  73%|███████▎  | 16/22 [00:04<00:01,  3.60it/s, loss=5.42, v_num=0]
Epoch 13:  77%|███████▋  | 17/22 [00:04<00:01,  3.73it/s, loss=5.42, v_num=0]
Epoch 13:  77%|███████▋  | 17/22 [00:04<00:01,  3.68it/s, loss=5.42, v_num=0]
Epoch 13:  82%|████████▏ | 18/22 [00:04<00:01,  3.83it/s, loss=5.42, v_num=0]
Epoch 13:  82%|████████▏ | 18/22 [00:04<00:01,  3.77it/s, loss=5.42, v_num=0]
Epoch 13:  86%|████████▋ | 19/22 [00:04<00:00,  3.91it/s, loss=5.42, v_num=0]
Epoch 13:  86%|████████▋ | 19/22 [00:04<00:00,  3.86it/s, loss=5.42, v_num=0]
Epoch 13:  91%|█████████ | 20/22 [00:05<00:00,  3.99it/s, loss=5.42, v_num=0]
Epoch 13:  91%|█████████ | 20/22 [00:05<00:00,  3.94it/s, loss=5.41, v_num=0]
Epoch 13:  95%|█████████▌| 21/22 [00:05<00:00,  4.06it/s, loss=5.41, v_num=0]
Epoch 13:  95%|█████████▌| 21/22 [00:05<00:00,  4.02it/s, loss=5.41, v_num=0]
Epoch 13: 100%|██████████| 22/22 [00:05<00:00,  4.14it/s, loss=5.41, v_num=0]
Epoch 13: 100%|██████████| 22/22 [00:05<00:00,  4.09it/s, loss=5.41, v_num=0]
Epoch 13: 100%|██████████| 22/22 [00:05<00:00,  4.09it/s, loss=5.41, v_num=0]
Epoch 13:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.41, v_num=0]
Epoch 14:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.41, v_num=0]
Epoch 14:   5%|▍         | 1/22 [00:02<00:42,  2.04s/it, loss=5.41, v_num=0]
Epoch 14:   5%|▍         | 1/22 [00:02<00:44,  2.10s/it, loss=5.4, v_num=0]
Epoch 14:   9%|▉         | 2/22 [00:02<00:21,  1.10s/it, loss=5.4, v_num=0]
Epoch 14:   9%|▉         | 2/22 [00:02<00:22,  1.13s/it, loss=5.4, v_num=0]
Epoch 14:  14%|█▎        | 3/22 [00:02<00:14,  1.28it/s, loss=5.4, v_num=0]
Epoch 14:  14%|█▎        | 3/22 [00:02<00:15,  1.24it/s, loss=5.4, v_num=0]
Epoch 14:  18%|█▊        | 4/22 [00:02<00:11,  1.59it/s, loss=5.4, v_num=0]
Epoch 14:  18%|█▊        | 4/22 [00:02<00:11,  1.56it/s, loss=5.39, v_num=0]
Epoch 14:  23%|██▎       | 5/22 [00:02<00:09,  1.88it/s, loss=5.39, v_num=0]
Epoch 14:  23%|██▎       | 5/22 [00:02<00:09,  1.84it/s, loss=5.39, v_num=0]
Epoch 14:  27%|██▋       | 6/22 [00:02<00:07,  2.13it/s, loss=5.39, v_num=0]
Epoch 14:  27%|██▋       | 6/22 [00:02<00:07,  2.09it/s, loss=5.4, v_num=0]
Epoch 14:  32%|███▏      | 7/22 [00:02<00:06,  2.36it/s, loss=5.4, v_num=0]
Epoch 14:  32%|███▏      | 7/22 [00:03<00:06,  2.31it/s, loss=5.39, v_num=0]
Epoch 14:  36%|███▋      | 8/22 [00:03<00:05,  2.56it/s, loss=5.39, v_num=0]
Epoch 14:  36%|███▋      | 8/22 [00:03<00:05,  2.52it/s, loss=5.4, v_num=0]
Epoch 14:  41%|████      | 9/22 [00:03<00:04,  2.73it/s, loss=5.4, v_num=0]
Epoch 14:  41%|████      | 9/22 [00:03<00:04,  2.68it/s, loss=5.4, v_num=0]
Epoch 14:  45%|████▌     | 10/22 [00:03<00:04,  2.89it/s, loss=5.4, v_num=0]
Epoch 14:  45%|████▌     | 10/22 [00:03<00:04,  2.85it/s, loss=5.41, v_num=0]
Epoch 14:  50%|█████     | 11/22 [00:03<00:03,  3.05it/s, loss=5.41, v_num=0]
Epoch 14:  50%|█████     | 11/22 [00:03<00:03,  3.00it/s, loss=5.41, v_num=0]
Epoch 14:  55%|█████▍    | 12/22 [00:03<00:03,  3.19it/s, loss=5.41, v_num=0]
Epoch 14:  55%|█████▍    | 12/22 [00:03<00:03,  3.14it/s, loss=5.41, v_num=0]
Epoch 14:  59%|█████▉    | 13/22 [00:03<00:02,  3.32it/s, loss=5.41, v_num=0]
Epoch 14:  59%|█████▉    | 13/22 [00:03<00:02,  3.27it/s, loss=5.41, v_num=0]
Epoch 14:  64%|██████▎   | 14/22 [00:04<00:02,  3.44it/s, loss=5.41, v_num=0]
Epoch 14:  64%|██████▎   | 14/22 [00:04<00:02,  3.39it/s, loss=5.41, v_num=0]
Epoch 14:  68%|██████▊   | 15/22 [00:04<00:01,  3.56it/s, loss=5.41, v_num=0]
Epoch 14:  68%|██████▊   | 15/22 [00:04<00:01,  3.51it/s, loss=5.4, v_num=0]
Epoch 14:  73%|███████▎  | 16/22 [00:04<00:01,  3.66it/s, loss=5.4, v_num=0]
Epoch 14:  73%|███████▎  | 16/22 [00:04<00:01,  3.61it/s, loss=5.4, v_num=0]
Epoch 14:  77%|███████▋  | 17/22 [00:04<00:01,  3.71it/s, loss=5.4, v_num=0]
Epoch 14:  77%|███████▋  | 17/22 [00:04<00:01,  3.66it/s, loss=5.4, v_num=0]
Epoch 14:  82%|████████▏ | 18/22 [00:04<00:01,  3.80it/s, loss=5.4, v_num=0]
Epoch 14:  82%|████████▏ | 18/22 [00:04<00:01,  3.75it/s, loss=5.4, v_num=0]
Epoch 14:  86%|████████▋ | 19/22 [00:04<00:00,  3.89it/s, loss=5.4, v_num=0]
Epoch 14:  86%|████████▋ | 19/22 [00:04<00:00,  3.84it/s, loss=5.4, v_num=0]
Epoch 14:  91%|█████████ | 20/22 [00:05<00:00,  3.97it/s, loss=5.4, v_num=0]
Epoch 14:  91%|█████████ | 20/22 [00:05<00:00,  3.92it/s, loss=5.4, v_num=0]
Epoch 14:  95%|█████████▌| 21/22 [00:05<00:00,  4.04it/s, loss=5.4, v_num=0]
Epoch 14:  95%|█████████▌| 21/22 [00:05<00:00,  4.00it/s, loss=5.4, v_num=0]
Epoch 14: 100%|██████████| 22/22 [00:05<00:00,  4.12it/s, loss=5.4, v_num=0]
Epoch 14: 100%|██████████| 22/22 [00:05<00:00,  4.07it/s, loss=5.4, v_num=0]
Epoch 14: 100%|██████████| 22/22 [00:05<00:00,  4.07it/s, loss=5.4, v_num=0]
Epoch 14:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.4, v_num=0]
Epoch 15:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.4, v_num=0]
Epoch 15:   5%|▍         | 1/22 [00:02<00:43,  2.06s/it, loss=5.4, v_num=0]
Epoch 15:   5%|▍         | 1/22 [00:02<00:44,  2.11s/it, loss=5.4, v_num=0]
Epoch 15:   9%|▉         | 2/22 [00:02<00:22,  1.11s/it, loss=5.4, v_num=0]
Epoch 15:   9%|▉         | 2/22 [00:02<00:22,  1.14s/it, loss=5.4, v_num=0]
Epoch 15:  14%|█▎        | 3/22 [00:02<00:14,  1.27it/s, loss=5.4, v_num=0]
Epoch 15:  14%|█▎        | 3/22 [00:02<00:15,  1.24it/s, loss=5.41, v_num=0]
Epoch 15:  18%|█▊        | 4/22 [00:02<00:11,  1.59it/s, loss=5.41, v_num=0]
Epoch 15:  18%|█▊        | 4/22 [00:02<00:11,  1.55it/s, loss=5.4, v_num=0]
Epoch 15:  23%|██▎       | 5/22 [00:02<00:09,  1.87it/s, loss=5.4, v_num=0]
Epoch 15:  23%|██▎       | 5/22 [00:02<00:09,  1.83it/s, loss=5.4, v_num=0]
Epoch 15:  27%|██▋       | 6/22 [00:02<00:07,  2.13it/s, loss=5.4, v_num=0]
Epoch 15:  27%|██▋       | 6/22 [00:02<00:07,  2.08it/s, loss=5.4, v_num=0]
Epoch 15:  32%|███▏      | 7/22 [00:02<00:06,  2.35it/s, loss=5.4, v_num=0]
Epoch 15:  32%|███▏      | 7/22 [00:03<00:06,  2.30it/s, loss=5.4, v_num=0]
Epoch 15:  36%|███▋      | 8/22 [00:03<00:05,  2.56it/s, loss=5.4, v_num=0]
Epoch 15:  36%|███▋      | 8/22 [00:03<00:05,  2.51it/s, loss=5.39, v_num=0]
Epoch 15:  41%|████      | 9/22 [00:03<00:04,  2.71it/s, loss=5.39, v_num=0]
Epoch 15:  41%|████      | 9/22 [00:03<00:04,  2.66it/s, loss=5.39, v_num=0]
Epoch 15:  45%|████▌     | 10/22 [00:03<00:04,  2.87it/s, loss=5.39, v_num=0]
Epoch 15:  45%|████▌     | 10/22 [00:03<00:04,  2.82it/s, loss=5.4, v_num=0]
Epoch 15:  50%|█████     | 11/22 [00:03<00:03,  3.03it/s, loss=5.4, v_num=0]
Epoch 15:  50%|█████     | 11/22 [00:03<00:03,  2.97it/s, loss=5.4, v_num=0]
Epoch 15:  55%|█████▍    | 12/22 [00:03<00:03,  3.17it/s, loss=5.4, v_num=0]
Epoch 15:  55%|█████▍    | 12/22 [00:03<00:03,  3.12it/s, loss=5.39, v_num=0]
Epoch 15:  59%|█████▉    | 13/22 [00:03<00:02,  3.30it/s, loss=5.39, v_num=0]
Epoch 15:  59%|█████▉    | 13/22 [00:04<00:02,  3.25it/s, loss=5.39, v_num=0]
Epoch 15:  64%|██████▎   | 14/22 [00:04<00:02,  3.42it/s, loss=5.39, v_num=0]
Epoch 15:  64%|██████▎   | 14/22 [00:04<00:02,  3.37it/s, loss=5.39, v_num=0]
Epoch 15:  68%|██████▊   | 15/22 [00:04<00:01,  3.54it/s, loss=5.39, v_num=0]
Epoch 15:  68%|██████▊   | 15/22 [00:04<00:02,  3.48it/s, loss=5.39, v_num=0]
Epoch 15:  73%|███████▎  | 16/22 [00:04<00:01,  3.64it/s, loss=5.39, v_num=0]
Epoch 15:  73%|███████▎  | 16/22 [00:04<00:01,  3.59it/s, loss=5.39, v_num=0]
Epoch 15:  77%|███████▋  | 17/22 [00:04<00:01,  3.71it/s, loss=5.39, v_num=0]
Epoch 15:  77%|███████▋  | 17/22 [00:04<00:01,  3.66it/s, loss=5.39, v_num=0]
Epoch 15:  82%|████████▏ | 18/22 [00:04<00:01,  3.80it/s, loss=5.39, v_num=0]
Epoch 15:  82%|████████▏ | 18/22 [00:04<00:01,  3.75it/s, loss=5.39, v_num=0]
Epoch 15:  86%|████████▋ | 19/22 [00:04<00:00,  3.89it/s, loss=5.39, v_num=0]
Epoch 15:  86%|████████▋ | 19/22 [00:04<00:00,  3.84it/s, loss=5.39, v_num=0]
Epoch 15:  91%|█████████ | 20/22 [00:05<00:00,  3.97it/s, loss=5.39, v_num=0]
Epoch 15:  91%|█████████ | 20/22 [00:05<00:00,  3.92it/s, loss=5.38, v_num=0]
Epoch 15:  95%|█████████▌| 21/22 [00:05<00:00,  4.04it/s, loss=5.38, v_num=0]
Epoch 15:  95%|█████████▌| 21/22 [00:05<00:00,  3.99it/s, loss=5.38, v_num=0]
Epoch 15: 100%|██████████| 22/22 [00:05<00:00,  4.12it/s, loss=5.38, v_num=0]
Epoch 15: 100%|██████████| 22/22 [00:05<00:00,  4.07it/s, loss=5.38, v_num=0]
Epoch 15: 100%|██████████| 22/22 [00:05<00:00,  4.07it/s, loss=5.38, v_num=0]
Epoch 15:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.38, v_num=0]
Epoch 16:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.38, v_num=0]
Epoch 16:   5%|▍         | 1/22 [00:02<00:44,  2.11s/it, loss=5.38, v_num=0]
Epoch 16:   5%|▍         | 1/22 [00:02<00:45,  2.17s/it, loss=5.37, v_num=0]
Epoch 16:   9%|▉         | 2/22 [00:02<00:22,  1.13s/it, loss=5.37, v_num=0]
Epoch 16:   9%|▉         | 2/22 [00:02<00:23,  1.16s/it, loss=5.37, v_num=0]
Epoch 16:  14%|█▎        | 3/22 [00:02<00:15,  1.24it/s, loss=5.37, v_num=0]
Epoch 16:  14%|█▎        | 3/22 [00:02<00:15,  1.21it/s, loss=5.37, v_num=0]
Epoch 16:  18%|█▊        | 4/22 [00:02<00:11,  1.56it/s, loss=5.37, v_num=0]
Epoch 16:  18%|█▊        | 4/22 [00:02<00:11,  1.52it/s, loss=5.37, v_num=0]
Epoch 16:  23%|██▎       | 5/22 [00:02<00:09,  1.84it/s, loss=5.37, v_num=0]
Epoch 16:  23%|██▎       | 5/22 [00:02<00:09,  1.80it/s, loss=5.36, v_num=0]
Epoch 16:  27%|██▋       | 6/22 [00:02<00:07,  2.09it/s, loss=5.36, v_num=0]
Epoch 16:  27%|██▋       | 6/22 [00:02<00:07,  2.04it/s, loss=5.36, v_num=0]
Epoch 16:  32%|███▏      | 7/22 [00:03<00:06,  2.31it/s, loss=5.36, v_num=0]
Epoch 16:  32%|███▏      | 7/22 [00:03<00:06,  2.27it/s, loss=5.36, v_num=0]
Epoch 16:  36%|███▋      | 8/22 [00:03<00:05,  2.51it/s, loss=5.36, v_num=0]
Epoch 16:  36%|███▋      | 8/22 [00:03<00:05,  2.47it/s, loss=5.36, v_num=0]
Epoch 16:  41%|████      | 9/22 [00:03<00:04,  2.67it/s, loss=5.36, v_num=0]
Epoch 16:  41%|████      | 9/22 [00:03<00:04,  2.62it/s, loss=5.35, v_num=0]
Epoch 16:  45%|████▌     | 10/22 [00:03<00:04,  2.83it/s, loss=5.35, v_num=0]
Epoch 16:  45%|████▌     | 10/22 [00:03<00:04,  2.78it/s, loss=5.35, v_num=0]
Epoch 16:  50%|█████     | 11/22 [00:03<00:03,  2.98it/s, loss=5.35, v_num=0]
Epoch 16:  50%|█████     | 11/22 [00:03<00:03,  2.94it/s, loss=5.35, v_num=0]
Epoch 16:  55%|█████▍    | 12/22 [00:03<00:03,  3.13it/s, loss=5.35, v_num=0]
Epoch 16:  55%|█████▍    | 12/22 [00:03<00:03,  3.08it/s, loss=5.35, v_num=0]
Epoch 16:  59%|█████▉    | 13/22 [00:03<00:02,  3.26it/s, loss=5.35, v_num=0]
Epoch 16:  59%|█████▉    | 13/22 [00:04<00:02,  3.21it/s, loss=5.35, v_num=0]
Epoch 16:  64%|██████▎   | 14/22 [00:04<00:02,  3.38it/s, loss=5.35, v_num=0]
Epoch 16:  64%|██████▎   | 14/22 [00:04<00:02,  3.33it/s, loss=5.35, v_num=0]
Epoch 16:  68%|██████▊   | 15/22 [00:04<00:02,  3.49it/s, loss=5.35, v_num=0]
Epoch 16:  68%|██████▊   | 15/22 [00:04<00:02,  3.44it/s, loss=5.34, v_num=0]
Epoch 16:  73%|███████▎  | 16/22 [00:04<00:01,  3.60it/s, loss=5.34, v_num=0]
Epoch 16:  73%|███████▎  | 16/22 [00:04<00:01,  3.55it/s, loss=5.34, v_num=0]
Epoch 16:  77%|███████▋  | 17/22 [00:04<00:01,  3.67it/s, loss=5.34, v_num=0]
Epoch 16:  77%|███████▋  | 17/22 [00:04<00:01,  3.62it/s, loss=5.34, v_num=0]
Epoch 16:  82%|████████▏ | 18/22 [00:04<00:01,  3.76it/s, loss=5.34, v_num=0]
Epoch 16:  82%|████████▏ | 18/22 [00:04<00:01,  3.72it/s, loss=5.34, v_num=0]
Epoch 16:  86%|████████▋ | 19/22 [00:04<00:00,  3.85it/s, loss=5.34, v_num=0]
Epoch 16:  86%|████████▋ | 19/22 [00:04<00:00,  3.80it/s, loss=5.35, v_num=0]
Epoch 16:  91%|█████████ | 20/22 [00:05<00:00,  3.93it/s, loss=5.35, v_num=0]
Epoch 16:  91%|█████████ | 20/22 [00:05<00:00,  3.88it/s, loss=5.35, v_num=0]
Epoch 16:  95%|█████████▌| 21/22 [00:05<00:00,  4.01it/s, loss=5.35, v_num=0]
Epoch 16:  95%|█████████▌| 21/22 [00:05<00:00,  3.96it/s, loss=5.35, v_num=0]
Epoch 16: 100%|██████████| 22/22 [00:05<00:00,  4.08it/s, loss=5.35, v_num=0]
Epoch 16: 100%|██████████| 22/22 [00:05<00:00,  4.03it/s, loss=5.35, v_num=0]
Epoch 16: 100%|██████████| 22/22 [00:05<00:00,  4.03it/s, loss=5.35, v_num=0]
Epoch 16:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.35, v_num=0]
Epoch 17:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.35, v_num=0]
Epoch 17:   5%|▍         | 1/22 [00:02<00:42,  2.04s/it, loss=5.35, v_num=0]
Epoch 17:   5%|▍         | 1/22 [00:02<00:43,  2.09s/it, loss=5.35, v_num=0]
Epoch 17:   9%|▉         | 2/22 [00:02<00:21,  1.10s/it, loss=5.35, v_num=0]
Epoch 17:   9%|▉         | 2/22 [00:02<00:22,  1.13s/it, loss=5.35, v_num=0]
Epoch 17:  14%|█▎        | 3/22 [00:02<00:14,  1.28it/s, loss=5.35, v_num=0]
Epoch 17:  14%|█▎        | 3/22 [00:02<00:15,  1.25it/s, loss=5.36, v_num=0]
Epoch 17:  18%|█▊        | 4/22 [00:02<00:11,  1.60it/s, loss=5.36, v_num=0]
Epoch 17:  18%|█▊        | 4/22 [00:02<00:11,  1.56it/s, loss=5.35, v_num=0]
Epoch 17:  23%|██▎       | 5/22 [00:02<00:09,  1.89it/s, loss=5.35, v_num=0]
Epoch 17:  23%|██▎       | 5/22 [00:02<00:09,  1.85it/s, loss=5.36, v_num=0]
Epoch 17:  27%|██▋       | 6/22 [00:02<00:07,  2.14it/s, loss=5.36, v_num=0]
Epoch 17:  27%|██▋       | 6/22 [00:02<00:07,  2.10it/s, loss=5.36, v_num=0]
Epoch 17:  32%|███▏      | 7/22 [00:02<00:06,  2.37it/s, loss=5.36, v_num=0]
Epoch 17:  32%|███▏      | 7/22 [00:03<00:06,  2.32it/s, loss=5.37, v_num=0]
Epoch 17:  36%|███▋      | 8/22 [00:03<00:05,  2.57it/s, loss=5.37, v_num=0]
Epoch 17:  36%|███▋      | 8/22 [00:03<00:05,  2.52it/s, loss=5.37, v_num=0]
Epoch 17:  41%|████      | 9/22 [00:03<00:04,  2.75it/s, loss=5.37, v_num=0]
Epoch 17:  41%|████      | 9/22 [00:03<00:04,  2.70it/s, loss=5.37, v_num=0]
Epoch 17:  45%|████▌     | 10/22 [00:03<00:04,  2.91it/s, loss=5.37, v_num=0]
Epoch 17:  45%|████▌     | 10/22 [00:03<00:04,  2.86it/s, loss=5.37, v_num=0]
Epoch 17:  50%|█████     | 11/22 [00:03<00:03,  3.07it/s, loss=5.37, v_num=0]
Epoch 17:  50%|█████     | 11/22 [00:03<00:03,  3.02it/s, loss=5.37, v_num=0]
Epoch 17:  55%|█████▍    | 12/22 [00:03<00:03,  3.21it/s, loss=5.37, v_num=0]
Epoch 17:  55%|█████▍    | 12/22 [00:03<00:03,  3.16it/s, loss=5.37, v_num=0]
Epoch 17:  59%|█████▉    | 13/22 [00:03<00:02,  3.35it/s, loss=5.37, v_num=0]
Epoch 17:  59%|█████▉    | 13/22 [00:03<00:02,  3.29it/s, loss=5.37, v_num=0]
Epoch 17:  64%|██████▎   | 14/22 [00:04<00:02,  3.47it/s, loss=5.37, v_num=0]
Epoch 17:  64%|██████▎   | 14/22 [00:04<00:02,  3.41it/s, loss=5.38, v_num=0]
Epoch 17:  68%|██████▊   | 15/22 [00:04<00:01,  3.58it/s, loss=5.38, v_num=0]
Epoch 17:  68%|██████▊   | 15/22 [00:04<00:01,  3.53it/s, loss=5.37, v_num=0]
Epoch 17:  73%|███████▎  | 16/22 [00:04<00:01,  3.69it/s, loss=5.37, v_num=0]
Epoch 17:  73%|███████▎  | 16/22 [00:04<00:01,  3.63it/s, loss=5.37, v_num=0]
Epoch 17:  77%|███████▋  | 17/22 [00:04<00:01,  3.74it/s, loss=5.37, v_num=0]
Epoch 17:  77%|███████▋  | 17/22 [00:04<00:01,  3.69it/s, loss=5.37, v_num=0]
Epoch 17:  82%|████████▏ | 18/22 [00:04<00:01,  3.83it/s, loss=5.37, v_num=0]
Epoch 17:  82%|████████▏ | 18/22 [00:04<00:01,  3.78it/s, loss=5.37, v_num=0]
Epoch 17:  86%|████████▋ | 19/22 [00:04<00:00,  3.92it/s, loss=5.37, v_num=0]
Epoch 17:  86%|████████▋ | 19/22 [00:04<00:00,  3.87it/s, loss=5.37, v_num=0]
Epoch 17:  91%|█████████ | 20/22 [00:05<00:00,  4.00it/s, loss=5.37, v_num=0]
Epoch 17:  91%|█████████ | 20/22 [00:05<00:00,  3.95it/s, loss=5.36, v_num=0]
Epoch 17:  95%|█████████▌| 21/22 [00:05<00:00,  4.07it/s, loss=5.36, v_num=0]
Epoch 17:  95%|█████████▌| 21/22 [00:05<00:00,  4.03it/s, loss=5.37, v_num=0]
Epoch 17: 100%|██████████| 22/22 [00:05<00:00,  4.14it/s, loss=5.37, v_num=0]
Epoch 17: 100%|██████████| 22/22 [00:05<00:00,  4.10it/s, loss=5.37, v_num=0]
Epoch 17: 100%|██████████| 22/22 [00:05<00:00,  4.10it/s, loss=5.37, v_num=0]
Epoch 17:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.37, v_num=0]
Epoch 18:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.37, v_num=0]
Epoch 18:   5%|▍         | 1/22 [00:02<00:42,  2.04s/it, loss=5.37, v_num=0]
Epoch 18:   5%|▍         | 1/22 [00:02<00:44,  2.10s/it, loss=5.37, v_num=0]
Epoch 18:   9%|▉         | 2/22 [00:02<00:21,  1.10s/it, loss=5.37, v_num=0]
Epoch 18:   9%|▉         | 2/22 [00:02<00:22,  1.13s/it, loss=5.37, v_num=0]
Epoch 18:  14%|█▎        | 3/22 [00:02<00:14,  1.28it/s, loss=5.37, v_num=0]
Epoch 18:  14%|█▎        | 3/22 [00:02<00:15,  1.24it/s, loss=5.37, v_num=0]
Epoch 18:  18%|█▊        | 4/22 [00:02<00:11,  1.59it/s, loss=5.37, v_num=0]
Epoch 18:  18%|█▊        | 4/22 [00:02<00:11,  1.56it/s, loss=5.36, v_num=0]
Epoch 18:  23%|██▎       | 5/22 [00:02<00:09,  1.88it/s, loss=5.36, v_num=0]
Epoch 18:  23%|██▎       | 5/22 [00:02<00:09,  1.84it/s, loss=5.36, v_num=0]
Epoch 18:  27%|██▋       | 6/22 [00:02<00:07,  2.13it/s, loss=5.36, v_num=0]
Epoch 18:  27%|██▋       | 6/22 [00:02<00:07,  2.08it/s, loss=5.36, v_num=0]
Epoch 18:  32%|███▏      | 7/22 [00:02<00:06,  2.36it/s, loss=5.36, v_num=0]
Epoch 18:  32%|███▏      | 7/22 [00:03<00:06,  2.31it/s, loss=5.36, v_num=0]
Epoch 18:  36%|███▋      | 8/22 [00:03<00:05,  2.56it/s, loss=5.36, v_num=0]
Epoch 18:  36%|███▋      | 8/22 [00:03<00:05,  2.51it/s, loss=5.36, v_num=0]
Epoch 18:  41%|████      | 9/22 [00:03<00:04,  2.70it/s, loss=5.36, v_num=0]
Epoch 18:  41%|████      | 9/22 [00:03<00:04,  2.66it/s, loss=5.36, v_num=0]
Epoch 18:  45%|████▌     | 10/22 [00:03<00:04,  2.87it/s, loss=5.36, v_num=0]
Epoch 18:  45%|████▌     | 10/22 [00:03<00:04,  2.82it/s, loss=5.36, v_num=0]
Epoch 18:  50%|█████     | 11/22 [00:03<00:03,  3.03it/s, loss=5.36, v_num=0]
Epoch 18:  50%|█████     | 11/22 [00:03<00:03,  2.98it/s, loss=5.36, v_num=0]
Epoch 18:  55%|█████▍    | 12/22 [00:03<00:03,  3.17it/s, loss=5.36, v_num=0]
Epoch 18:  55%|█████▍    | 12/22 [00:03<00:03,  3.12it/s, loss=5.36, v_num=0]
Epoch 18:  59%|█████▉    | 13/22 [00:03<00:02,  3.30it/s, loss=5.36, v_num=0]
Epoch 18:  59%|█████▉    | 13/22 [00:04<00:02,  3.25it/s, loss=5.37, v_num=0]
Epoch 18:  64%|██████▎   | 14/22 [00:04<00:02,  3.42it/s, loss=5.37, v_num=0]
Epoch 18:  64%|██████▎   | 14/22 [00:04<00:02,  3.37it/s, loss=5.37, v_num=0]
Epoch 18:  68%|██████▊   | 15/22 [00:04<00:01,  3.53it/s, loss=5.37, v_num=0]
Epoch 18:  68%|██████▊   | 15/22 [00:04<00:02,  3.48it/s, loss=5.36, v_num=0]
Epoch 18:  73%|███████▎  | 16/22 [00:04<00:01,  3.63it/s, loss=5.36, v_num=0]
Epoch 18:  73%|███████▎  | 16/22 [00:04<00:01,  3.58it/s, loss=5.36, v_num=0]
Epoch 18:  77%|███████▋  | 17/22 [00:04<00:01,  3.72it/s, loss=5.36, v_num=0]
Epoch 18:  77%|███████▋  | 17/22 [00:04<00:01,  3.67it/s, loss=5.36, v_num=0]
Epoch 18:  82%|████████▏ | 18/22 [00:04<00:01,  3.81it/s, loss=5.36, v_num=0]
Epoch 18:  82%|████████▏ | 18/22 [00:04<00:01,  3.76it/s, loss=5.36, v_num=0]
Epoch 18:  86%|████████▋ | 19/22 [00:04<00:00,  3.90it/s, loss=5.36, v_num=0]
Epoch 18:  86%|████████▋ | 19/22 [00:04<00:00,  3.85it/s, loss=5.36, v_num=0]
Epoch 18:  91%|█████████ | 20/22 [00:05<00:00,  3.98it/s, loss=5.36, v_num=0]
Epoch 18:  91%|█████████ | 20/22 [00:05<00:00,  3.93it/s, loss=5.36, v_num=0]
Epoch 18:  95%|█████████▌| 21/22 [00:05<00:00,  4.05it/s, loss=5.36, v_num=0]
Epoch 18:  95%|█████████▌| 21/22 [00:05<00:00,  4.00it/s, loss=5.35, v_num=0]
Epoch 18: 100%|██████████| 22/22 [00:05<00:00,  4.13it/s, loss=5.35, v_num=0]
Epoch 18: 100%|██████████| 22/22 [00:05<00:00,  4.08it/s, loss=5.35, v_num=0]
Epoch 18: 100%|██████████| 22/22 [00:05<00:00,  4.08it/s, loss=5.35, v_num=0]
Epoch 18:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.35, v_num=0]
Epoch 19:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.35, v_num=0]
Epoch 19:   5%|▍         | 1/22 [00:02<00:43,  2.06s/it, loss=5.35, v_num=0]
Epoch 19:   5%|▍         | 1/22 [00:02<00:44,  2.12s/it, loss=5.36, v_num=0]
Epoch 19:   9%|▉         | 2/22 [00:02<00:22,  1.10s/it, loss=5.36, v_num=0]
Epoch 19:   9%|▉         | 2/22 [00:02<00:22,  1.14s/it, loss=5.36, v_num=0]
Epoch 19:  14%|█▎        | 3/22 [00:02<00:14,  1.27it/s, loss=5.36, v_num=0]
Epoch 19:  14%|█▎        | 3/22 [00:02<00:15,  1.24it/s, loss=5.36, v_num=0]
Epoch 19:  18%|█▊        | 4/22 [00:02<00:11,  1.59it/s, loss=5.36, v_num=0]
Epoch 19:  18%|█▊        | 4/22 [00:02<00:11,  1.55it/s, loss=5.36, v_num=0]
Epoch 19:  23%|██▎       | 5/22 [00:02<00:09,  1.87it/s, loss=5.36, v_num=0]
Epoch 19:  23%|██▎       | 5/22 [00:02<00:09,  1.83it/s, loss=5.36, v_num=0]
Epoch 19:  27%|██▋       | 6/22 [00:02<00:07,  2.12it/s, loss=5.36, v_num=0]
Epoch 19:  27%|██▋       | 6/22 [00:02<00:07,  2.07it/s, loss=5.36, v_num=0]
Epoch 19:  32%|███▏      | 7/22 [00:02<00:06,  2.34it/s, loss=5.36, v_num=0]
Epoch 19:  32%|███▏      | 7/22 [00:03<00:06,  2.30it/s, loss=5.36, v_num=0]
Epoch 19:  36%|███▋      | 8/22 [00:03<00:05,  2.55it/s, loss=5.36, v_num=0]
Epoch 19:  36%|███▋      | 8/22 [00:03<00:05,  2.50it/s, loss=5.36, v_num=0]
Epoch 19:  41%|████      | 9/22 [00:03<00:04,  2.69it/s, loss=5.36, v_num=0]
Epoch 19:  41%|████      | 9/22 [00:03<00:04,  2.64it/s, loss=5.36, v_num=0]
Epoch 19:  45%|████▌     | 10/22 [00:03<00:04,  2.86it/s, loss=5.36, v_num=0]
Epoch 19:  45%|████▌     | 10/22 [00:03<00:04,  2.81it/s, loss=5.36, v_num=0]
Epoch 19:  50%|█████     | 11/22 [00:03<00:03,  3.01it/s, loss=5.36, v_num=0]
Epoch 19:  50%|█████     | 11/22 [00:03<00:03,  2.96it/s, loss=5.36, v_num=0]
Epoch 19:  55%|█████▍    | 12/22 [00:03<00:03,  3.15it/s, loss=5.36, v_num=0]
Epoch 19:  55%|█████▍    | 12/22 [00:03<00:03,  3.10it/s, loss=5.35, v_num=0]
Epoch 19:  59%|█████▉    | 13/22 [00:03<00:02,  3.29it/s, loss=5.35, v_num=0]
Epoch 19:  59%|█████▉    | 13/22 [00:04<00:02,  3.23it/s, loss=5.35, v_num=0]
Epoch 19:  64%|██████▎   | 14/22 [00:04<00:02,  3.40it/s, loss=5.35, v_num=0]
Epoch 19:  64%|██████▎   | 14/22 [00:04<00:02,  3.35it/s, loss=5.36, v_num=0]
Epoch 19:  68%|██████▊   | 15/22 [00:04<00:01,  3.52it/s, loss=5.36, v_num=0]
Epoch 19:  68%|██████▊   | 15/22 [00:04<00:02,  3.47it/s, loss=5.36, v_num=0]
Epoch 19:  73%|███████▎  | 16/22 [00:04<00:01,  3.62it/s, loss=5.36, v_num=0]
Epoch 19:  73%|███████▎  | 16/22 [00:04<00:01,  3.57it/s, loss=5.36, v_num=0]
Epoch 19:  77%|███████▋  | 17/22 [00:04<00:01,  3.71it/s, loss=5.36, v_num=0]
Epoch 19:  77%|███████▋  | 17/22 [00:04<00:01,  3.66it/s, loss=5.36, v_num=0]
Epoch 19:  82%|████████▏ | 18/22 [00:04<00:01,  3.80it/s, loss=5.36, v_num=0]
Epoch 19:  82%|████████▏ | 18/22 [00:04<00:01,  3.75it/s, loss=5.36, v_num=0]
Epoch 19:  86%|████████▋ | 19/22 [00:04<00:00,  3.89it/s, loss=5.36, v_num=0]
Epoch 19:  86%|████████▋ | 19/22 [00:04<00:00,  3.84it/s, loss=5.36, v_num=0]
Epoch 19:  91%|█████████ | 20/22 [00:05<00:00,  3.96it/s, loss=5.36, v_num=0]
Epoch 19:  91%|█████████ | 20/22 [00:05<00:00,  3.92it/s, loss=5.36, v_num=0]
Epoch 19:  95%|█████████▌| 21/22 [00:05<00:00,  4.04it/s, loss=5.36, v_num=0]
Epoch 19:  95%|█████████▌| 21/22 [00:05<00:00,  3.99it/s, loss=5.36, v_num=0]
Epoch 19: 100%|██████████| 22/22 [00:05<00:00,  4.11it/s, loss=5.36, v_num=0]
Epoch 19: 100%|██████████| 22/22 [00:05<00:00,  4.07it/s, loss=5.36, v_num=0]
Epoch 19: 100%|██████████| 22/22 [00:05<00:00,  4.07it/s, loss=5.36, v_num=0]
Epoch 19: 100%|██████████| 22/22 [00:05<00:00,  3.94it/s, loss=5.36, v_num=0]

Next we create a helper function to generate embeddings from our test images using the model we just trained. Note that only the backbone is needed to generate embeddings, the projection head is only required for the training. Make sure to put the model into eval mode for this part!

def generate_embeddings(model, dataloader):
    """Generates representations for all images in the dataloader with
    the given model
    """

    embeddings = []
    filenames = []
    with torch.no_grad():
        for img, _, fnames in dataloader:
            img = img.to(model.device)
            emb = model.backbone(img).flatten(start_dim=1)
            embeddings.append(emb)
            filenames.extend(fnames)

    embeddings = torch.cat(embeddings, 0)
    embeddings = normalize(embeddings)
    return embeddings, filenames


model.eval()
embeddings, filenames = generate_embeddings(model, dataloader_test)

Visualize Nearest Neighbors

Let’s look at the trained embedding and visualize the nearest neighbors for a few random samples.

We create some helper functions to simplify the work

def get_image_as_np_array(filename: str):
    """Returns an image as an numpy array"""
    img = Image.open(filename)
    return np.asarray(img)


def plot_knn_examples(embeddings, filenames, n_neighbors=3, num_examples=6):
    """Plots multiple rows of random images with their nearest neighbors"""
    # lets look at the nearest neighbors for some samples
    # we use the sklearn library
    nbrs = NearestNeighbors(n_neighbors=n_neighbors).fit(embeddings)
    distances, indices = nbrs.kneighbors(embeddings)

    # get 5 random samples
    samples_idx = np.random.choice(len(indices), size=num_examples, replace=False)

    # loop through our randomly picked samples
    for idx in samples_idx:
        fig = plt.figure()
        # loop through their nearest neighbors
        for plot_x_offset, neighbor_idx in enumerate(indices[idx]):
            # add the subplot
            ax = fig.add_subplot(1, len(indices[idx]), plot_x_offset + 1)
            # get the correponding filename for the current index
            fname = os.path.join(path_to_data, filenames[neighbor_idx])
            # plot the image
            plt.imshow(get_image_as_np_array(fname))
            # set the title to the distance of the neighbor
            ax.set_title(f"d={distances[idx][plot_x_offset]:.3f}")
            # let's disable the axis
            plt.axis("off")

Let’s do the plot of the images. The leftmost image is the query image whereas the ones next to it on the same row are the nearest neighbors. In the title we see the distance of the neigbor.

plot_knn_examples(embeddings, filenames)
  • d=0.000, d=0.151, d=0.158
  • d=0.000, d=0.157, d=0.157
  • d=0.000, d=0.248, d=0.249
  • d=0.000, d=0.187, d=0.206
  • d=0.000, d=0.186, d=0.189
  • d=0.000, d=0.233, d=0.247

Color Invariance

Let’s train again without color augmentation. This will force our model to respect the colors in the images.

# Set color jitter and gray scale probability to 0
new_transform = SimCLRTransform(
    input_size=input_size, vf_prob=0.5, rr_prob=0.5, cj_prob=0.0, random_gray_scale=0.0
)

# let's update the transform on the training dataset
dataset_train_simclr.transform = new_transform

# then train a new model
model = SimCLRModel()
trainer = pl.Trainer(max_epochs=max_epochs, devices=1, accelerator="gpu")
trainer.fit(model, dataloader_train_simclr)

# and generate again embeddings from the test set
model.eval()
embeddings, filenames = generate_embeddings(model, dataloader_test)
/datasets/actions-runner/core_gpu_runner_01/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/lightning_fabric/plugins/environments/slurm.py:165: PossibleUserWarning: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python /datasets/actions-runner/core_gpu_runner_01/_work/li ...
  rank_zero_warn(
/datasets/actions-runner/core_gpu_runner_01/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1609: PossibleUserWarning: The number of training batches (22) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
  rank_zero_warn(

Training: 0it [00:00, ?it/s]
Training:   0%|          | 0/22 [00:00<?, ?it/s]
Epoch 0:   0%|          | 0/22 [00:00<?, ?it/s]
Epoch 0:   5%|▍         | 1/22 [00:01<00:30,  1.47s/it]
Epoch 0:   5%|▍         | 1/22 [00:01<00:32,  1.54s/it, loss=5.84, v_num=1]
Epoch 0:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, loss=5.84, v_num=1]
Epoch 0:   9%|▉         | 2/22 [00:01<00:16,  1.18it/s, loss=5.78, v_num=1]
Epoch 0:  14%|█▎        | 3/22 [00:01<00:11,  1.68it/s, loss=5.78, v_num=1]
Epoch 0:  14%|█▎        | 3/22 [00:01<00:11,  1.63it/s, loss=5.71, v_num=1]
Epoch 0:  18%|█▊        | 4/22 [00:01<00:08,  2.07it/s, loss=5.71, v_num=1]
Epoch 0:  18%|█▊        | 4/22 [00:01<00:08,  2.01it/s, loss=5.59, v_num=1]
Epoch 0:  23%|██▎       | 5/22 [00:02<00:07,  2.40it/s, loss=5.59, v_num=1]
Epoch 0:  23%|██▎       | 5/22 [00:02<00:07,  2.33it/s, loss=5.49, v_num=1]
Epoch 0:  27%|██▋       | 6/22 [00:02<00:05,  2.67it/s, loss=5.49, v_num=1]
Epoch 0:  27%|██▋       | 6/22 [00:02<00:06,  2.60it/s, loss=5.41, v_num=1]
Epoch 0:  32%|███▏      | 7/22 [00:02<00:05,  2.92it/s, loss=5.41, v_num=1]
Epoch 0:  32%|███▏      | 7/22 [00:02<00:05,  2.85it/s, loss=5.35, v_num=1]
Epoch 0:  36%|███▋      | 8/22 [00:02<00:04,  3.15it/s, loss=5.35, v_num=1]
Epoch 0:  36%|███▋      | 8/22 [00:02<00:04,  3.07it/s, loss=5.3, v_num=1]
Epoch 0:  41%|████      | 9/22 [00:02<00:03,  3.33it/s, loss=5.3, v_num=1]
Epoch 0:  41%|████      | 9/22 [00:02<00:03,  3.26it/s, loss=5.26, v_num=1]
Epoch 0:  45%|████▌     | 10/22 [00:02<00:03,  3.51it/s, loss=5.26, v_num=1]
Epoch 0:  45%|████▌     | 10/22 [00:02<00:03,  3.44it/s, loss=5.22, v_num=1]
Epoch 0:  50%|█████     | 11/22 [00:03<00:03,  3.66it/s, loss=5.22, v_num=1]
Epoch 0:  50%|█████     | 11/22 [00:03<00:03,  3.59it/s, loss=5.19, v_num=1]
Epoch 0:  55%|█████▍    | 12/22 [00:03<00:02,  3.80it/s, loss=5.19, v_num=1]
Epoch 0:  55%|█████▍    | 12/22 [00:03<00:02,  3.73it/s, loss=5.16, v_num=1]
Epoch 0:  59%|█████▉    | 13/22 [00:03<00:02,  3.93it/s, loss=5.16, v_num=1]
Epoch 0:  59%|█████▉    | 13/22 [00:03<00:02,  3.86it/s, loss=5.13, v_num=1]
Epoch 0:  64%|██████▎   | 14/22 [00:03<00:01,  4.05it/s, loss=5.13, v_num=1]
Epoch 0:  64%|██████▎   | 14/22 [00:03<00:02,  3.98it/s, loss=5.11, v_num=1]
Epoch 0:  68%|██████▊   | 15/22 [00:03<00:01,  4.16it/s, loss=5.11, v_num=1]
Epoch 0:  68%|██████▊   | 15/22 [00:03<00:01,  4.09it/s, loss=5.09, v_num=1]
Epoch 0:  73%|███████▎  | 16/22 [00:03<00:01,  4.26it/s, loss=5.09, v_num=1]
Epoch 0:  73%|███████▎  | 16/22 [00:03<00:01,  4.19it/s, loss=5.07, v_num=1]
Epoch 0:  77%|███████▋  | 17/22 [00:03<00:01,  4.35it/s, loss=5.07, v_num=1]
Epoch 0:  77%|███████▋  | 17/22 [00:03<00:01,  4.28it/s, loss=5.06, v_num=1]
Epoch 0:  82%|████████▏ | 18/22 [00:04<00:00,  4.44it/s, loss=5.06, v_num=1]
Epoch 0:  82%|████████▏ | 18/22 [00:04<00:00,  4.37it/s, loss=5.04, v_num=1]
Epoch 0:  86%|████████▋ | 19/22 [00:04<00:00,  4.51it/s, loss=5.04, v_num=1]
Epoch 0:  86%|████████▋ | 19/22 [00:04<00:00,  4.45it/s, loss=5.03, v_num=1]
Epoch 0:  91%|█████████ | 20/22 [00:04<00:00,  4.59it/s, loss=5.03, v_num=1]
Epoch 0:  91%|█████████ | 20/22 [00:04<00:00,  4.53it/s, loss=5.01, v_num=1]
Epoch 0:  95%|█████████▌| 21/22 [00:04<00:00,  4.66it/s, loss=5.01, v_num=1]
Epoch 0:  95%|█████████▌| 21/22 [00:04<00:00,  4.60it/s, loss=4.96, v_num=1]
Epoch 0: 100%|██████████| 22/22 [00:04<00:00,  4.72it/s, loss=4.96, v_num=1]
Epoch 0: 100%|██████████| 22/22 [00:04<00:00,  4.66it/s, loss=4.91, v_num=1]
Epoch 0: 100%|██████████| 22/22 [00:04<00:00,  4.66it/s, loss=4.91, v_num=1]
Epoch 0:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.91, v_num=1]
Epoch 1:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.91, v_num=1]
Epoch 1:   5%|▍         | 1/22 [00:01<00:36,  1.74s/it, loss=4.91, v_num=1]
Epoch 1:   5%|▍         | 1/22 [00:01<00:37,  1.80s/it, loss=4.87, v_num=1]
Epoch 1:   9%|▉         | 2/22 [00:01<00:18,  1.05it/s, loss=4.87, v_num=1]
Epoch 1:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, loss=4.84, v_num=1]
Epoch 1:  14%|█▎        | 3/22 [00:02<00:12,  1.46it/s, loss=4.84, v_num=1]
Epoch 1:  14%|█▎        | 3/22 [00:02<00:13,  1.42it/s, loss=4.83, v_num=1]
Epoch 1:  18%|█▊        | 4/22 [00:02<00:09,  1.82it/s, loss=4.83, v_num=1]
Epoch 1:  18%|█▊        | 4/22 [00:02<00:10,  1.77it/s, loss=4.82, v_num=1]
Epoch 1:  23%|██▎       | 5/22 [00:02<00:08,  2.12it/s, loss=4.82, v_num=1]
Epoch 1:  23%|██▎       | 5/22 [00:02<00:08,  2.07it/s, loss=4.8, v_num=1]
Epoch 1:  27%|██▋       | 6/22 [00:02<00:06,  2.38it/s, loss=4.8, v_num=1]
Epoch 1:  27%|██▋       | 6/22 [00:02<00:06,  2.33it/s, loss=4.79, v_num=1]
Epoch 1:  32%|███▏      | 7/22 [00:02<00:05,  2.62it/s, loss=4.79, v_num=1]
Epoch 1:  32%|███▏      | 7/22 [00:02<00:05,  2.56it/s, loss=4.79, v_num=1]
Epoch 1:  36%|███▋      | 8/22 [00:02<00:04,  2.84it/s, loss=4.79, v_num=1]
Epoch 1:  36%|███▋      | 8/22 [00:02<00:05,  2.77it/s, loss=4.78, v_num=1]
Epoch 1:  41%|████      | 9/22 [00:02<00:04,  3.02it/s, loss=4.78, v_num=1]
Epoch 1:  41%|████      | 9/22 [00:03<00:04,  2.96it/s, loss=4.77, v_num=1]
Epoch 1:  45%|████▌     | 10/22 [00:03<00:03,  3.20it/s, loss=4.77, v_num=1]
Epoch 1:  45%|████▌     | 10/22 [00:03<00:03,  3.14it/s, loss=4.77, v_num=1]
Epoch 1:  50%|█████     | 11/22 [00:03<00:03,  3.35it/s, loss=4.77, v_num=1]
Epoch 1:  50%|█████     | 11/22 [00:03<00:03,  3.29it/s, loss=4.76, v_num=1]
Epoch 1:  55%|█████▍    | 12/22 [00:03<00:02,  3.50it/s, loss=4.76, v_num=1]
Epoch 1:  55%|█████▍    | 12/22 [00:03<00:02,  3.44it/s, loss=4.75, v_num=1]
Epoch 1:  59%|█████▉    | 13/22 [00:03<00:02,  3.64it/s, loss=4.75, v_num=1]
Epoch 1:  59%|█████▉    | 13/22 [00:03<00:02,  3.57it/s, loss=4.75, v_num=1]
Epoch 1:  64%|██████▎   | 14/22 [00:03<00:02,  3.76it/s, loss=4.75, v_num=1]
Epoch 1:  64%|██████▎   | 14/22 [00:03<00:02,  3.70it/s, loss=4.74, v_num=1]
Epoch 1:  68%|██████▊   | 15/22 [00:03<00:01,  3.87it/s, loss=4.74, v_num=1]
Epoch 1:  68%|██████▊   | 15/22 [00:03<00:01,  3.81it/s, loss=4.74, v_num=1]
Epoch 1:  73%|███████▎  | 16/22 [00:04<00:01,  3.98it/s, loss=4.74, v_num=1]
Epoch 1:  73%|███████▎  | 16/22 [00:04<00:01,  3.92it/s, loss=4.73, v_num=1]
Epoch 1:  77%|███████▋  | 17/22 [00:04<00:01,  4.07it/s, loss=4.73, v_num=1]
Epoch 1:  77%|███████▋  | 17/22 [00:04<00:01,  4.01it/s, loss=4.73, v_num=1]
Epoch 1:  82%|████████▏ | 18/22 [00:04<00:00,  4.17it/s, loss=4.73, v_num=1]
Epoch 1:  82%|████████▏ | 18/22 [00:04<00:00,  4.11it/s, loss=4.73, v_num=1]
Epoch 1:  86%|████████▋ | 19/22 [00:04<00:00,  4.24it/s, loss=4.73, v_num=1]
Epoch 1:  86%|████████▋ | 19/22 [00:04<00:00,  4.19it/s, loss=4.72, v_num=1]
Epoch 1:  91%|█████████ | 20/22 [00:04<00:00,  4.33it/s, loss=4.72, v_num=1]
Epoch 1:  91%|█████████ | 20/22 [00:04<00:00,  4.27it/s, loss=4.72, v_num=1]
Epoch 1:  95%|█████████▌| 21/22 [00:04<00:00,  4.40it/s, loss=4.72, v_num=1]
Epoch 1:  95%|█████████▌| 21/22 [00:04<00:00,  4.35it/s, loss=4.71, v_num=1]
Epoch 1: 100%|██████████| 22/22 [00:04<00:00,  4.47it/s, loss=4.71, v_num=1]
Epoch 1: 100%|██████████| 22/22 [00:04<00:00,  4.42it/s, loss=4.71, v_num=1]
Epoch 1: 100%|██████████| 22/22 [00:04<00:00,  4.42it/s, loss=4.71, v_num=1]
Epoch 1:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.71, v_num=1]
Epoch 2:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.71, v_num=1]
Epoch 2:   5%|▍         | 1/22 [00:01<00:35,  1.71s/it, loss=4.71, v_num=1]
Epoch 2:   5%|▍         | 1/22 [00:01<00:37,  1.77s/it, loss=4.71, v_num=1]
Epoch 2:   9%|▉         | 2/22 [00:01<00:18,  1.07it/s, loss=4.71, v_num=1]
Epoch 2:   9%|▉         | 2/22 [00:01<00:19,  1.04it/s, loss=4.7, v_num=1]
Epoch 2:  14%|█▎        | 3/22 [00:02<00:12,  1.49it/s, loss=4.7, v_num=1]
Epoch 2:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, loss=4.7, v_num=1]
Epoch 2:  18%|█▊        | 4/22 [00:02<00:09,  1.85it/s, loss=4.7, v_num=1]
Epoch 2:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, loss=4.69, v_num=1]
Epoch 2:  23%|██▎       | 5/22 [00:02<00:07,  2.16it/s, loss=4.69, v_num=1]
Epoch 2:  23%|██▎       | 5/22 [00:02<00:08,  2.10it/s, loss=4.69, v_num=1]
Epoch 2:  27%|██▋       | 6/22 [00:02<00:06,  2.42it/s, loss=4.69, v_num=1]
Epoch 2:  27%|██▋       | 6/22 [00:02<00:06,  2.36it/s, loss=4.69, v_num=1]
Epoch 2:  32%|███▏      | 7/22 [00:02<00:05,  2.66it/s, loss=4.69, v_num=1]
Epoch 2:  32%|███▏      | 7/22 [00:02<00:05,  2.60it/s, loss=4.68, v_num=1]
Epoch 2:  36%|███▋      | 8/22 [00:02<00:04,  2.87it/s, loss=4.68, v_num=1]
Epoch 2:  36%|███▋      | 8/22 [00:02<00:04,  2.81it/s, loss=4.68, v_num=1]
Epoch 2:  41%|████      | 9/22 [00:02<00:04,  3.06it/s, loss=4.68, v_num=1]
Epoch 2:  41%|████      | 9/22 [00:03<00:04,  3.00it/s, loss=4.68, v_num=1]
Epoch 2:  45%|████▌     | 10/22 [00:03<00:03,  3.23it/s, loss=4.68, v_num=1]
Epoch 2:  45%|████▌     | 10/22 [00:03<00:03,  3.17it/s, loss=4.68, v_num=1]
Epoch 2:  50%|█████     | 11/22 [00:03<00:03,  3.39it/s, loss=4.68, v_num=1]
Epoch 2:  50%|█████     | 11/22 [00:03<00:03,  3.33it/s, loss=4.67, v_num=1]
Epoch 2:  55%|█████▍    | 12/22 [00:03<00:02,  3.54it/s, loss=4.67, v_num=1]
Epoch 2:  55%|█████▍    | 12/22 [00:03<00:02,  3.47it/s, loss=4.67, v_num=1]
Epoch 2:  59%|█████▉    | 13/22 [00:03<00:02,  3.67it/s, loss=4.67, v_num=1]
Epoch 2:  59%|█████▉    | 13/22 [00:03<00:02,  3.61it/s, loss=4.67, v_num=1]
Epoch 2:  64%|██████▎   | 14/22 [00:03<00:02,  3.79it/s, loss=4.67, v_num=1]
Epoch 2:  64%|██████▎   | 14/22 [00:03<00:02,  3.73it/s, loss=4.67, v_num=1]
Epoch 2:  68%|██████▊   | 15/22 [00:03<00:01,  3.91it/s, loss=4.67, v_num=1]
Epoch 2:  68%|██████▊   | 15/22 [00:03<00:01,  3.84it/s, loss=4.66, v_num=1]
Epoch 2:  73%|███████▎  | 16/22 [00:03<00:01,  4.01it/s, loss=4.66, v_num=1]
Epoch 2:  73%|███████▎  | 16/22 [00:04<00:01,  3.95it/s, loss=4.66, v_num=1]
Epoch 2:  77%|███████▋  | 17/22 [00:04<00:01,  4.11it/s, loss=4.66, v_num=1]
Epoch 2:  77%|███████▋  | 17/22 [00:04<00:01,  4.05it/s, loss=4.66, v_num=1]
Epoch 2:  82%|████████▏ | 18/22 [00:04<00:00,  4.20it/s, loss=4.66, v_num=1]
Epoch 2:  82%|████████▏ | 18/22 [00:04<00:00,  4.14it/s, loss=4.66, v_num=1]
Epoch 2:  86%|████████▋ | 19/22 [00:04<00:00,  4.28it/s, loss=4.66, v_num=1]
Epoch 2:  86%|████████▋ | 19/22 [00:04<00:00,  4.22it/s, loss=4.66, v_num=1]
Epoch 2:  91%|█████████ | 20/22 [00:04<00:00,  4.36it/s, loss=4.66, v_num=1]
Epoch 2:  91%|█████████ | 20/22 [00:04<00:00,  4.30it/s, loss=4.65, v_num=1]
Epoch 2:  95%|█████████▌| 21/22 [00:04<00:00,  4.43it/s, loss=4.65, v_num=1]
Epoch 2:  95%|█████████▌| 21/22 [00:04<00:00,  4.37it/s, loss=4.65, v_num=1]
Epoch 2: 100%|██████████| 22/22 [00:04<00:00,  4.50it/s, loss=4.65, v_num=1]
Epoch 2: 100%|██████████| 22/22 [00:04<00:00,  4.44it/s, loss=4.65, v_num=1]
Epoch 2: 100%|██████████| 22/22 [00:04<00:00,  4.44it/s, loss=4.65, v_num=1]
Epoch 2:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.65, v_num=1]
Epoch 3:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.65, v_num=1]
Epoch 3:   5%|▍         | 1/22 [00:01<00:35,  1.68s/it, loss=4.65, v_num=1]
Epoch 3:   5%|▍         | 1/22 [00:01<00:36,  1.73s/it, loss=4.64, v_num=1]
Epoch 3:   9%|▉         | 2/22 [00:01<00:18,  1.09it/s, loss=4.64, v_num=1]
Epoch 3:   9%|▉         | 2/22 [00:01<00:18,  1.06it/s, loss=4.64, v_num=1]
Epoch 3:  14%|█▎        | 3/22 [00:01<00:12,  1.52it/s, loss=4.64, v_num=1]
Epoch 3:  14%|█▎        | 3/22 [00:02<00:12,  1.47it/s, loss=4.64, v_num=1]
Epoch 3:  18%|█▊        | 4/22 [00:02<00:09,  1.88it/s, loss=4.64, v_num=1]
Epoch 3:  18%|█▊        | 4/22 [00:02<00:09,  1.82it/s, loss=4.64, v_num=1]
Epoch 3:  23%|██▎       | 5/22 [00:02<00:07,  2.19it/s, loss=4.64, v_num=1]
Epoch 3:  23%|██▎       | 5/22 [00:02<00:07,  2.13it/s, loss=4.64, v_num=1]
Epoch 3:  27%|██▋       | 6/22 [00:02<00:06,  2.45it/s, loss=4.64, v_num=1]
Epoch 3:  27%|██▋       | 6/22 [00:02<00:06,  2.39it/s, loss=4.64, v_num=1]
Epoch 3:  32%|███▏      | 7/22 [00:02<00:05,  2.70it/s, loss=4.64, v_num=1]
Epoch 3:  32%|███▏      | 7/22 [00:02<00:05,  2.64it/s, loss=4.64, v_num=1]
Epoch 3:  36%|███▋      | 8/22 [00:02<00:04,  2.92it/s, loss=4.64, v_num=1]
Epoch 3:  36%|███▋      | 8/22 [00:02<00:04,  2.85it/s, loss=4.63, v_num=1]
Epoch 3:  41%|████      | 9/22 [00:02<00:04,  3.10it/s, loss=4.63, v_num=1]
Epoch 3:  41%|████      | 9/22 [00:02<00:04,  3.04it/s, loss=4.63, v_num=1]
Epoch 3:  45%|████▌     | 10/22 [00:03<00:03,  3.28it/s, loss=4.63, v_num=1]
Epoch 3:  45%|████▌     | 10/22 [00:03<00:03,  3.21it/s, loss=4.63, v_num=1]
Epoch 3:  50%|█████     | 11/22 [00:03<00:03,  3.43it/s, loss=4.63, v_num=1]
Epoch 3:  50%|█████     | 11/22 [00:03<00:03,  3.37it/s, loss=4.63, v_num=1]
Epoch 3:  55%|█████▍    | 12/22 [00:03<00:02,  3.58it/s, loss=4.63, v_num=1]
Epoch 3:  55%|█████▍    | 12/22 [00:03<00:02,  3.51it/s, loss=4.63, v_num=1]
Epoch 3:  59%|█████▉    | 13/22 [00:03<00:02,  3.71it/s, loss=4.63, v_num=1]
Epoch 3:  59%|█████▉    | 13/22 [00:03<00:02,  3.65it/s, loss=4.63, v_num=1]
Epoch 3:  64%|██████▎   | 14/22 [00:03<00:02,  3.84it/s, loss=4.63, v_num=1]
Epoch 3:  64%|██████▎   | 14/22 [00:03<00:02,  3.77it/s, loss=4.63, v_num=1]
Epoch 3:  68%|██████▊   | 15/22 [00:03<00:01,  3.94it/s, loss=4.63, v_num=1]
Epoch 3:  68%|██████▊   | 15/22 [00:03<00:01,  3.88it/s, loss=4.63, v_num=1]
Epoch 3:  73%|███████▎  | 16/22 [00:03<00:01,  4.05it/s, loss=4.63, v_num=1]
Epoch 3:  73%|███████▎  | 16/22 [00:04<00:01,  3.99it/s, loss=4.63, v_num=1]
Epoch 3:  77%|███████▋  | 17/22 [00:04<00:01,  4.14it/s, loss=4.63, v_num=1]
Epoch 3:  77%|███████▋  | 17/22 [00:04<00:01,  4.08it/s, loss=4.63, v_num=1]
Epoch 3:  82%|████████▏ | 18/22 [00:04<00:00,  4.23it/s, loss=4.63, v_num=1]
Epoch 3:  82%|████████▏ | 18/22 [00:04<00:00,  4.17it/s, loss=4.63, v_num=1]
Epoch 3:  86%|████████▋ | 19/22 [00:04<00:00,  4.32it/s, loss=4.63, v_num=1]
Epoch 3:  86%|████████▋ | 19/22 [00:04<00:00,  4.26it/s, loss=4.63, v_num=1]
Epoch 3:  91%|█████████ | 20/22 [00:04<00:00,  4.39it/s, loss=4.63, v_num=1]
Epoch 3:  91%|█████████ | 20/22 [00:04<00:00,  4.34it/s, loss=4.63, v_num=1]
Epoch 3:  95%|█████████▌| 21/22 [00:04<00:00,  4.47it/s, loss=4.63, v_num=1]
Epoch 3:  95%|█████████▌| 21/22 [00:04<00:00,  4.41it/s, loss=4.63, v_num=1]
Epoch 3: 100%|██████████| 22/22 [00:04<00:00,  4.54it/s, loss=4.63, v_num=1]
Epoch 3: 100%|██████████| 22/22 [00:04<00:00,  4.48it/s, loss=4.63, v_num=1]
Epoch 3: 100%|██████████| 22/22 [00:04<00:00,  4.48it/s, loss=4.63, v_num=1]
Epoch 3:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.63, v_num=1]
Epoch 4:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.63, v_num=1]
Epoch 4:   5%|▍         | 1/22 [00:01<00:35,  1.68s/it, loss=4.63, v_num=1]
Epoch 4:   5%|▍         | 1/22 [00:01<00:36,  1.74s/it, loss=4.63, v_num=1]
Epoch 4:   9%|▉         | 2/22 [00:01<00:18,  1.09it/s, loss=4.63, v_num=1]
Epoch 4:   9%|▉         | 2/22 [00:01<00:18,  1.06it/s, loss=4.63, v_num=1]
Epoch 4:  14%|█▎        | 3/22 [00:01<00:12,  1.52it/s, loss=4.63, v_num=1]
Epoch 4:  14%|█▎        | 3/22 [00:02<00:12,  1.47it/s, loss=4.63, v_num=1]
Epoch 4:  18%|█▊        | 4/22 [00:02<00:09,  1.88it/s, loss=4.63, v_num=1]
Epoch 4:  18%|█▊        | 4/22 [00:02<00:09,  1.82it/s, loss=4.63, v_num=1]
Epoch 4:  23%|██▎       | 5/22 [00:02<00:07,  2.19it/s, loss=4.63, v_num=1]
Epoch 4:  23%|██▎       | 5/22 [00:02<00:07,  2.13it/s, loss=4.62, v_num=1]
Epoch 4:  27%|██▋       | 6/22 [00:02<00:06,  2.46it/s, loss=4.62, v_num=1]
Epoch 4:  27%|██▋       | 6/22 [00:02<00:06,  2.40it/s, loss=4.63, v_num=1]
Epoch 4:  32%|███▏      | 7/22 [00:02<00:05,  2.69it/s, loss=4.63, v_num=1]
Epoch 4:  32%|███▏      | 7/22 [00:02<00:05,  2.63it/s, loss=4.63, v_num=1]
Epoch 4:  36%|███▋      | 8/22 [00:02<00:04,  2.91it/s, loss=4.63, v_num=1]
Epoch 4:  36%|███▋      | 8/22 [00:02<00:04,  2.85it/s, loss=4.62, v_num=1]
Epoch 4:  41%|████      | 9/22 [00:02<00:04,  3.10it/s, loss=4.62, v_num=1]
Epoch 4:  41%|████      | 9/22 [00:02<00:04,  3.04it/s, loss=4.63, v_num=1]
Epoch 4:  45%|████▌     | 10/22 [00:03<00:03,  3.27it/s, loss=4.63, v_num=1]
Epoch 4:  45%|████▌     | 10/22 [00:03<00:03,  3.21it/s, loss=4.62, v_num=1]
Epoch 4:  50%|█████     | 11/22 [00:03<00:03,  3.43it/s, loss=4.62, v_num=1]
Epoch 4:  50%|█████     | 11/22 [00:03<00:03,  3.37it/s, loss=4.62, v_num=1]
Epoch 4:  55%|█████▍    | 12/22 [00:03<00:02,  3.57it/s, loss=4.62, v_num=1]
Epoch 4:  55%|█████▍    | 12/22 [00:03<00:02,  3.51it/s, loss=4.62, v_num=1]
Epoch 4:  59%|█████▉    | 13/22 [00:03<00:02,  3.71it/s, loss=4.62, v_num=1]
Epoch 4:  59%|█████▉    | 13/22 [00:03<00:02,  3.64it/s, loss=4.62, v_num=1]
Epoch 4:  64%|██████▎   | 14/22 [00:03<00:02,  3.83it/s, loss=4.62, v_num=1]
Epoch 4:  64%|██████▎   | 14/22 [00:03<00:02,  3.77it/s, loss=4.62, v_num=1]
Epoch 4:  68%|██████▊   | 15/22 [00:03<00:01,  3.94it/s, loss=4.62, v_num=1]
Epoch 4:  68%|██████▊   | 15/22 [00:03<00:01,  3.88it/s, loss=4.62, v_num=1]
Epoch 4:  73%|███████▎  | 16/22 [00:03<00:01,  4.05it/s, loss=4.62, v_num=1]
Epoch 4:  73%|███████▎  | 16/22 [00:04<00:01,  3.99it/s, loss=4.61, v_num=1]
Epoch 4:  77%|███████▋  | 17/22 [00:04<00:01,  4.14it/s, loss=4.61, v_num=1]
Epoch 4:  77%|███████▋  | 17/22 [00:04<00:01,  4.08it/s, loss=4.61, v_num=1]
Epoch 4:  82%|████████▏ | 18/22 [00:04<00:00,  4.23it/s, loss=4.61, v_num=1]
Epoch 4:  82%|████████▏ | 18/22 [00:04<00:00,  4.17it/s, loss=4.61, v_num=1]
Epoch 4:  86%|████████▋ | 19/22 [00:04<00:00,  4.32it/s, loss=4.61, v_num=1]
Epoch 4:  86%|████████▋ | 19/22 [00:04<00:00,  4.25it/s, loss=4.61, v_num=1]
Epoch 4:  91%|█████████ | 20/22 [00:04<00:00,  4.39it/s, loss=4.61, v_num=1]
Epoch 4:  91%|█████████ | 20/22 [00:04<00:00,  4.33it/s, loss=4.61, v_num=1]
Epoch 4:  95%|█████████▌| 21/22 [00:04<00:00,  4.46it/s, loss=4.61, v_num=1]
Epoch 4:  95%|█████████▌| 21/22 [00:04<00:00,  4.41it/s, loss=4.61, v_num=1]
Epoch 4: 100%|██████████| 22/22 [00:04<00:00,  4.53it/s, loss=4.61, v_num=1]
Epoch 4: 100%|██████████| 22/22 [00:04<00:00,  4.48it/s, loss=4.61, v_num=1]
Epoch 4: 100%|██████████| 22/22 [00:04<00:00,  4.48it/s, loss=4.61, v_num=1]
Epoch 4:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.61, v_num=1]
Epoch 5:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.61, v_num=1]
Epoch 5:   5%|▍         | 1/22 [00:01<00:35,  1.71s/it, loss=4.61, v_num=1]
Epoch 5:   5%|▍         | 1/22 [00:01<00:37,  1.77s/it, loss=4.61, v_num=1]
Epoch 5:   9%|▉         | 2/22 [00:01<00:18,  1.07it/s, loss=4.61, v_num=1]
Epoch 5:   9%|▉         | 2/22 [00:01<00:19,  1.04it/s, loss=4.61, v_num=1]
Epoch 5:  14%|█▎        | 3/22 [00:02<00:12,  1.49it/s, loss=4.61, v_num=1]
Epoch 5:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, loss=4.61, v_num=1]
Epoch 5:  18%|█▊        | 4/22 [00:02<00:09,  1.85it/s, loss=4.61, v_num=1]
Epoch 5:  18%|█▊        | 4/22 [00:02<00:10,  1.80it/s, loss=4.6, v_num=1]
Epoch 5:  23%|██▎       | 5/22 [00:02<00:07,  2.15it/s, loss=4.6, v_num=1]
Epoch 5:  23%|██▎       | 5/22 [00:02<00:08,  2.10it/s, loss=4.6, v_num=1]
Epoch 5:  27%|██▋       | 6/22 [00:02<00:06,  2.42it/s, loss=4.6, v_num=1]
Epoch 5:  27%|██▋       | 6/22 [00:02<00:06,  2.36it/s, loss=4.6, v_num=1]
Epoch 5:  32%|███▏      | 7/22 [00:02<00:05,  2.66it/s, loss=4.6, v_num=1]
Epoch 5:  32%|███▏      | 7/22 [00:02<00:05,  2.60it/s, loss=4.6, v_num=1]
Epoch 5:  36%|███▋      | 8/22 [00:02<00:04,  2.88it/s, loss=4.6, v_num=1]
Epoch 5:  36%|███▋      | 8/22 [00:02<00:04,  2.82it/s, loss=4.6, v_num=1]
Epoch 5:  41%|████      | 9/22 [00:02<00:04,  3.06it/s, loss=4.6, v_num=1]
Epoch 5:  41%|████      | 9/22 [00:02<00:04,  3.00it/s, loss=4.6, v_num=1]
Epoch 5:  45%|████▌     | 10/22 [00:03<00:03,  3.24it/s, loss=4.6, v_num=1]
Epoch 5:  45%|████▌     | 10/22 [00:03<00:03,  3.18it/s, loss=4.6, v_num=1]
Epoch 5:  50%|█████     | 11/22 [00:03<00:03,  3.40it/s, loss=4.6, v_num=1]
Epoch 5:  50%|█████     | 11/22 [00:03<00:03,  3.33it/s, loss=4.6, v_num=1]
Epoch 5:  55%|█████▍    | 12/22 [00:03<00:02,  3.54it/s, loss=4.6, v_num=1]
Epoch 5:  55%|█████▍    | 12/22 [00:03<00:02,  3.48it/s, loss=4.6, v_num=1]
Epoch 5:  59%|█████▉    | 13/22 [00:03<00:02,  3.68it/s, loss=4.6, v_num=1]
Epoch 5:  59%|█████▉    | 13/22 [00:03<00:02,  3.61it/s, loss=4.59, v_num=1]
Epoch 5:  64%|██████▎   | 14/22 [00:03<00:02,  3.80it/s, loss=4.59, v_num=1]
Epoch 5:  64%|██████▎   | 14/22 [00:03<00:02,  3.73it/s, loss=4.59, v_num=1]
Epoch 5:  68%|██████▊   | 15/22 [00:03<00:01,  3.91it/s, loss=4.59, v_num=1]
Epoch 5:  68%|██████▊   | 15/22 [00:03<00:01,  3.85it/s, loss=4.59, v_num=1]
Epoch 5:  73%|███████▎  | 16/22 [00:03<00:01,  4.02it/s, loss=4.59, v_num=1]
Epoch 5:  73%|███████▎  | 16/22 [00:04<00:01,  3.95it/s, loss=4.59, v_num=1]
Epoch 5:  77%|███████▋  | 17/22 [00:04<00:01,  4.11it/s, loss=4.59, v_num=1]
Epoch 5:  77%|███████▋  | 17/22 [00:04<00:01,  4.05it/s, loss=4.59, v_num=1]
Epoch 5:  82%|████████▏ | 18/22 [00:04<00:00,  4.20it/s, loss=4.59, v_num=1]
Epoch 5:  82%|████████▏ | 18/22 [00:04<00:00,  4.14it/s, loss=4.59, v_num=1]
Epoch 5:  86%|████████▋ | 19/22 [00:04<00:00,  4.28it/s, loss=4.59, v_num=1]
Epoch 5:  86%|████████▋ | 19/22 [00:04<00:00,  4.22it/s, loss=4.59, v_num=1]
Epoch 5:  91%|█████████ | 20/22 [00:04<00:00,  4.36it/s, loss=4.59, v_num=1]
Epoch 5:  91%|█████████ | 20/22 [00:04<00:00,  4.30it/s, loss=4.59, v_num=1]
Epoch 5:  95%|█████████▌| 21/22 [00:04<00:00,  4.44it/s, loss=4.59, v_num=1]
Epoch 5:  95%|█████████▌| 21/22 [00:04<00:00,  4.38it/s, loss=4.59, v_num=1]
Epoch 5: 100%|██████████| 22/22 [00:04<00:00,  4.51it/s, loss=4.59, v_num=1]
Epoch 5: 100%|██████████| 22/22 [00:04<00:00,  4.45it/s, loss=4.6, v_num=1]
Epoch 5: 100%|██████████| 22/22 [00:04<00:00,  4.45it/s, loss=4.6, v_num=1]
Epoch 5:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.6, v_num=1]
Epoch 6:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.6, v_num=1]
Epoch 6:   5%|▍         | 1/22 [00:01<00:36,  1.72s/it, loss=4.6, v_num=1]
Epoch 6:   5%|▍         | 1/22 [00:01<00:37,  1.78s/it, loss=4.6, v_num=1]
Epoch 6:   9%|▉         | 2/22 [00:01<00:18,  1.07it/s, loss=4.6, v_num=1]
Epoch 6:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, loss=4.6, v_num=1]
Epoch 6:  14%|█▎        | 3/22 [00:02<00:12,  1.48it/s, loss=4.6, v_num=1]
Epoch 6:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, loss=4.59, v_num=1]
Epoch 6:  18%|█▊        | 4/22 [00:02<00:09,  1.84it/s, loss=4.59, v_num=1]
Epoch 6:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, loss=4.59, v_num=1]
Epoch 6:  23%|██▎       | 5/22 [00:02<00:07,  2.14it/s, loss=4.59, v_num=1]
Epoch 6:  23%|██▎       | 5/22 [00:02<00:08,  2.09it/s, loss=4.59, v_num=1]
Epoch 6:  27%|██▋       | 6/22 [00:02<00:06,  2.41it/s, loss=4.59, v_num=1]
Epoch 6:  27%|██▋       | 6/22 [00:02<00:06,  2.35it/s, loss=4.59, v_num=1]
Epoch 6:  32%|███▏      | 7/22 [00:02<00:05,  2.65it/s, loss=4.59, v_num=1]
Epoch 6:  32%|███▏      | 7/22 [00:02<00:05,  2.59it/s, loss=4.59, v_num=1]
Epoch 6:  36%|███▋      | 8/22 [00:02<00:04,  2.87it/s, loss=4.59, v_num=1]
Epoch 6:  36%|███▋      | 8/22 [00:02<00:04,  2.81it/s, loss=4.59, v_num=1]
Epoch 6:  41%|████      | 9/22 [00:02<00:04,  3.05it/s, loss=4.59, v_num=1]
Epoch 6:  41%|████      | 9/22 [00:03<00:04,  2.99it/s, loss=4.59, v_num=1]
Epoch 6:  45%|████▌     | 10/22 [00:03<00:03,  3.23it/s, loss=4.59, v_num=1]
Epoch 6:  45%|████▌     | 10/22 [00:03<00:03,  3.17it/s, loss=4.59, v_num=1]
Epoch 6:  50%|█████     | 11/22 [00:03<00:03,  3.38it/s, loss=4.59, v_num=1]
Epoch 6:  50%|█████     | 11/22 [00:03<00:03,  3.32it/s, loss=4.59, v_num=1]
Epoch 6:  55%|█████▍    | 12/22 [00:03<00:02,  3.53it/s, loss=4.59, v_num=1]
Epoch 6:  55%|█████▍    | 12/22 [00:03<00:02,  3.46it/s, loss=4.59, v_num=1]
Epoch 6:  59%|█████▉    | 13/22 [00:03<00:02,  3.66it/s, loss=4.59, v_num=1]
Epoch 6:  59%|█████▉    | 13/22 [00:03<00:02,  3.60it/s, loss=4.59, v_num=1]
Epoch 6:  64%|██████▎   | 14/22 [00:03<00:02,  3.78it/s, loss=4.59, v_num=1]
Epoch 6:  64%|██████▎   | 14/22 [00:03<00:02,  3.72it/s, loss=4.59, v_num=1]
Epoch 6:  68%|██████▊   | 15/22 [00:03<00:01,  3.90it/s, loss=4.59, v_num=1]
Epoch 6:  68%|██████▊   | 15/22 [00:03<00:01,  3.83it/s, loss=4.59, v_num=1]
Epoch 6:  73%|███████▎  | 16/22 [00:03<00:01,  4.00it/s, loss=4.59, v_num=1]
Epoch 6:  73%|███████▎  | 16/22 [00:04<00:01,  3.94it/s, loss=4.59, v_num=1]
Epoch 6:  77%|███████▋  | 17/22 [00:04<00:01,  4.10it/s, loss=4.59, v_num=1]
Epoch 6:  77%|███████▋  | 17/22 [00:04<00:01,  4.03it/s, loss=4.59, v_num=1]
Epoch 6:  82%|████████▏ | 18/22 [00:04<00:00,  4.18it/s, loss=4.59, v_num=1]
Epoch 6:  82%|████████▏ | 18/22 [00:04<00:00,  4.12it/s, loss=4.59, v_num=1]
Epoch 6:  86%|████████▋ | 19/22 [00:04<00:00,  4.26it/s, loss=4.59, v_num=1]
Epoch 6:  86%|████████▋ | 19/22 [00:04<00:00,  4.21it/s, loss=4.59, v_num=1]
Epoch 6:  91%|█████████ | 20/22 [00:04<00:00,  4.35it/s, loss=4.59, v_num=1]
Epoch 6:  91%|█████████ | 20/22 [00:04<00:00,  4.29it/s, loss=4.59, v_num=1]
Epoch 6:  95%|█████████▌| 21/22 [00:04<00:00,  4.42it/s, loss=4.59, v_num=1]
Epoch 6:  95%|█████████▌| 21/22 [00:04<00:00,  4.36it/s, loss=4.59, v_num=1]
Epoch 6: 100%|██████████| 22/22 [00:04<00:00,  4.49it/s, loss=4.59, v_num=1]
Epoch 6: 100%|██████████| 22/22 [00:04<00:00,  4.43it/s, loss=4.59, v_num=1]
Epoch 6: 100%|██████████| 22/22 [00:04<00:00,  4.43it/s, loss=4.59, v_num=1]
Epoch 6:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.59, v_num=1]
Epoch 7:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.59, v_num=1]
Epoch 7:   5%|▍         | 1/22 [00:01<00:35,  1.69s/it, loss=4.59, v_num=1]
Epoch 7:   5%|▍         | 1/22 [00:01<00:36,  1.75s/it, loss=4.59, v_num=1]
Epoch 7:   9%|▉         | 2/22 [00:01<00:18,  1.08it/s, loss=4.59, v_num=1]
Epoch 7:   9%|▉         | 2/22 [00:01<00:19,  1.05it/s, loss=4.59, v_num=1]
Epoch 7:  14%|█▎        | 3/22 [00:01<00:12,  1.50it/s, loss=4.59, v_num=1]
Epoch 7:  14%|█▎        | 3/22 [00:02<00:13,  1.46it/s, loss=4.58, v_num=1]
Epoch 7:  18%|█▊        | 4/22 [00:02<00:09,  1.85it/s, loss=4.58, v_num=1]
Epoch 7:  18%|█▊        | 4/22 [00:02<00:09,  1.81it/s, loss=4.58, v_num=1]
Epoch 7:  23%|██▎       | 5/22 [00:02<00:07,  2.17it/s, loss=4.58, v_num=1]
Epoch 7:  23%|██▎       | 5/22 [00:02<00:08,  2.11it/s, loss=4.59, v_num=1]
Epoch 7:  27%|██▋       | 6/22 [00:02<00:06,  2.43it/s, loss=4.59, v_num=1]
Epoch 7:  27%|██▋       | 6/22 [00:02<00:06,  2.37it/s, loss=4.58, v_num=1]
Epoch 7:  32%|███▏      | 7/22 [00:02<00:05,  2.67it/s, loss=4.58, v_num=1]
Epoch 7:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, loss=4.58, v_num=1]
Epoch 7:  36%|███▋      | 8/22 [00:02<00:04,  2.89it/s, loss=4.58, v_num=1]
Epoch 7:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, loss=4.58, v_num=1]
Epoch 7:  41%|████      | 9/22 [00:02<00:04,  3.07it/s, loss=4.58, v_num=1]
Epoch 7:  41%|████      | 9/22 [00:02<00:04,  3.01it/s, loss=4.58, v_num=1]
Epoch 7:  45%|████▌     | 10/22 [00:03<00:03,  3.25it/s, loss=4.58, v_num=1]
Epoch 7:  45%|████▌     | 10/22 [00:03<00:03,  3.19it/s, loss=4.58, v_num=1]
Epoch 7:  50%|█████     | 11/22 [00:03<00:03,  3.41it/s, loss=4.58, v_num=1]
Epoch 7:  50%|█████     | 11/22 [00:03<00:03,  3.34it/s, loss=4.58, v_num=1]
Epoch 7:  55%|█████▍    | 12/22 [00:03<00:02,  3.55it/s, loss=4.58, v_num=1]
Epoch 7:  55%|█████▍    | 12/22 [00:03<00:02,  3.48it/s, loss=4.58, v_num=1]
Epoch 7:  59%|█████▉    | 13/22 [00:03<00:02,  3.68it/s, loss=4.58, v_num=1]
Epoch 7:  59%|█████▉    | 13/22 [00:03<00:02,  3.61it/s, loss=4.58, v_num=1]
Epoch 7:  64%|██████▎   | 14/22 [00:03<00:02,  3.80it/s, loss=4.58, v_num=1]
Epoch 7:  64%|██████▎   | 14/22 [00:03<00:02,  3.74it/s, loss=4.58, v_num=1]
Epoch 7:  68%|██████▊   | 15/22 [00:03<00:01,  3.91it/s, loss=4.58, v_num=1]
Epoch 7:  68%|██████▊   | 15/22 [00:03<00:01,  3.85it/s, loss=4.58, v_num=1]
Epoch 7:  73%|███████▎  | 16/22 [00:03<00:01,  4.02it/s, loss=4.58, v_num=1]
Epoch 7:  73%|███████▎  | 16/22 [00:04<00:01,  3.96it/s, loss=4.58, v_num=1]
Epoch 7:  77%|███████▋  | 17/22 [00:04<00:01,  4.11it/s, loss=4.58, v_num=1]
Epoch 7:  77%|███████▋  | 17/22 [00:04<00:01,  4.05it/s, loss=4.58, v_num=1]
Epoch 7:  82%|████████▏ | 18/22 [00:04<00:00,  4.20it/s, loss=4.58, v_num=1]
Epoch 7:  82%|████████▏ | 18/22 [00:04<00:00,  4.14it/s, loss=4.58, v_num=1]
Epoch 7:  86%|████████▋ | 19/22 [00:04<00:00,  4.28it/s, loss=4.58, v_num=1]
Epoch 7:  86%|████████▋ | 19/22 [00:04<00:00,  4.22it/s, loss=4.58, v_num=1]
Epoch 7:  91%|█████████ | 20/22 [00:04<00:00,  4.36it/s, loss=4.58, v_num=1]
Epoch 7:  91%|█████████ | 20/22 [00:04<00:00,  4.30it/s, loss=4.58, v_num=1]
Epoch 7:  95%|█████████▌| 21/22 [00:04<00:00,  4.43it/s, loss=4.58, v_num=1]
Epoch 7:  95%|█████████▌| 21/22 [00:04<00:00,  4.37it/s, loss=4.57, v_num=1]
Epoch 7: 100%|██████████| 22/22 [00:04<00:00,  4.50it/s, loss=4.57, v_num=1]
Epoch 7: 100%|██████████| 22/22 [00:04<00:00,  4.44it/s, loss=4.57, v_num=1]
Epoch 7: 100%|██████████| 22/22 [00:04<00:00,  4.44it/s, loss=4.57, v_num=1]
Epoch 7:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.57, v_num=1]
Epoch 8:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.57, v_num=1]
Epoch 8:   5%|▍         | 1/22 [00:01<00:36,  1.73s/it, loss=4.57, v_num=1]
Epoch 8:   5%|▍         | 1/22 [00:01<00:37,  1.79s/it, loss=4.57, v_num=1]
Epoch 8:   9%|▉         | 2/22 [00:01<00:18,  1.06it/s, loss=4.57, v_num=1]
Epoch 8:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, loss=4.57, v_num=1]
Epoch 8:  14%|█▎        | 3/22 [00:02<00:12,  1.47it/s, loss=4.57, v_num=1]
Epoch 8:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, loss=4.57, v_num=1]
Epoch 8:  18%|█▊        | 4/22 [00:02<00:09,  1.82it/s, loss=4.57, v_num=1]
Epoch 8:  18%|█▊        | 4/22 [00:02<00:10,  1.78it/s, loss=4.57, v_num=1]
Epoch 8:  23%|██▎       | 5/22 [00:02<00:07,  2.13it/s, loss=4.57, v_num=1]
Epoch 8:  23%|██▎       | 5/22 [00:02<00:08,  2.08it/s, loss=4.57, v_num=1]
Epoch 8:  27%|██▋       | 6/22 [00:02<00:06,  2.40it/s, loss=4.57, v_num=1]
Epoch 8:  27%|██▋       | 6/22 [00:02<00:06,  2.34it/s, loss=4.57, v_num=1]
Epoch 8:  32%|███▏      | 7/22 [00:02<00:05,  2.64it/s, loss=4.57, v_num=1]
Epoch 8:  32%|███▏      | 7/22 [00:02<00:05,  2.58it/s, loss=4.57, v_num=1]
Epoch 8:  36%|███▋      | 8/22 [00:02<00:04,  2.85it/s, loss=4.57, v_num=1]
Epoch 8:  36%|███▋      | 8/22 [00:02<00:05,  2.79it/s, loss=4.57, v_num=1]
Epoch 8:  41%|████      | 9/22 [00:02<00:04,  3.04it/s, loss=4.57, v_num=1]
Epoch 8:  41%|████      | 9/22 [00:03<00:04,  2.98it/s, loss=4.57, v_num=1]
Epoch 8:  45%|████▌     | 10/22 [00:03<00:03,  3.21it/s, loss=4.57, v_num=1]
Epoch 8:  45%|████▌     | 10/22 [00:03<00:03,  3.15it/s, loss=4.57, v_num=1]
Epoch 8:  50%|█████     | 11/22 [00:03<00:03,  3.36it/s, loss=4.57, v_num=1]
Epoch 8:  50%|█████     | 11/22 [00:03<00:03,  3.30it/s, loss=4.57, v_num=1]
Epoch 8:  55%|█████▍    | 12/22 [00:03<00:02,  3.51it/s, loss=4.57, v_num=1]
Epoch 8:  55%|█████▍    | 12/22 [00:03<00:02,  3.45it/s, loss=4.57, v_num=1]
Epoch 8:  59%|█████▉    | 13/22 [00:03<00:02,  3.64it/s, loss=4.57, v_num=1]
Epoch 8:  59%|█████▉    | 13/22 [00:03<00:02,  3.58it/s, loss=4.57, v_num=1]
Epoch 8:  64%|██████▎   | 14/22 [00:03<00:02,  3.76it/s, loss=4.57, v_num=1]
Epoch 8:  64%|██████▎   | 14/22 [00:03<00:02,  3.71it/s, loss=4.57, v_num=1]
Epoch 8:  68%|██████▊   | 15/22 [00:03<00:01,  3.88it/s, loss=4.57, v_num=1]
Epoch 8:  68%|██████▊   | 15/22 [00:03<00:01,  3.82it/s, loss=4.57, v_num=1]
Epoch 8:  73%|███████▎  | 16/22 [00:04<00:01,  3.98it/s, loss=4.57, v_num=1]
Epoch 8:  73%|███████▎  | 16/22 [00:04<00:01,  3.92it/s, loss=4.57, v_num=1]
Epoch 8:  77%|███████▋  | 17/22 [00:04<00:01,  4.07it/s, loss=4.57, v_num=1]
Epoch 8:  77%|███████▋  | 17/22 [00:04<00:01,  4.02it/s, loss=4.57, v_num=1]
Epoch 8:  82%|████████▏ | 18/22 [00:04<00:00,  4.17it/s, loss=4.57, v_num=1]
Epoch 8:  82%|████████▏ | 18/22 [00:04<00:00,  4.11it/s, loss=4.57, v_num=1]
Epoch 8:  86%|████████▋ | 19/22 [00:04<00:00,  4.26it/s, loss=4.57, v_num=1]
Epoch 8:  86%|████████▋ | 19/22 [00:04<00:00,  4.19it/s, loss=4.57, v_num=1]
Epoch 8:  91%|█████████ | 20/22 [00:04<00:00,  4.33it/s, loss=4.57, v_num=1]
Epoch 8:  91%|█████████ | 20/22 [00:04<00:00,  4.27it/s, loss=4.57, v_num=1]
Epoch 8:  95%|█████████▌| 21/22 [00:04<00:00,  4.41it/s, loss=4.57, v_num=1]
Epoch 8:  95%|█████████▌| 21/22 [00:04<00:00,  4.35it/s, loss=4.57, v_num=1]
Epoch 8: 100%|██████████| 22/22 [00:04<00:00,  4.48it/s, loss=4.57, v_num=1]
Epoch 8: 100%|██████████| 22/22 [00:04<00:00,  4.42it/s, loss=4.57, v_num=1]
Epoch 8: 100%|██████████| 22/22 [00:04<00:00,  4.42it/s, loss=4.57, v_num=1]
Epoch 8:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.57, v_num=1]
Epoch 9:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.57, v_num=1]
Epoch 9:   5%|▍         | 1/22 [00:01<00:36,  1.75s/it, loss=4.57, v_num=1]
Epoch 9:   5%|▍         | 1/22 [00:01<00:37,  1.81s/it, loss=4.57, v_num=1]
Epoch 9:   9%|▉         | 2/22 [00:01<00:19,  1.05it/s, loss=4.57, v_num=1]
Epoch 9:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, loss=4.57, v_num=1]
Epoch 9:  14%|█▎        | 3/22 [00:02<00:13,  1.46it/s, loss=4.57, v_num=1]
Epoch 9:  14%|█▎        | 3/22 [00:02<00:13,  1.42it/s, loss=4.57, v_num=1]
Epoch 9:  18%|█▊        | 4/22 [00:02<00:09,  1.81it/s, loss=4.57, v_num=1]
Epoch 9:  18%|█▊        | 4/22 [00:02<00:10,  1.76it/s, loss=4.56, v_num=1]
Epoch 9:  23%|██▎       | 5/22 [00:02<00:08,  2.11it/s, loss=4.56, v_num=1]
Epoch 9:  23%|██▎       | 5/22 [00:02<00:08,  2.06it/s, loss=4.56, v_num=1]
Epoch 9:  27%|██▋       | 6/22 [00:02<00:06,  2.37it/s, loss=4.56, v_num=1]
Epoch 9:  27%|██▋       | 6/22 [00:02<00:06,  2.32it/s, loss=4.56, v_num=1]
Epoch 9:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, loss=4.56, v_num=1]
Epoch 9:  32%|███▏      | 7/22 [00:02<00:05,  2.55it/s, loss=4.56, v_num=1]
Epoch 9:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, loss=4.56, v_num=1]
Epoch 9:  36%|███▋      | 8/22 [00:02<00:05,  2.77it/s, loss=4.56, v_num=1]
Epoch 9:  41%|████      | 9/22 [00:02<00:04,  3.02it/s, loss=4.56, v_num=1]
Epoch 9:  41%|████      | 9/22 [00:03<00:04,  2.96it/s, loss=4.56, v_num=1]
Epoch 9:  45%|████▌     | 10/22 [00:03<00:03,  3.19it/s, loss=4.56, v_num=1]
Epoch 9:  45%|████▌     | 10/22 [00:03<00:03,  3.13it/s, loss=4.56, v_num=1]
Epoch 9:  50%|█████     | 11/22 [00:03<00:03,  3.34it/s, loss=4.56, v_num=1]
Epoch 9:  50%|█████     | 11/22 [00:03<00:03,  3.28it/s, loss=4.56, v_num=1]
Epoch 9:  55%|█████▍    | 12/22 [00:03<00:02,  3.49it/s, loss=4.56, v_num=1]
Epoch 9:  55%|█████▍    | 12/22 [00:03<00:02,  3.43it/s, loss=4.56, v_num=1]
Epoch 9:  59%|█████▉    | 13/22 [00:03<00:02,  3.62it/s, loss=4.56, v_num=1]
Epoch 9:  59%|█████▉    | 13/22 [00:03<00:02,  3.56it/s, loss=4.55, v_num=1]
Epoch 9:  64%|██████▎   | 14/22 [00:03<00:02,  3.74it/s, loss=4.55, v_num=1]
Epoch 9:  64%|██████▎   | 14/22 [00:03<00:02,  3.68it/s, loss=4.55, v_num=1]
Epoch 9:  68%|██████▊   | 15/22 [00:03<00:01,  3.86it/s, loss=4.55, v_num=1]
Epoch 9:  68%|██████▊   | 15/22 [00:03<00:01,  3.80it/s, loss=4.55, v_num=1]
Epoch 9:  73%|███████▎  | 16/22 [00:04<00:01,  3.97it/s, loss=4.55, v_num=1]
Epoch 9:  73%|███████▎  | 16/22 [00:04<00:01,  3.90it/s, loss=4.55, v_num=1]
Epoch 9:  77%|███████▋  | 17/22 [00:04<00:01,  4.06it/s, loss=4.55, v_num=1]
Epoch 9:  77%|███████▋  | 17/22 [00:04<00:01,  4.00it/s, loss=4.55, v_num=1]
Epoch 9:  82%|████████▏ | 18/22 [00:04<00:00,  4.15it/s, loss=4.55, v_num=1]
Epoch 9:  82%|████████▏ | 18/22 [00:04<00:00,  4.09it/s, loss=4.55, v_num=1]
Epoch 9:  86%|████████▋ | 19/22 [00:04<00:00,  4.23it/s, loss=4.55, v_num=1]
Epoch 9:  86%|████████▋ | 19/22 [00:04<00:00,  4.17it/s, loss=4.55, v_num=1]
Epoch 9:  91%|█████████ | 20/22 [00:04<00:00,  4.31it/s, loss=4.55, v_num=1]
Epoch 9:  91%|█████████ | 20/22 [00:04<00:00,  4.25it/s, loss=4.55, v_num=1]
Epoch 9:  95%|█████████▌| 21/22 [00:04<00:00,  4.38it/s, loss=4.55, v_num=1]
Epoch 9:  95%|█████████▌| 21/22 [00:04<00:00,  4.33it/s, loss=4.55, v_num=1]
Epoch 9: 100%|██████████| 22/22 [00:04<00:00,  4.46it/s, loss=4.55, v_num=1]
Epoch 9: 100%|██████████| 22/22 [00:05<00:00,  4.40it/s, loss=4.55, v_num=1]
Epoch 9: 100%|██████████| 22/22 [00:05<00:00,  4.40it/s, loss=4.55, v_num=1]
Epoch 9:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.55, v_num=1]
Epoch 10:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.55, v_num=1]
Epoch 10:   5%|▍         | 1/22 [00:01<00:36,  1.76s/it, loss=4.55, v_num=1]
Epoch 10:   5%|▍         | 1/22 [00:01<00:38,  1.82s/it, loss=4.55, v_num=1]
Epoch 10:   9%|▉         | 2/22 [00:01<00:19,  1.04it/s, loss=4.55, v_num=1]
Epoch 10:   9%|▉         | 2/22 [00:01<00:19,  1.01it/s, loss=4.55, v_num=1]
Epoch 10:  14%|█▎        | 3/22 [00:02<00:13,  1.45it/s, loss=4.55, v_num=1]
Epoch 10:  14%|█▎        | 3/22 [00:02<00:13,  1.41it/s, loss=4.55, v_num=1]
Epoch 10:  18%|█▊        | 4/22 [00:02<00:09,  1.80it/s, loss=4.55, v_num=1]
Epoch 10:  18%|█▊        | 4/22 [00:02<00:10,  1.75it/s, loss=4.55, v_num=1]
Epoch 10:  23%|██▎       | 5/22 [00:02<00:08,  2.11it/s, loss=4.55, v_num=1]
Epoch 10:  23%|██▎       | 5/22 [00:02<00:08,  2.06it/s, loss=4.55, v_num=1]
Epoch 10:  27%|██▋       | 6/22 [00:02<00:06,  2.36it/s, loss=4.55, v_num=1]
Epoch 10:  27%|██▋       | 6/22 [00:02<00:06,  2.31it/s, loss=4.55, v_num=1]
Epoch 10:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, loss=4.55, v_num=1]
Epoch 10:  32%|███▏      | 7/22 [00:02<00:05,  2.55it/s, loss=4.55, v_num=1]
Epoch 10:  36%|███▋      | 8/22 [00:02<00:04,  2.82it/s, loss=4.55, v_num=1]
Epoch 10:  36%|███▋      | 8/22 [00:02<00:05,  2.76it/s, loss=4.55, v_num=1]
Epoch 10:  41%|████      | 9/22 [00:02<00:04,  3.01it/s, loss=4.55, v_num=1]
Epoch 10:  41%|████      | 9/22 [00:03<00:04,  2.95it/s, loss=4.55, v_num=1]
Epoch 10:  45%|████▌     | 10/22 [00:03<00:03,  3.18it/s, loss=4.55, v_num=1]
Epoch 10:  45%|████▌     | 10/22 [00:03<00:03,  3.12it/s, loss=4.55, v_num=1]
Epoch 10:  50%|█████     | 11/22 [00:03<00:03,  3.34it/s, loss=4.55, v_num=1]
Epoch 10:  50%|█████     | 11/22 [00:03<00:03,  3.28it/s, loss=4.55, v_num=1]
Epoch 10:  55%|█████▍    | 12/22 [00:03<00:02,  3.49it/s, loss=4.55, v_num=1]
Epoch 10:  55%|█████▍    | 12/22 [00:03<00:02,  3.42it/s, loss=4.55, v_num=1]
Epoch 10:  59%|█████▉    | 13/22 [00:03<00:02,  3.61it/s, loss=4.55, v_num=1]
Epoch 10:  59%|█████▉    | 13/22 [00:03<00:02,  3.55it/s, loss=4.55, v_num=1]
Epoch 10:  64%|██████▎   | 14/22 [00:03<00:02,  3.74it/s, loss=4.55, v_num=1]
Epoch 10:  64%|██████▎   | 14/22 [00:03<00:02,  3.68it/s, loss=4.55, v_num=1]
Epoch 10:  68%|██████▊   | 15/22 [00:03<00:01,  3.85it/s, loss=4.55, v_num=1]
Epoch 10:  68%|██████▊   | 15/22 [00:03<00:01,  3.79it/s, loss=4.55, v_num=1]
Epoch 10:  73%|███████▎  | 16/22 [00:04<00:01,  3.95it/s, loss=4.55, v_num=1]
Epoch 10:  73%|███████▎  | 16/22 [00:04<00:01,  3.89it/s, loss=4.55, v_num=1]
Epoch 10:  77%|███████▋  | 17/22 [00:04<00:01,  4.05it/s, loss=4.55, v_num=1]
Epoch 10:  77%|███████▋  | 17/22 [00:04<00:01,  3.99it/s, loss=4.55, v_num=1]
Epoch 10:  82%|████████▏ | 18/22 [00:04<00:00,  4.14it/s, loss=4.55, v_num=1]
Epoch 10:  82%|████████▏ | 18/22 [00:04<00:00,  4.08it/s, loss=4.55, v_num=1]
Epoch 10:  86%|████████▋ | 19/22 [00:04<00:00,  4.22it/s, loss=4.55, v_num=1]
Epoch 10:  86%|████████▋ | 19/22 [00:04<00:00,  4.16it/s, loss=4.55, v_num=1]
Epoch 10:  91%|█████████ | 20/22 [00:04<00:00,  4.30it/s, loss=4.55, v_num=1]
Epoch 10:  91%|█████████ | 20/22 [00:04<00:00,  4.24it/s, loss=4.55, v_num=1]
Epoch 10:  95%|█████████▌| 21/22 [00:04<00:00,  4.37it/s, loss=4.55, v_num=1]
Epoch 10:  95%|█████████▌| 21/22 [00:04<00:00,  4.32it/s, loss=4.55, v_num=1]
Epoch 10: 100%|██████████| 22/22 [00:04<00:00,  4.44it/s, loss=4.55, v_num=1]
Epoch 10: 100%|██████████| 22/22 [00:05<00:00,  4.39it/s, loss=4.55, v_num=1]
Epoch 10: 100%|██████████| 22/22 [00:05<00:00,  4.39it/s, loss=4.55, v_num=1]
Epoch 10:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.55, v_num=1]
Epoch 11:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.55, v_num=1]
Epoch 11:   5%|▍         | 1/22 [00:01<00:34,  1.66s/it, loss=4.55, v_num=1]
Epoch 11:   5%|▍         | 1/22 [00:01<00:36,  1.72s/it, loss=4.55, v_num=1]
Epoch 11:   9%|▉         | 2/22 [00:01<00:18,  1.10it/s, loss=4.55, v_num=1]
Epoch 11:   9%|▉         | 2/22 [00:01<00:18,  1.07it/s, loss=4.55, v_num=1]
Epoch 11:  14%|█▎        | 3/22 [00:01<00:12,  1.52it/s, loss=4.55, v_num=1]
Epoch 11:  14%|█▎        | 3/22 [00:02<00:12,  1.47it/s, loss=4.55, v_num=1]
Epoch 11:  18%|█▊        | 4/22 [00:02<00:09,  1.88it/s, loss=4.55, v_num=1]
Epoch 11:  18%|█▊        | 4/22 [00:02<00:09,  1.83it/s, loss=4.55, v_num=1]
Epoch 11:  23%|██▎       | 5/22 [00:02<00:07,  2.20it/s, loss=4.55, v_num=1]
Epoch 11:  23%|██▎       | 5/22 [00:02<00:07,  2.14it/s, loss=4.55, v_num=1]
Epoch 11:  27%|██▋       | 6/22 [00:02<00:06,  2.47it/s, loss=4.55, v_num=1]
Epoch 11:  27%|██▋       | 6/22 [00:02<00:06,  2.40it/s, loss=4.55, v_num=1]
Epoch 11:  32%|███▏      | 7/22 [00:02<00:05,  2.71it/s, loss=4.55, v_num=1]
Epoch 11:  32%|███▏      | 7/22 [00:02<00:05,  2.64it/s, loss=4.55, v_num=1]
Epoch 11:  36%|███▋      | 8/22 [00:02<00:04,  2.93it/s, loss=4.55, v_num=1]
Epoch 11:  36%|███▋      | 8/22 [00:02<00:04,  2.86it/s, loss=4.55, v_num=1]
Epoch 11:  41%|████      | 9/22 [00:02<00:04,  3.12it/s, loss=4.55, v_num=1]
Epoch 11:  41%|████      | 9/22 [00:02<00:04,  3.05it/s, loss=4.55, v_num=1]
Epoch 11:  45%|████▌     | 10/22 [00:03<00:03,  3.28it/s, loss=4.55, v_num=1]
Epoch 11:  45%|████▌     | 10/22 [00:03<00:03,  3.22it/s, loss=4.55, v_num=1]
Epoch 11:  50%|█████     | 11/22 [00:03<00:03,  3.43it/s, loss=4.55, v_num=1]
Epoch 11:  50%|█████     | 11/22 [00:03<00:03,  3.37it/s, loss=4.55, v_num=1]
Epoch 11:  55%|█████▍    | 12/22 [00:03<00:02,  3.58it/s, loss=4.55, v_num=1]
Epoch 11:  55%|█████▍    | 12/22 [00:03<00:02,  3.51it/s, loss=4.55, v_num=1]
Epoch 11:  59%|█████▉    | 13/22 [00:03<00:02,  3.71it/s, loss=4.55, v_num=1]
Epoch 11:  59%|█████▉    | 13/22 [00:03<00:02,  3.65it/s, loss=4.55, v_num=1]
Epoch 11:  64%|██████▎   | 14/22 [00:03<00:02,  3.83it/s, loss=4.55, v_num=1]
Epoch 11:  64%|██████▎   | 14/22 [00:03<00:02,  3.77it/s, loss=4.55, v_num=1]
Epoch 11:  68%|██████▊   | 15/22 [00:03<00:01,  3.94it/s, loss=4.55, v_num=1]
Epoch 11:  68%|██████▊   | 15/22 [00:03<00:01,  3.88it/s, loss=4.55, v_num=1]
Epoch 11:  73%|███████▎  | 16/22 [00:03<00:01,  4.05it/s, loss=4.55, v_num=1]
Epoch 11:  73%|███████▎  | 16/22 [00:04<00:01,  3.99it/s, loss=4.55, v_num=1]
Epoch 11:  77%|███████▋  | 17/22 [00:04<00:01,  4.14it/s, loss=4.55, v_num=1]
Epoch 11:  77%|███████▋  | 17/22 [00:04<00:01,  4.08it/s, loss=4.55, v_num=1]
Epoch 11:  82%|████████▏ | 18/22 [00:04<00:00,  4.22it/s, loss=4.55, v_num=1]
Epoch 11:  82%|████████▏ | 18/22 [00:04<00:00,  4.17it/s, loss=4.55, v_num=1]
Epoch 11:  86%|████████▋ | 19/22 [00:04<00:00,  4.31it/s, loss=4.55, v_num=1]
Epoch 11:  86%|████████▋ | 19/22 [00:04<00:00,  4.25it/s, loss=4.55, v_num=1]
Epoch 11:  91%|█████████ | 20/22 [00:04<00:00,  4.39it/s, loss=4.55, v_num=1]
Epoch 11:  91%|█████████ | 20/22 [00:04<00:00,  4.33it/s, loss=4.55, v_num=1]
Epoch 11:  95%|█████████▌| 21/22 [00:04<00:00,  4.46it/s, loss=4.55, v_num=1]
Epoch 11:  95%|█████████▌| 21/22 [00:04<00:00,  4.40it/s, loss=4.55, v_num=1]
Epoch 11: 100%|██████████| 22/22 [00:04<00:00,  4.53it/s, loss=4.55, v_num=1]
Epoch 11: 100%|██████████| 22/22 [00:04<00:00,  4.47it/s, loss=4.55, v_num=1]
Epoch 11: 100%|██████████| 22/22 [00:04<00:00,  4.47it/s, loss=4.55, v_num=1]
Epoch 11:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.55, v_num=1]
Epoch 12:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.55, v_num=1]
Epoch 12:   5%|▍         | 1/22 [00:01<00:36,  1.76s/it, loss=4.55, v_num=1]
Epoch 12:   5%|▍         | 1/22 [00:01<00:38,  1.81s/it, loss=4.55, v_num=1]
Epoch 12:   9%|▉         | 2/22 [00:01<00:19,  1.05it/s, loss=4.55, v_num=1]
Epoch 12:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, loss=4.55, v_num=1]
Epoch 12:  14%|█▎        | 3/22 [00:02<00:13,  1.46it/s, loss=4.55, v_num=1]
Epoch 12:  14%|█▎        | 3/22 [00:02<00:13,  1.42it/s, loss=4.54, v_num=1]
Epoch 12:  18%|█▊        | 4/22 [00:02<00:09,  1.81it/s, loss=4.54, v_num=1]
Epoch 12:  18%|█▊        | 4/22 [00:02<00:10,  1.76it/s, loss=4.54, v_num=1]
Epoch 12:  23%|██▎       | 5/22 [00:02<00:08,  2.12it/s, loss=4.54, v_num=1]
Epoch 12:  23%|██▎       | 5/22 [00:02<00:08,  2.06it/s, loss=4.55, v_num=1]
Epoch 12:  27%|██▋       | 6/22 [00:02<00:06,  2.38it/s, loss=4.55, v_num=1]
Epoch 12:  27%|██▋       | 6/22 [00:02<00:06,  2.33it/s, loss=4.55, v_num=1]
Epoch 12:  32%|███▏      | 7/22 [00:02<00:05,  2.62it/s, loss=4.55, v_num=1]
Epoch 12:  32%|███▏      | 7/22 [00:02<00:05,  2.56it/s, loss=4.55, v_num=1]
Epoch 12:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, loss=4.55, v_num=1]
Epoch 12:  36%|███▋      | 8/22 [00:02<00:05,  2.77it/s, loss=4.55, v_num=1]
Epoch 12:  41%|████      | 9/22 [00:02<00:04,  3.02it/s, loss=4.55, v_num=1]
Epoch 12:  41%|████      | 9/22 [00:03<00:04,  2.96it/s, loss=4.54, v_num=1]
Epoch 12:  45%|████▌     | 10/22 [00:03<00:03,  3.20it/s, loss=4.54, v_num=1]
Epoch 12:  45%|████▌     | 10/22 [00:03<00:03,  3.14it/s, loss=4.55, v_num=1]
Epoch 12:  50%|█████     | 11/22 [00:03<00:03,  3.35it/s, loss=4.55, v_num=1]
Epoch 12:  50%|█████     | 11/22 [00:03<00:03,  3.29it/s, loss=4.55, v_num=1]
Epoch 12:  55%|█████▍    | 12/22 [00:03<00:02,  3.49it/s, loss=4.55, v_num=1]
Epoch 12:  55%|█████▍    | 12/22 [00:03<00:02,  3.43it/s, loss=4.55, v_num=1]
Epoch 12:  59%|█████▉    | 13/22 [00:03<00:02,  3.62it/s, loss=4.55, v_num=1]
Epoch 12:  59%|█████▉    | 13/22 [00:03<00:02,  3.56it/s, loss=4.55, v_num=1]
Epoch 12:  64%|██████▎   | 14/22 [00:03<00:02,  3.75it/s, loss=4.55, v_num=1]
Epoch 12:  64%|██████▎   | 14/22 [00:03<00:02,  3.69it/s, loss=4.54, v_num=1]
Epoch 12:  68%|██████▊   | 15/22 [00:03<00:01,  3.86it/s, loss=4.54, v_num=1]
Epoch 12:  68%|██████▊   | 15/22 [00:03<00:01,  3.80it/s, loss=4.54, v_num=1]
Epoch 12:  73%|███████▎  | 16/22 [00:04<00:01,  3.97it/s, loss=4.54, v_num=1]
Epoch 12:  73%|███████▎  | 16/22 [00:04<00:01,  3.91it/s, loss=4.55, v_num=1]
Epoch 12:  77%|███████▋  | 17/22 [00:04<00:01,  4.06it/s, loss=4.55, v_num=1]
Epoch 12:  77%|███████▋  | 17/22 [00:04<00:01,  4.00it/s, loss=4.55, v_num=1]
Epoch 12:  82%|████████▏ | 18/22 [00:04<00:00,  4.15it/s, loss=4.55, v_num=1]
Epoch 12:  82%|████████▏ | 18/22 [00:04<00:00,  4.09it/s, loss=4.55, v_num=1]
Epoch 12:  86%|████████▋ | 19/22 [00:04<00:00,  4.23it/s, loss=4.55, v_num=1]
Epoch 12:  86%|████████▋ | 19/22 [00:04<00:00,  4.18it/s, loss=4.55, v_num=1]
Epoch 12:  91%|█████████ | 20/22 [00:04<00:00,  4.31it/s, loss=4.55, v_num=1]
Epoch 12:  91%|█████████ | 20/22 [00:04<00:00,  4.26it/s, loss=4.55, v_num=1]
Epoch 12:  95%|█████████▌| 21/22 [00:04<00:00,  4.39it/s, loss=4.55, v_num=1]
Epoch 12:  95%|█████████▌| 21/22 [00:04<00:00,  4.33it/s, loss=4.55, v_num=1]
Epoch 12: 100%|██████████| 22/22 [00:04<00:00,  4.46it/s, loss=4.55, v_num=1]
Epoch 12: 100%|██████████| 22/22 [00:04<00:00,  4.40it/s, loss=4.55, v_num=1]
Epoch 12: 100%|██████████| 22/22 [00:04<00:00,  4.40it/s, loss=4.55, v_num=1]
Epoch 12:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.55, v_num=1]
Epoch 13:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.55, v_num=1]
Epoch 13:   5%|▍         | 1/22 [00:01<00:36,  1.72s/it, loss=4.55, v_num=1]
Epoch 13:   5%|▍         | 1/22 [00:01<00:37,  1.78s/it, loss=4.55, v_num=1]
Epoch 13:   9%|▉         | 2/22 [00:01<00:18,  1.07it/s, loss=4.55, v_num=1]
Epoch 13:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, loss=4.55, v_num=1]
Epoch 13:  14%|█▎        | 3/22 [00:02<00:12,  1.48it/s, loss=4.55, v_num=1]
Epoch 13:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, loss=4.54, v_num=1]
Epoch 13:  18%|█▊        | 4/22 [00:02<00:09,  1.83it/s, loss=4.54, v_num=1]
Epoch 13:  18%|█▊        | 4/22 [00:02<00:10,  1.78it/s, loss=4.54, v_num=1]
Epoch 13:  23%|██▎       | 5/22 [00:02<00:07,  2.14it/s, loss=4.54, v_num=1]
Epoch 13:  23%|██▎       | 5/22 [00:02<00:08,  2.08it/s, loss=4.54, v_num=1]
Epoch 13:  27%|██▋       | 6/22 [00:02<00:06,  2.39it/s, loss=4.54, v_num=1]
Epoch 13:  27%|██▋       | 6/22 [00:02<00:06,  2.34it/s, loss=4.54, v_num=1]
Epoch 13:  32%|███▏      | 7/22 [00:02<00:05,  2.64it/s, loss=4.54, v_num=1]
Epoch 13:  32%|███▏      | 7/22 [00:02<00:05,  2.57it/s, loss=4.54, v_num=1]
Epoch 13:  36%|███▋      | 8/22 [00:02<00:04,  2.85it/s, loss=4.54, v_num=1]
Epoch 13:  36%|███▋      | 8/22 [00:02<00:05,  2.79it/s, loss=4.54, v_num=1]
Epoch 13:  41%|████      | 9/22 [00:02<00:04,  3.04it/s, loss=4.54, v_num=1]
Epoch 13:  41%|████      | 9/22 [00:03<00:04,  2.98it/s, loss=4.54, v_num=1]
Epoch 13:  45%|████▌     | 10/22 [00:03<00:03,  3.21it/s, loss=4.54, v_num=1]
Epoch 13:  45%|████▌     | 10/22 [00:03<00:03,  3.15it/s, loss=4.54, v_num=1]
Epoch 13:  50%|█████     | 11/22 [00:03<00:03,  3.37it/s, loss=4.54, v_num=1]
Epoch 13:  50%|█████     | 11/22 [00:03<00:03,  3.31it/s, loss=4.54, v_num=1]
Epoch 13:  55%|█████▍    | 12/22 [00:03<00:02,  3.52it/s, loss=4.54, v_num=1]
Epoch 13:  55%|█████▍    | 12/22 [00:03<00:02,  3.45it/s, loss=4.54, v_num=1]
Epoch 13:  59%|█████▉    | 13/22 [00:03<00:02,  3.64it/s, loss=4.54, v_num=1]
Epoch 13:  59%|█████▉    | 13/22 [00:03<00:02,  3.58it/s, loss=4.54, v_num=1]
Epoch 13:  64%|██████▎   | 14/22 [00:03<00:02,  3.77it/s, loss=4.54, v_num=1]
Epoch 13:  64%|██████▎   | 14/22 [00:03<00:02,  3.71it/s, loss=4.54, v_num=1]
Epoch 13:  68%|██████▊   | 15/22 [00:03<00:01,  3.88it/s, loss=4.54, v_num=1]
Epoch 13:  68%|██████▊   | 15/22 [00:03<00:01,  3.82it/s, loss=4.54, v_num=1]
Epoch 13:  73%|███████▎  | 16/22 [00:04<00:01,  3.98it/s, loss=4.54, v_num=1]
Epoch 13:  73%|███████▎  | 16/22 [00:04<00:01,  3.92it/s, loss=4.54, v_num=1]
Epoch 13:  77%|███████▋  | 17/22 [00:04<00:01,  4.08it/s, loss=4.54, v_num=1]
Epoch 13:  77%|███████▋  | 17/22 [00:04<00:01,  4.02it/s, loss=4.54, v_num=1]
Epoch 13:  82%|████████▏ | 18/22 [00:04<00:00,  4.17it/s, loss=4.54, v_num=1]
Epoch 13:  82%|████████▏ | 18/22 [00:04<00:00,  4.11it/s, loss=4.54, v_num=1]
Epoch 13:  86%|████████▋ | 19/22 [00:04<00:00,  4.25it/s, loss=4.54, v_num=1]
Epoch 13:  86%|████████▋ | 19/22 [00:04<00:00,  4.19it/s, loss=4.54, v_num=1]
Epoch 13:  91%|█████████ | 20/22 [00:04<00:00,  4.33it/s, loss=4.54, v_num=1]
Epoch 13:  91%|█████████ | 20/22 [00:04<00:00,  4.27it/s, loss=4.54, v_num=1]
Epoch 13:  95%|█████████▌| 21/22 [00:04<00:00,  4.40it/s, loss=4.54, v_num=1]
Epoch 13:  95%|█████████▌| 21/22 [00:04<00:00,  4.34it/s, loss=4.54, v_num=1]
Epoch 13: 100%|██████████| 22/22 [00:04<00:00,  4.47it/s, loss=4.54, v_num=1]
Epoch 13: 100%|██████████| 22/22 [00:04<00:00,  4.41it/s, loss=4.54, v_num=1]
Epoch 13: 100%|██████████| 22/22 [00:04<00:00,  4.41it/s, loss=4.54, v_num=1]
Epoch 13:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 14:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 14:   5%|▍         | 1/22 [00:01<00:36,  1.73s/it, loss=4.54, v_num=1]
Epoch 14:   5%|▍         | 1/22 [00:01<00:37,  1.79s/it, loss=4.54, v_num=1]
Epoch 14:   9%|▉         | 2/22 [00:01<00:18,  1.07it/s, loss=4.54, v_num=1]
Epoch 14:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, loss=4.54, v_num=1]
Epoch 14:  14%|█▎        | 3/22 [00:02<00:12,  1.47it/s, loss=4.54, v_num=1]
Epoch 14:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, loss=4.54, v_num=1]
Epoch 14:  18%|█▊        | 4/22 [00:02<00:09,  1.83it/s, loss=4.54, v_num=1]
Epoch 14:  18%|█▊        | 4/22 [00:02<00:10,  1.78it/s, loss=4.54, v_num=1]
Epoch 14:  23%|██▎       | 5/22 [00:02<00:07,  2.13it/s, loss=4.54, v_num=1]
Epoch 14:  23%|██▎       | 5/22 [00:02<00:08,  2.08it/s, loss=4.54, v_num=1]
Epoch 14:  27%|██▋       | 6/22 [00:02<00:06,  2.39it/s, loss=4.54, v_num=1]
Epoch 14:  27%|██▋       | 6/22 [00:02<00:06,  2.33it/s, loss=4.54, v_num=1]
Epoch 14:  32%|███▏      | 7/22 [00:02<00:05,  2.63it/s, loss=4.54, v_num=1]
Epoch 14:  32%|███▏      | 7/22 [00:02<00:05,  2.57it/s, loss=4.54, v_num=1]
Epoch 14:  36%|███▋      | 8/22 [00:02<00:04,  2.84it/s, loss=4.54, v_num=1]
Epoch 14:  36%|███▋      | 8/22 [00:02<00:05,  2.78it/s, loss=4.54, v_num=1]
Epoch 14:  41%|████      | 9/22 [00:02<00:04,  3.03it/s, loss=4.54, v_num=1]
Epoch 14:  41%|████      | 9/22 [00:03<00:04,  2.97it/s, loss=4.54, v_num=1]
Epoch 14:  45%|████▌     | 10/22 [00:03<00:03,  3.20it/s, loss=4.54, v_num=1]
Epoch 14:  45%|████▌     | 10/22 [00:03<00:03,  3.14it/s, loss=4.54, v_num=1]
Epoch 14:  50%|█████     | 11/22 [00:03<00:03,  3.36it/s, loss=4.54, v_num=1]
Epoch 14:  50%|█████     | 11/22 [00:03<00:03,  3.30it/s, loss=4.54, v_num=1]
Epoch 14:  55%|█████▍    | 12/22 [00:03<00:02,  3.51it/s, loss=4.54, v_num=1]
Epoch 14:  55%|█████▍    | 12/22 [00:03<00:02,  3.44it/s, loss=4.54, v_num=1]
Epoch 14:  59%|█████▉    | 13/22 [00:03<00:02,  3.64it/s, loss=4.54, v_num=1]
Epoch 14:  59%|█████▉    | 13/22 [00:03<00:02,  3.57it/s, loss=4.54, v_num=1]
Epoch 14:  64%|██████▎   | 14/22 [00:03<00:02,  3.76it/s, loss=4.54, v_num=1]
Epoch 14:  64%|██████▎   | 14/22 [00:03<00:02,  3.70it/s, loss=4.54, v_num=1]
Epoch 14:  68%|██████▊   | 15/22 [00:03<00:01,  3.87it/s, loss=4.54, v_num=1]
Epoch 14:  68%|██████▊   | 15/22 [00:03<00:01,  3.81it/s, loss=4.54, v_num=1]
Epoch 14:  73%|███████▎  | 16/22 [00:04<00:01,  3.97it/s, loss=4.54, v_num=1]
Epoch 14:  73%|███████▎  | 16/22 [00:04<00:01,  3.92it/s, loss=4.54, v_num=1]
Epoch 14:  77%|███████▋  | 17/22 [00:04<00:01,  4.07it/s, loss=4.54, v_num=1]
Epoch 14:  77%|███████▋  | 17/22 [00:04<00:01,  4.01it/s, loss=4.53, v_num=1]
Epoch 14:  82%|████████▏ | 18/22 [00:04<00:00,  4.16it/s, loss=4.53, v_num=1]
Epoch 14:  82%|████████▏ | 18/22 [00:04<00:00,  4.10it/s, loss=4.53, v_num=1]
Epoch 14:  86%|████████▋ | 19/22 [00:04<00:00,  4.24it/s, loss=4.53, v_num=1]
Epoch 14:  86%|████████▋ | 19/22 [00:04<00:00,  4.19it/s, loss=4.54, v_num=1]
Epoch 14:  91%|█████████ | 20/22 [00:04<00:00,  4.32it/s, loss=4.54, v_num=1]
Epoch 14:  91%|█████████ | 20/22 [00:04<00:00,  4.26it/s, loss=4.53, v_num=1]
Epoch 14:  95%|█████████▌| 21/22 [00:04<00:00,  4.40it/s, loss=4.53, v_num=1]
Epoch 14:  95%|█████████▌| 21/22 [00:04<00:00,  4.34it/s, loss=4.54, v_num=1]
Epoch 14: 100%|██████████| 22/22 [00:04<00:00,  4.47it/s, loss=4.54, v_num=1]
Epoch 14: 100%|██████████| 22/22 [00:04<00:00,  4.41it/s, loss=4.53, v_num=1]
Epoch 14: 100%|██████████| 22/22 [00:04<00:00,  4.41it/s, loss=4.53, v_num=1]
Epoch 14:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 15:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 15:   5%|▍         | 1/22 [00:01<00:36,  1.76s/it, loss=4.53, v_num=1]
Epoch 15:   5%|▍         | 1/22 [00:01<00:38,  1.82s/it, loss=4.54, v_num=1]
Epoch 15:   9%|▉         | 2/22 [00:01<00:19,  1.05it/s, loss=4.54, v_num=1]
Epoch 15:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, loss=4.54, v_num=1]
Epoch 15:  14%|█▎        | 3/22 [00:02<00:13,  1.46it/s, loss=4.54, v_num=1]
Epoch 15:  14%|█▎        | 3/22 [00:02<00:13,  1.41it/s, loss=4.54, v_num=1]
Epoch 15:  18%|█▊        | 4/22 [00:02<00:09,  1.81it/s, loss=4.54, v_num=1]
Epoch 15:  18%|█▊        | 4/22 [00:02<00:10,  1.76it/s, loss=4.54, v_num=1]
Epoch 15:  23%|██▎       | 5/22 [00:02<00:08,  2.11it/s, loss=4.54, v_num=1]
Epoch 15:  23%|██▎       | 5/22 [00:02<00:08,  2.06it/s, loss=4.54, v_num=1]
Epoch 15:  27%|██▋       | 6/22 [00:02<00:06,  2.37it/s, loss=4.54, v_num=1]
Epoch 15:  27%|██▋       | 6/22 [00:02<00:06,  2.32it/s, loss=4.54, v_num=1]
Epoch 15:  32%|███▏      | 7/22 [00:02<00:05,  2.62it/s, loss=4.54, v_num=1]
Epoch 15:  32%|███▏      | 7/22 [00:02<00:05,  2.55it/s, loss=4.54, v_num=1]
Epoch 15:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, loss=4.54, v_num=1]
Epoch 15:  36%|███▋      | 8/22 [00:02<00:05,  2.77it/s, loss=4.54, v_num=1]
Epoch 15:  41%|████      | 9/22 [00:02<00:04,  3.02it/s, loss=4.54, v_num=1]
Epoch 15:  41%|████      | 9/22 [00:03<00:04,  2.96it/s, loss=4.54, v_num=1]
Epoch 15:  45%|████▌     | 10/22 [00:03<00:03,  3.19it/s, loss=4.54, v_num=1]
Epoch 15:  45%|████▌     | 10/22 [00:03<00:03,  3.13it/s, loss=4.54, v_num=1]
Epoch 15:  50%|█████     | 11/22 [00:03<00:03,  3.35it/s, loss=4.54, v_num=1]
Epoch 15:  50%|█████     | 11/22 [00:03<00:03,  3.29it/s, loss=4.53, v_num=1]
Epoch 15:  55%|█████▍    | 12/22 [00:03<00:02,  3.49it/s, loss=4.53, v_num=1]
Epoch 15:  55%|█████▍    | 12/22 [00:03<00:02,  3.43it/s, loss=4.53, v_num=1]
Epoch 15:  59%|█████▉    | 13/22 [00:03<00:02,  3.63it/s, loss=4.53, v_num=1]
Epoch 15:  59%|█████▉    | 13/22 [00:03<00:02,  3.56it/s, loss=4.53, v_num=1]
Epoch 15:  64%|██████▎   | 14/22 [00:03<00:02,  3.75it/s, loss=4.53, v_num=1]
Epoch 15:  64%|██████▎   | 14/22 [00:03<00:02,  3.69it/s, loss=4.53, v_num=1]
Epoch 15:  68%|██████▊   | 15/22 [00:03<00:01,  3.86it/s, loss=4.53, v_num=1]
Epoch 15:  68%|██████▊   | 15/22 [00:03<00:01,  3.80it/s, loss=4.53, v_num=1]
Epoch 15:  73%|███████▎  | 16/22 [00:04<00:01,  3.96it/s, loss=4.53, v_num=1]
Epoch 15:  73%|███████▎  | 16/22 [00:04<00:01,  3.91it/s, loss=4.53, v_num=1]
Epoch 15:  77%|███████▋  | 17/22 [00:04<00:01,  4.06it/s, loss=4.53, v_num=1]
Epoch 15:  77%|███████▋  | 17/22 [00:04<00:01,  4.00it/s, loss=4.53, v_num=1]
Epoch 15:  82%|████████▏ | 18/22 [00:04<00:00,  4.15it/s, loss=4.53, v_num=1]
Epoch 15:  82%|████████▏ | 18/22 [00:04<00:00,  4.09it/s, loss=4.53, v_num=1]
Epoch 15:  86%|████████▋ | 19/22 [00:04<00:00,  4.24it/s, loss=4.53, v_num=1]
Epoch 15:  86%|████████▋ | 19/22 [00:04<00:00,  4.17it/s, loss=4.53, v_num=1]
Epoch 15:  91%|█████████ | 20/22 [00:04<00:00,  4.31it/s, loss=4.53, v_num=1]
Epoch 15:  91%|█████████ | 20/22 [00:04<00:00,  4.25it/s, loss=4.53, v_num=1]
Epoch 15:  95%|█████████▌| 21/22 [00:04<00:00,  4.39it/s, loss=4.53, v_num=1]
Epoch 15:  95%|█████████▌| 21/22 [00:04<00:00,  4.33it/s, loss=4.53, v_num=1]
Epoch 15: 100%|██████████| 22/22 [00:04<00:00,  4.45it/s, loss=4.53, v_num=1]
Epoch 15: 100%|██████████| 22/22 [00:05<00:00,  4.40it/s, loss=4.53, v_num=1]
Epoch 15: 100%|██████████| 22/22 [00:05<00:00,  4.40it/s, loss=4.53, v_num=1]
Epoch 15:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 16:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 16:   5%|▍         | 1/22 [00:01<00:35,  1.69s/it, loss=4.53, v_num=1]
Epoch 16:   5%|▍         | 1/22 [00:01<00:36,  1.75s/it, loss=4.53, v_num=1]
Epoch 16:   9%|▉         | 2/22 [00:01<00:18,  1.09it/s, loss=4.53, v_num=1]
Epoch 16:   9%|▉         | 2/22 [00:01<00:19,  1.05it/s, loss=4.53, v_num=1]
Epoch 16:  14%|█▎        | 3/22 [00:01<00:12,  1.51it/s, loss=4.53, v_num=1]
Epoch 16:  14%|█▎        | 3/22 [00:02<00:13,  1.46it/s, loss=4.53, v_num=1]
Epoch 16:  18%|█▊        | 4/22 [00:02<00:09,  1.86it/s, loss=4.53, v_num=1]
Epoch 16:  18%|█▊        | 4/22 [00:02<00:09,  1.81it/s, loss=4.53, v_num=1]
Epoch 16:  23%|██▎       | 5/22 [00:02<00:07,  2.18it/s, loss=4.53, v_num=1]
Epoch 16:  23%|██▎       | 5/22 [00:02<00:08,  2.12it/s, loss=4.53, v_num=1]
Epoch 16:  27%|██▋       | 6/22 [00:02<00:06,  2.44it/s, loss=4.53, v_num=1]
Epoch 16:  27%|██▋       | 6/22 [00:02<00:06,  2.38it/s, loss=4.53, v_num=1]
Epoch 16:  32%|███▏      | 7/22 [00:02<00:05,  2.68it/s, loss=4.53, v_num=1]
Epoch 16:  32%|███▏      | 7/22 [00:02<00:05,  2.62it/s, loss=4.53, v_num=1]
Epoch 16:  36%|███▋      | 8/22 [00:02<00:04,  2.89it/s, loss=4.53, v_num=1]
Epoch 16:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, loss=4.53, v_num=1]
Epoch 16:  41%|████      | 9/22 [00:02<00:04,  3.08it/s, loss=4.53, v_num=1]
Epoch 16:  41%|████      | 9/22 [00:02<00:04,  3.02it/s, loss=4.53, v_num=1]
Epoch 16:  45%|████▌     | 10/22 [00:03<00:03,  3.26it/s, loss=4.53, v_num=1]
Epoch 16:  45%|████▌     | 10/22 [00:03<00:03,  3.20it/s, loss=4.53, v_num=1]
Epoch 16:  50%|█████     | 11/22 [00:03<00:03,  3.41it/s, loss=4.53, v_num=1]
Epoch 16:  50%|█████     | 11/22 [00:03<00:03,  3.35it/s, loss=4.53, v_num=1]
Epoch 16:  55%|█████▍    | 12/22 [00:03<00:02,  3.56it/s, loss=4.53, v_num=1]
Epoch 16:  55%|█████▍    | 12/22 [00:03<00:02,  3.50it/s, loss=4.54, v_num=1]
Epoch 16:  59%|█████▉    | 13/22 [00:03<00:02,  3.69it/s, loss=4.54, v_num=1]
Epoch 16:  59%|█████▉    | 13/22 [00:03<00:02,  3.63it/s, loss=4.54, v_num=1]
Epoch 16:  64%|██████▎   | 14/22 [00:03<00:02,  3.81it/s, loss=4.54, v_num=1]
Epoch 16:  64%|██████▎   | 14/22 [00:03<00:02,  3.75it/s, loss=4.53, v_num=1]
Epoch 16:  68%|██████▊   | 15/22 [00:03<00:01,  3.93it/s, loss=4.53, v_num=1]
Epoch 16:  68%|██████▊   | 15/22 [00:03<00:01,  3.86it/s, loss=4.54, v_num=1]
Epoch 16:  73%|███████▎  | 16/22 [00:03<00:01,  4.03it/s, loss=4.54, v_num=1]
Epoch 16:  73%|███████▎  | 16/22 [00:04<00:01,  3.97it/s, loss=4.54, v_num=1]
Epoch 16:  77%|███████▋  | 17/22 [00:04<00:01,  4.13it/s, loss=4.54, v_num=1]
Epoch 16:  77%|███████▋  | 17/22 [00:04<00:01,  4.06it/s, loss=4.54, v_num=1]
Epoch 16:  82%|████████▏ | 18/22 [00:04<00:00,  4.22it/s, loss=4.54, v_num=1]
Epoch 16:  82%|████████▏ | 18/22 [00:04<00:00,  4.15it/s, loss=4.54, v_num=1]
Epoch 16:  86%|████████▋ | 19/22 [00:04<00:00,  4.29it/s, loss=4.54, v_num=1]
Epoch 16:  86%|████████▋ | 19/22 [00:04<00:00,  4.24it/s, loss=4.54, v_num=1]
Epoch 16:  91%|█████████ | 20/22 [00:04<00:00,  4.37it/s, loss=4.54, v_num=1]
Epoch 16:  91%|█████████ | 20/22 [00:04<00:00,  4.31it/s, loss=4.54, v_num=1]
Epoch 16:  95%|█████████▌| 21/22 [00:04<00:00,  4.45it/s, loss=4.54, v_num=1]
Epoch 16:  95%|█████████▌| 21/22 [00:04<00:00,  4.39it/s, loss=4.54, v_num=1]
Epoch 16: 100%|██████████| 22/22 [00:04<00:00,  4.51it/s, loss=4.54, v_num=1]
Epoch 16: 100%|██████████| 22/22 [00:04<00:00,  4.46it/s, loss=4.53, v_num=1]
Epoch 16: 100%|██████████| 22/22 [00:04<00:00,  4.46it/s, loss=4.53, v_num=1]
Epoch 16:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 17:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 17:   5%|▍         | 1/22 [00:01<00:36,  1.73s/it, loss=4.53, v_num=1]
Epoch 17:   5%|▍         | 1/22 [00:01<00:37,  1.79s/it, loss=4.53, v_num=1]
Epoch 17:   9%|▉         | 2/22 [00:01<00:18,  1.06it/s, loss=4.53, v_num=1]
Epoch 17:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, loss=4.53, v_num=1]
Epoch 17:  14%|█▎        | 3/22 [00:02<00:12,  1.47it/s, loss=4.53, v_num=1]
Epoch 17:  14%|█▎        | 3/22 [00:02<00:13,  1.42it/s, loss=4.53, v_num=1]
Epoch 17:  18%|█▊        | 4/22 [00:02<00:09,  1.82it/s, loss=4.53, v_num=1]
Epoch 17:  18%|█▊        | 4/22 [00:02<00:10,  1.77it/s, loss=4.54, v_num=1]
Epoch 17:  23%|██▎       | 5/22 [00:02<00:07,  2.13it/s, loss=4.54, v_num=1]
Epoch 17:  23%|██▎       | 5/22 [00:02<00:08,  2.07it/s, loss=4.54, v_num=1]
Epoch 17:  27%|██▋       | 6/22 [00:02<00:06,  2.39it/s, loss=4.54, v_num=1]
Epoch 17:  27%|██▋       | 6/22 [00:02<00:06,  2.33it/s, loss=4.53, v_num=1]
Epoch 17:  32%|███▏      | 7/22 [00:02<00:05,  2.63it/s, loss=4.53, v_num=1]
Epoch 17:  32%|███▏      | 7/22 [00:02<00:05,  2.57it/s, loss=4.53, v_num=1]
Epoch 17:  36%|███▋      | 8/22 [00:02<00:04,  2.85it/s, loss=4.53, v_num=1]
Epoch 17:  36%|███▋      | 8/22 [00:02<00:05,  2.78it/s, loss=4.53, v_num=1]
Epoch 17:  41%|████      | 9/22 [00:02<00:04,  3.04it/s, loss=4.53, v_num=1]
Epoch 17:  41%|████      | 9/22 [00:03<00:04,  2.97it/s, loss=4.53, v_num=1]
Epoch 17:  45%|████▌     | 10/22 [00:03<00:03,  3.21it/s, loss=4.53, v_num=1]
Epoch 17:  45%|████▌     | 10/22 [00:03<00:03,  3.15it/s, loss=4.53, v_num=1]
Epoch 17:  50%|█████     | 11/22 [00:03<00:03,  3.36it/s, loss=4.53, v_num=1]
Epoch 17:  50%|█████     | 11/22 [00:03<00:03,  3.30it/s, loss=4.53, v_num=1]
Epoch 17:  55%|█████▍    | 12/22 [00:03<00:02,  3.51it/s, loss=4.53, v_num=1]
Epoch 17:  55%|█████▍    | 12/22 [00:03<00:02,  3.44it/s, loss=4.53, v_num=1]
Epoch 17:  59%|█████▉    | 13/22 [00:03<00:02,  3.64it/s, loss=4.53, v_num=1]
Epoch 17:  59%|█████▉    | 13/22 [00:03<00:02,  3.58it/s, loss=4.53, v_num=1]
Epoch 17:  64%|██████▎   | 14/22 [00:03<00:02,  3.76it/s, loss=4.53, v_num=1]
Epoch 17:  64%|██████▎   | 14/22 [00:03<00:02,  3.70it/s, loss=4.53, v_num=1]
Epoch 17:  68%|██████▊   | 15/22 [00:03<00:01,  3.87it/s, loss=4.53, v_num=1]
Epoch 17:  68%|██████▊   | 15/22 [00:03<00:01,  3.81it/s, loss=4.53, v_num=1]
Epoch 17:  73%|███████▎  | 16/22 [00:04<00:01,  3.98it/s, loss=4.53, v_num=1]
Epoch 17:  73%|███████▎  | 16/22 [00:04<00:01,  3.92it/s, loss=4.53, v_num=1]
Epoch 17:  77%|███████▋  | 17/22 [00:04<00:01,  4.07it/s, loss=4.53, v_num=1]
Epoch 17:  77%|███████▋  | 17/22 [00:04<00:01,  4.02it/s, loss=4.53, v_num=1]
Epoch 17:  82%|████████▏ | 18/22 [00:04<00:00,  4.17it/s, loss=4.53, v_num=1]
Epoch 17:  82%|████████▏ | 18/22 [00:04<00:00,  4.11it/s, loss=4.53, v_num=1]
Epoch 17:  86%|████████▋ | 19/22 [00:04<00:00,  4.25it/s, loss=4.53, v_num=1]
Epoch 17:  86%|████████▋ | 19/22 [00:04<00:00,  4.19it/s, loss=4.53, v_num=1]
Epoch 17:  91%|█████████ | 20/22 [00:04<00:00,  4.32it/s, loss=4.53, v_num=1]
Epoch 17:  91%|█████████ | 20/22 [00:04<00:00,  4.27it/s, loss=4.53, v_num=1]
Epoch 17:  95%|█████████▌| 21/22 [00:04<00:00,  4.40it/s, loss=4.53, v_num=1]
Epoch 17:  95%|█████████▌| 21/22 [00:04<00:00,  4.34it/s, loss=4.53, v_num=1]
Epoch 17: 100%|██████████| 22/22 [00:04<00:00,  4.47it/s, loss=4.53, v_num=1]
Epoch 17: 100%|██████████| 22/22 [00:04<00:00,  4.41it/s, loss=4.53, v_num=1]
Epoch 17: 100%|██████████| 22/22 [00:04<00:00,  4.41it/s, loss=4.53, v_num=1]
Epoch 17:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 18:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 18:   5%|▍         | 1/22 [00:01<00:35,  1.71s/it, loss=4.53, v_num=1]
Epoch 18:   5%|▍         | 1/22 [00:01<00:37,  1.77s/it, loss=4.53, v_num=1]
Epoch 18:   9%|▉         | 2/22 [00:01<00:18,  1.07it/s, loss=4.53, v_num=1]
Epoch 18:   9%|▉         | 2/22 [00:01<00:19,  1.04it/s, loss=4.53, v_num=1]
Epoch 18:  14%|█▎        | 3/22 [00:02<00:12,  1.49it/s, loss=4.53, v_num=1]
Epoch 18:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, loss=4.53, v_num=1]
Epoch 18:  18%|█▊        | 4/22 [00:02<00:09,  1.84it/s, loss=4.53, v_num=1]
Epoch 18:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, loss=4.53, v_num=1]
Epoch 18:  23%|██▎       | 5/22 [00:02<00:07,  2.15it/s, loss=4.53, v_num=1]
Epoch 18:  23%|██▎       | 5/22 [00:02<00:08,  2.09it/s, loss=4.53, v_num=1]
Epoch 18:  27%|██▋       | 6/22 [00:02<00:06,  2.41it/s, loss=4.53, v_num=1]
Epoch 18:  27%|██▋       | 6/22 [00:02<00:06,  2.35it/s, loss=4.53, v_num=1]
Epoch 18:  32%|███▏      | 7/22 [00:02<00:05,  2.65it/s, loss=4.53, v_num=1]
Epoch 18:  32%|███▏      | 7/22 [00:02<00:05,  2.59it/s, loss=4.53, v_num=1]
Epoch 18:  36%|███▋      | 8/22 [00:02<00:04,  2.86it/s, loss=4.53, v_num=1]
Epoch 18:  36%|███▋      | 8/22 [00:02<00:04,  2.80it/s, loss=4.53, v_num=1]
Epoch 18:  41%|████      | 9/22 [00:02<00:04,  3.05it/s, loss=4.53, v_num=1]
Epoch 18:  41%|████      | 9/22 [00:03<00:04,  2.99it/s, loss=4.53, v_num=1]
Epoch 18:  45%|████▌     | 10/22 [00:03<00:03,  3.23it/s, loss=4.53, v_num=1]
Epoch 18:  45%|████▌     | 10/22 [00:03<00:03,  3.16it/s, loss=4.53, v_num=1]
Epoch 18:  50%|█████     | 11/22 [00:03<00:03,  3.38it/s, loss=4.53, v_num=1]
Epoch 18:  50%|█████     | 11/22 [00:03<00:03,  3.32it/s, loss=4.53, v_num=1]
Epoch 18:  55%|█████▍    | 12/22 [00:03<00:02,  3.53it/s, loss=4.53, v_num=1]
Epoch 18:  55%|█████▍    | 12/22 [00:03<00:02,  3.46it/s, loss=4.53, v_num=1]
Epoch 18:  59%|█████▉    | 13/22 [00:03<00:02,  3.66it/s, loss=4.53, v_num=1]
Epoch 18:  59%|█████▉    | 13/22 [00:03<00:02,  3.59it/s, loss=4.53, v_num=1]
Epoch 18:  64%|██████▎   | 14/22 [00:03<00:02,  3.78it/s, loss=4.53, v_num=1]
Epoch 18:  64%|██████▎   | 14/22 [00:03<00:02,  3.72it/s, loss=4.53, v_num=1]
Epoch 18:  68%|██████▊   | 15/22 [00:03<00:01,  3.89it/s, loss=4.53, v_num=1]
Epoch 18:  68%|██████▊   | 15/22 [00:03<00:01,  3.83it/s, loss=4.53, v_num=1]
Epoch 18:  73%|███████▎  | 16/22 [00:04<00:01,  3.99it/s, loss=4.53, v_num=1]
Epoch 18:  73%|███████▎  | 16/22 [00:04<00:01,  3.93it/s, loss=4.53, v_num=1]
Epoch 18:  77%|███████▋  | 17/22 [00:04<00:01,  4.09it/s, loss=4.53, v_num=1]
Epoch 18:  77%|███████▋  | 17/22 [00:04<00:01,  4.03it/s, loss=4.53, v_num=1]
Epoch 18:  82%|████████▏ | 18/22 [00:04<00:00,  4.17it/s, loss=4.53, v_num=1]
Epoch 18:  82%|████████▏ | 18/22 [00:04<00:00,  4.12it/s, loss=4.53, v_num=1]
Epoch 18:  86%|████████▋ | 19/22 [00:04<00:00,  4.26it/s, loss=4.53, v_num=1]
Epoch 18:  86%|████████▋ | 19/22 [00:04<00:00,  4.20it/s, loss=4.53, v_num=1]
Epoch 18:  91%|█████████ | 20/22 [00:04<00:00,  4.34it/s, loss=4.53, v_num=1]
Epoch 18:  91%|█████████ | 20/22 [00:04<00:00,  4.28it/s, loss=4.53, v_num=1]
Epoch 18:  95%|█████████▌| 21/22 [00:04<00:00,  4.41it/s, loss=4.53, v_num=1]
Epoch 18:  95%|█████████▌| 21/22 [00:04<00:00,  4.35it/s, loss=4.53, v_num=1]
Epoch 18: 100%|██████████| 22/22 [00:04<00:00,  4.48it/s, loss=4.53, v_num=1]
Epoch 18: 100%|██████████| 22/22 [00:04<00:00,  4.42it/s, loss=4.53, v_num=1]
Epoch 18: 100%|██████████| 22/22 [00:04<00:00,  4.42it/s, loss=4.53, v_num=1]
Epoch 18:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 19:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 19:   5%|▍         | 1/22 [00:01<00:36,  1.75s/it, loss=4.53, v_num=1]
Epoch 19:   5%|▍         | 1/22 [00:01<00:37,  1.81s/it, loss=4.53, v_num=1]
Epoch 19:   9%|▉         | 2/22 [00:01<00:19,  1.05it/s, loss=4.53, v_num=1]
Epoch 19:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, loss=4.53, v_num=1]
Epoch 19:  14%|█▎        | 3/22 [00:02<00:13,  1.45it/s, loss=4.53, v_num=1]
Epoch 19:  14%|█▎        | 3/22 [00:02<00:13,  1.41it/s, loss=4.53, v_num=1]
Epoch 19:  18%|█▊        | 4/22 [00:02<00:09,  1.81it/s, loss=4.53, v_num=1]
Epoch 19:  18%|█▊        | 4/22 [00:02<00:10,  1.76it/s, loss=4.53, v_num=1]
Epoch 19:  23%|██▎       | 5/22 [00:02<00:08,  2.11it/s, loss=4.53, v_num=1]
Epoch 19:  23%|██▎       | 5/22 [00:02<00:08,  2.06it/s, loss=4.53, v_num=1]
Epoch 19:  27%|██▋       | 6/22 [00:02<00:06,  2.37it/s, loss=4.53, v_num=1]
Epoch 19:  27%|██▋       | 6/22 [00:02<00:06,  2.32it/s, loss=4.53, v_num=1]
Epoch 19:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, loss=4.53, v_num=1]
Epoch 19:  32%|███▏      | 7/22 [00:02<00:05,  2.55it/s, loss=4.53, v_num=1]
Epoch 19:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, loss=4.53, v_num=1]
Epoch 19:  36%|███▋      | 8/22 [00:02<00:05,  2.77it/s, loss=4.53, v_num=1]
Epoch 19:  41%|████      | 9/22 [00:02<00:04,  3.02it/s, loss=4.53, v_num=1]
Epoch 19:  41%|████      | 9/22 [00:03<00:04,  2.95it/s, loss=4.53, v_num=1]
Epoch 19:  45%|████▌     | 10/22 [00:03<00:03,  3.19it/s, loss=4.53, v_num=1]
Epoch 19:  45%|████▌     | 10/22 [00:03<00:03,  3.13it/s, loss=4.53, v_num=1]
Epoch 19:  50%|█████     | 11/22 [00:03<00:03,  3.34it/s, loss=4.53, v_num=1]
Epoch 19:  50%|█████     | 11/22 [00:03<00:03,  3.28it/s, loss=4.53, v_num=1]
Epoch 19:  55%|█████▍    | 12/22 [00:03<00:02,  3.49it/s, loss=4.53, v_num=1]
Epoch 19:  55%|█████▍    | 12/22 [00:03<00:02,  3.43it/s, loss=4.53, v_num=1]
Epoch 19:  59%|█████▉    | 13/22 [00:03<00:02,  3.63it/s, loss=4.53, v_num=1]
Epoch 19:  59%|█████▉    | 13/22 [00:03<00:02,  3.56it/s, loss=4.53, v_num=1]
Epoch 19:  64%|██████▎   | 14/22 [00:03<00:02,  3.75it/s, loss=4.53, v_num=1]
Epoch 19:  64%|██████▎   | 14/22 [00:03<00:02,  3.69it/s, loss=4.53, v_num=1]
Epoch 19:  68%|██████▊   | 15/22 [00:03<00:01,  3.86it/s, loss=4.53, v_num=1]
Epoch 19:  68%|██████▊   | 15/22 [00:03<00:01,  3.80it/s, loss=4.53, v_num=1]
Epoch 19:  73%|███████▎  | 16/22 [00:04<00:01,  3.97it/s, loss=4.53, v_num=1]
Epoch 19:  73%|███████▎  | 16/22 [00:04<00:01,  3.91it/s, loss=4.53, v_num=1]
Epoch 19:  77%|███████▋  | 17/22 [00:04<00:01,  4.06it/s, loss=4.53, v_num=1]
Epoch 19:  77%|███████▋  | 17/22 [00:04<00:01,  4.00it/s, loss=4.53, v_num=1]
Epoch 19:  82%|████████▏ | 18/22 [00:04<00:00,  4.15it/s, loss=4.53, v_num=1]
Epoch 19:  82%|████████▏ | 18/22 [00:04<00:00,  4.09it/s, loss=4.53, v_num=1]
Epoch 19:  86%|████████▋ | 19/22 [00:04<00:00,  4.24it/s, loss=4.53, v_num=1]
Epoch 19:  86%|████████▋ | 19/22 [00:04<00:00,  4.18it/s, loss=4.52, v_num=1]
Epoch 19:  91%|█████████ | 20/22 [00:04<00:00,  4.32it/s, loss=4.52, v_num=1]
Epoch 19:  91%|█████████ | 20/22 [00:04<00:00,  4.26it/s, loss=4.53, v_num=1]
Epoch 19:  95%|█████████▌| 21/22 [00:04<00:00,  4.39it/s, loss=4.53, v_num=1]
Epoch 19:  95%|█████████▌| 21/22 [00:04<00:00,  4.33it/s, loss=4.53, v_num=1]
Epoch 19: 100%|██████████| 22/22 [00:04<00:00,  4.46it/s, loss=4.53, v_num=1]
Epoch 19: 100%|██████████| 22/22 [00:04<00:00,  4.40it/s, loss=4.53, v_num=1]
Epoch 19: 100%|██████████| 22/22 [00:04<00:00,  4.40it/s, loss=4.53, v_num=1]
Epoch 19: 100%|██████████| 22/22 [00:05<00:00,  4.26it/s, loss=4.53, v_num=1]

other example

plot_knn_examples(embeddings, filenames)
  • d=0.000, d=0.194, d=0.211
  • d=0.000, d=0.241, d=0.270
  • d=0.000, d=0.156, d=0.164
  • d=0.000, d=0.137, d=0.170
  • d=0.000, d=0.217, d=0.226
  • d=0.000, d=0.206, d=0.208

What’s next?

# You could use the pre-trained model and train a classifier on top.
pretrained_resnet_backbone = model.backbone

# you can also store the backbone and use it in another code
state_dict = {"resnet18_parameters": pretrained_resnet_backbone.state_dict()}
torch.save(state_dict, "model.pth")

THIS COULD BE IN A NEW FILE (e.g. inference.py)

Make sure you place the model.pth file in the same folder as this code

# load the model in a new file for inference
resnet18_new = torchvision.models.resnet18()

# note that we need to create exactly the same backbone in order to load the weights
backbone_new = nn.Sequential(*list(resnet18_new.children())[:-1])

ckpt = torch.load("model.pth")
backbone_new.load_state_dict(ckpt["resnet18_parameters"])
<All keys matched successfully>