Tutorial 3: Train SimCLR on Clothing

In this tutorial, we will train a SimCLR model using lightly. The model, augmentations and training procedure is from A Simple Framework for Contrastive Learning of Visual Representations.

The paper explores a rather simple training procedure for contrastive learning. Since we use the typical contrastive learning loss based on NCE the method greatly benefits from having larger batch sizes. In this example, we use a batch size of 256 and paired with the input resolution per image of 64x64 pixels and a resnet-18 model this example requires 16GB of GPU memory.

We use the clothing dataset from Alex Grigorev for this tutorial.

In this tutorial you will learn:

  • How to create a SimCLR model

  • How to generate image representations

  • How different augmentations impact the learned representations

Imports

Import the Python frameworks we need for this tutorial.

import os

import matplotlib.pyplot as plt
import numpy as np
import pytorch_lightning as pl
import torch
import torch.nn as nn
import torchvision
from PIL import Image
from sklearn.neighbors import NearestNeighbors
from sklearn.preprocessing import normalize

from lightly.data import LightlyDataset, SimCLRCollateFunction, collate

Configuration

We set some configuration parameters for our experiment. Feel free to change them and analyze the effect.

The default configuration with a batch size of 256 and input resolution of 128 requires 6GB of GPU memory.

num_workers = 8
batch_size = 256
seed = 1
max_epochs = 20
input_size = 128
num_ftrs = 32

Let’s set the seed for our experiments

pl.seed_everything(seed)
1

Make sure path_to_data points to the downloaded clothing dataset. You can download it using git clone https://github.com/alexeygrigorev/clothing-dataset.git

path_to_data = "/datasets/clothing-dataset/images"

Setup data augmentations and loaders

The images from the dataset have been taken from above when the clothing was on a table, bed or floor. Therefore, we can make use of additional augmentations such as vertical flip or random rotation (90 degrees). By adding these augmentations we learn our model invariance regarding the orientation of the clothing piece. E.g. we don’t care if a shirt is upside down but more about the strcture which make it a shirt.

You can learn more about the different augmentations and learned invariances here: Advanced Concepts in Self-Supervised Learning.

collate_fn = SimCLRCollateFunction(input_size=input_size, vf_prob=0.5, rr_prob=0.5)

# We create a torchvision transformation for embedding the dataset after
# training
test_transforms = torchvision.transforms.Compose(
    [
        torchvision.transforms.Resize((input_size, input_size)),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize(
            mean=collate.imagenet_normalize["mean"],
            std=collate.imagenet_normalize["std"],
        ),
    ]
)

dataset_train_simclr = LightlyDataset(input_dir=path_to_data)

dataset_test = LightlyDataset(input_dir=path_to_data, transform=test_transforms)

dataloader_train_simclr = torch.utils.data.DataLoader(
    dataset_train_simclr,
    batch_size=batch_size,
    shuffle=True,
    collate_fn=collate_fn,
    drop_last=True,
    num_workers=num_workers,
)

dataloader_test = torch.utils.data.DataLoader(
    dataset_test,
    batch_size=batch_size,
    shuffle=False,
    drop_last=False,
    num_workers=num_workers,
)

Create the SimCLR Model

Now we create the SimCLR model. We implement it as a PyTorch Lightning Module and use a ResNet-18 backbone from Torchvision. Lightly provides implementations of the SimCLR projection head and loss function in the SimCLRProjectionHead and NTXentLoss classes. We can simply import them and combine the building blocks in the module.

from lightly.loss import NTXentLoss
from lightly.models.modules.heads import SimCLRProjectionHead


class SimCLRModel(pl.LightningModule):
    def __init__(self):
        super().__init__()

        # create a ResNet backbone and remove the classification head
        resnet = torchvision.models.resnet18()
        self.backbone = nn.Sequential(*list(resnet.children())[:-1])

        hidden_dim = resnet.fc.in_features
        self.projection_head = SimCLRProjectionHead(hidden_dim, hidden_dim, 128)

        self.criterion = NTXentLoss()

    def forward(self, x):
        h = self.backbone(x).flatten(start_dim=1)
        z = self.projection_head(h)
        return z

    def training_step(self, batch, batch_idx):
        (x0, x1), _, _ = batch
        z0 = self.forward(x0)
        z1 = self.forward(x1)
        loss = self.criterion(z0, z1)
        self.log("train_loss_ssl", loss)
        return loss

    def configure_optimizers(self):
        optim = torch.optim.SGD(
            self.parameters(), lr=6e-2, momentum=0.9, weight_decay=5e-4
        )
        scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim, max_epochs)
        return [optim], [scheduler]

Train the module using the PyTorch Lightning Trainer on a single GPU.

model = SimCLRModel()
trainer = pl.Trainer(max_epochs=max_epochs, devices=1, accelerator="gpu")
trainer.fit(model, dataloader_train_simclr)
/opt/runner_04/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1555: PossibleUserWarning: The number of training batches (22) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
  rank_zero_warn(

Training: 0it [00:00, ?it/s]
Training:   0%|          | 0/22 [00:00<?, ?it/s]
Epoch 0:   0%|          | 0/22 [00:00<?, ?it/s]
Epoch 0:   5%|4         | 1/22 [00:04<01:32,  4.40s/it]
Epoch 0:   5%|4         | 1/22 [00:04<01:33,  4.46s/it, loss=6.21, v_num=0]
Epoch 0:   9%|9         | 2/22 [00:04<00:45,  2.28s/it, loss=6.21, v_num=0]
Epoch 0:   9%|9         | 2/22 [00:04<00:46,  2.31s/it, loss=6.2, v_num=0]
Epoch 0:  14%|#3        | 3/22 [00:04<00:30,  1.58s/it, loss=6.2, v_num=0]
Epoch 0:  14%|#3        | 3/22 [00:04<00:30,  1.60s/it, loss=6.19, v_num=0]
Epoch 0:  18%|#8        | 4/22 [00:04<00:21,  1.22s/it, loss=6.19, v_num=0]
Epoch 0:  18%|#8        | 4/22 [00:04<00:22,  1.23s/it, loss=6.16, v_num=0]
Epoch 0:  23%|##2       | 5/22 [00:05<00:17,  1.01s/it, loss=6.16, v_num=0]
Epoch 0:  23%|##2       | 5/22 [00:05<00:17,  1.02s/it, loss=6.12, v_num=0]
Epoch 0:  27%|##7       | 6/22 [00:05<00:13,  1.16it/s, loss=6.12, v_num=0]
Epoch 0:  27%|##7       | 6/22 [00:05<00:13,  1.15it/s, loss=6.09, v_num=0]
Epoch 0:  32%|###1      | 7/22 [00:05<00:11,  1.31it/s, loss=6.09, v_num=0]
Epoch 0:  32%|###1      | 7/22 [00:05<00:11,  1.30it/s, loss=6.05, v_num=0]
Epoch 0:  36%|###6      | 8/22 [00:05<00:09,  1.46it/s, loss=6.05, v_num=0]
Epoch 0:  36%|###6      | 8/22 [00:05<00:09,  1.44it/s, loss=6.01, v_num=0]
Epoch 0:  41%|####      | 9/22 [00:05<00:08,  1.58it/s, loss=6.01, v_num=0]
Epoch 0:  41%|####      | 9/22 [00:05<00:08,  1.57it/s, loss=5.99, v_num=0]
Epoch 0:  45%|####5     | 10/22 [00:05<00:06,  1.72it/s, loss=5.99, v_num=0]
Epoch 0:  45%|####5     | 10/22 [00:05<00:07,  1.70it/s, loss=5.96, v_num=0]
Epoch 0:  50%|#####     | 11/22 [00:05<00:05,  1.84it/s, loss=5.96, v_num=0]
Epoch 0:  50%|#####     | 11/22 [00:06<00:06,  1.82it/s, loss=5.95, v_num=0]
Epoch 0:  55%|#####4    | 12/22 [00:06<00:05,  1.96it/s, loss=5.95, v_num=0]
Epoch 0:  55%|#####4    | 12/22 [00:06<00:05,  1.94it/s, loss=5.92, v_num=0]
Epoch 0:  59%|#####9    | 13/22 [00:06<00:04,  2.07it/s, loss=5.92, v_num=0]
Epoch 0:  59%|#####9    | 13/22 [00:06<00:04,  2.05it/s, loss=5.9, v_num=0]
Epoch 0:  64%|######3   | 14/22 [00:06<00:03,  2.18it/s, loss=5.9, v_num=0]
Epoch 0:  64%|######3   | 14/22 [00:06<00:03,  2.16it/s, loss=5.87, v_num=0]
Epoch 0:  68%|######8   | 15/22 [00:06<00:03,  2.28it/s, loss=5.87, v_num=0]
Epoch 0:  68%|######8   | 15/22 [00:06<00:03,  2.26it/s, loss=5.86, v_num=0]
Epoch 0:  73%|#######2  | 16/22 [00:06<00:02,  2.38it/s, loss=5.86, v_num=0]
Epoch 0:  73%|#######2  | 16/22 [00:06<00:02,  2.36it/s, loss=5.84, v_num=0]
Epoch 0:  77%|#######7  | 17/22 [00:06<00:02,  2.48it/s, loss=5.84, v_num=0]
Epoch 0:  77%|#######7  | 17/22 [00:06<00:02,  2.46it/s, loss=5.82, v_num=0]
Epoch 0:  82%|########1 | 18/22 [00:07<00:01,  2.57it/s, loss=5.82, v_num=0]
Epoch 0:  82%|########1 | 18/22 [00:07<00:01,  2.55it/s, loss=5.81, v_num=0]
Epoch 0:  86%|########6 | 19/22 [00:07<00:01,  2.66it/s, loss=5.81, v_num=0]
Epoch 0:  86%|########6 | 19/22 [00:07<00:01,  2.63it/s, loss=5.8, v_num=0]
Epoch 0:  91%|######### | 20/22 [00:07<00:00,  2.74it/s, loss=5.8, v_num=0]
Epoch 0:  91%|######### | 20/22 [00:07<00:00,  2.72it/s, loss=5.79, v_num=0]
Epoch 0:  95%|#########5| 21/22 [00:07<00:00,  2.82it/s, loss=5.79, v_num=0]
Epoch 0:  95%|#########5| 21/22 [00:07<00:00,  2.80it/s, loss=5.76, v_num=0]
Epoch 0: 100%|##########| 22/22 [00:07<00:00,  2.90it/s, loss=5.76, v_num=0]
Epoch 0: 100%|##########| 22/22 [00:07<00:00,  2.87it/s, loss=5.72, v_num=0]
Epoch 0: 100%|##########| 22/22 [00:07<00:00,  2.87it/s, loss=5.72, v_num=0]
Epoch 0:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.72, v_num=0]
Epoch 1:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.72, v_num=0]
Epoch 1:   5%|4         | 1/22 [00:02<00:42,  2.04s/it, loss=5.72, v_num=0]
Epoch 1:   5%|4         | 1/22 [00:02<00:43,  2.09s/it, loss=5.69, v_num=0]
Epoch 1:   9%|9         | 2/22 [00:02<00:21,  1.10s/it, loss=5.69, v_num=0]
Epoch 1:   9%|9         | 2/22 [00:02<00:22,  1.13s/it, loss=5.67, v_num=0]
Epoch 1:  14%|#3        | 3/22 [00:02<00:14,  1.28it/s, loss=5.67, v_num=0]
Epoch 1:  14%|#3        | 3/22 [00:02<00:15,  1.25it/s, loss=5.64, v_num=0]
Epoch 1:  18%|#8        | 4/22 [00:02<00:11,  1.61it/s, loss=5.64, v_num=0]
Epoch 1:  18%|#8        | 4/22 [00:02<00:11,  1.57it/s, loss=5.62, v_num=0]
Epoch 1:  23%|##2       | 5/22 [00:02<00:08,  1.89it/s, loss=5.62, v_num=0]
Epoch 1:  23%|##2       | 5/22 [00:02<00:09,  1.85it/s, loss=5.61, v_num=0]
Epoch 1:  27%|##7       | 6/22 [00:02<00:07,  2.15it/s, loss=5.61, v_num=0]
Epoch 1:  27%|##7       | 6/22 [00:02<00:07,  2.10it/s, loss=5.59, v_num=0]
Epoch 1:  32%|###1      | 7/22 [00:02<00:06,  2.38it/s, loss=5.59, v_num=0]
Epoch 1:  32%|###1      | 7/22 [00:03<00:06,  2.33it/s, loss=5.58, v_num=0]
Epoch 1:  36%|###6      | 8/22 [00:03<00:05,  2.59it/s, loss=5.58, v_num=0]
Epoch 1:  36%|###6      | 8/22 [00:03<00:05,  2.54it/s, loss=5.57, v_num=0]
Epoch 1:  41%|####      | 9/22 [00:03<00:04,  2.70it/s, loss=5.57, v_num=0]
Epoch 1:  41%|####      | 9/22 [00:03<00:04,  2.65it/s, loss=5.55, v_num=0]
Epoch 1:  45%|####5     | 10/22 [00:03<00:04,  2.86it/s, loss=5.55, v_num=0]
Epoch 1:  45%|####5     | 10/22 [00:03<00:04,  2.81it/s, loss=5.55, v_num=0]
Epoch 1:  50%|#####     | 11/22 [00:03<00:03,  3.02it/s, loss=5.55, v_num=0]
Epoch 1:  50%|#####     | 11/22 [00:03<00:03,  2.97it/s, loss=5.54, v_num=0]
Epoch 1:  55%|#####4    | 12/22 [00:03<00:03,  3.17it/s, loss=5.54, v_num=0]
Epoch 1:  55%|#####4    | 12/22 [00:03<00:03,  3.12it/s, loss=5.54, v_num=0]
Epoch 1:  59%|#####9    | 13/22 [00:03<00:02,  3.30it/s, loss=5.54, v_num=0]
Epoch 1:  59%|#####9    | 13/22 [00:03<00:02,  3.25it/s, loss=5.53, v_num=0]
Epoch 1:  64%|######3   | 14/22 [00:04<00:02,  3.43it/s, loss=5.53, v_num=0]
Epoch 1:  64%|######3   | 14/22 [00:04<00:02,  3.38it/s, loss=5.53, v_num=0]
Epoch 1:  68%|######8   | 15/22 [00:04<00:01,  3.54it/s, loss=5.53, v_num=0]
Epoch 1:  68%|######8   | 15/22 [00:04<00:02,  3.50it/s, loss=5.52, v_num=0]
Epoch 1:  73%|#######2  | 16/22 [00:04<00:01,  3.65it/s, loss=5.52, v_num=0]
Epoch 1:  73%|#######2  | 16/22 [00:04<00:01,  3.60it/s, loss=5.52, v_num=0]
Epoch 1:  77%|#######7  | 17/22 [00:04<00:01,  3.70it/s, loss=5.52, v_num=0]
Epoch 1:  77%|#######7  | 17/22 [00:04<00:01,  3.65it/s, loss=5.51, v_num=0]
Epoch 1:  82%|########1 | 18/22 [00:04<00:01,  3.79it/s, loss=5.51, v_num=0]
Epoch 1:  82%|########1 | 18/22 [00:04<00:01,  3.74it/s, loss=5.5, v_num=0]
Epoch 1:  86%|########6 | 19/22 [00:04<00:00,  3.88it/s, loss=5.5, v_num=0]
Epoch 1:  86%|########6 | 19/22 [00:04<00:00,  3.83it/s, loss=5.5, v_num=0]
Epoch 1:  91%|######### | 20/22 [00:05<00:00,  3.96it/s, loss=5.5, v_num=0]
Epoch 1:  91%|######### | 20/22 [00:05<00:00,  3.92it/s, loss=5.5, v_num=0]
Epoch 1:  95%|#########5| 21/22 [00:05<00:00,  4.04it/s, loss=5.5, v_num=0]
Epoch 1:  95%|#########5| 21/22 [00:05<00:00,  4.00it/s, loss=5.49, v_num=0]
Epoch 1: 100%|##########| 22/22 [00:05<00:00,  4.12it/s, loss=5.49, v_num=0]
Epoch 1: 100%|##########| 22/22 [00:05<00:00,  4.07it/s, loss=5.49, v_num=0]
Epoch 1: 100%|##########| 22/22 [00:05<00:00,  4.07it/s, loss=5.49, v_num=0]
Epoch 1:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.49, v_num=0]
Epoch 2:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.49, v_num=0]
Epoch 2:   5%|4         | 1/22 [00:02<00:43,  2.09s/it, loss=5.49, v_num=0]
Epoch 2:   5%|4         | 1/22 [00:02<00:45,  2.15s/it, loss=5.48, v_num=0]
Epoch 2:   9%|9         | 2/22 [00:02<00:22,  1.12s/it, loss=5.48, v_num=0]
Epoch 2:   9%|9         | 2/22 [00:02<00:23,  1.15s/it, loss=5.48, v_num=0]
Epoch 2:  14%|#3        | 3/22 [00:02<00:15,  1.25it/s, loss=5.48, v_num=0]
Epoch 2:  14%|#3        | 3/22 [00:02<00:15,  1.22it/s, loss=5.48, v_num=0]
Epoch 2:  18%|#8        | 4/22 [00:02<00:11,  1.57it/s, loss=5.48, v_num=0]
Epoch 2:  18%|#8        | 4/22 [00:02<00:11,  1.53it/s, loss=5.47, v_num=0]
Epoch 2:  23%|##2       | 5/22 [00:02<00:09,  1.85it/s, loss=5.47, v_num=0]
Epoch 2:  23%|##2       | 5/22 [00:02<00:09,  1.81it/s, loss=5.47, v_num=0]
Epoch 2:  27%|##7       | 6/22 [00:02<00:07,  2.11it/s, loss=5.47, v_num=0]
Epoch 2:  27%|##7       | 6/22 [00:02<00:07,  2.06it/s, loss=5.46, v_num=0]
Epoch 2:  32%|###1      | 7/22 [00:02<00:06,  2.34it/s, loss=5.46, v_num=0]
Epoch 2:  32%|###1      | 7/22 [00:03<00:06,  2.29it/s, loss=5.46, v_num=0]
Epoch 2:  36%|###6      | 8/22 [00:03<00:05,  2.54it/s, loss=5.46, v_num=0]
Epoch 2:  36%|###6      | 8/22 [00:03<00:05,  2.49it/s, loss=5.45, v_num=0]
Epoch 2:  41%|####      | 9/22 [00:03<00:04,  2.60it/s, loss=5.45, v_num=0]
Epoch 2:  41%|####      | 9/22 [00:03<00:05,  2.56it/s, loss=5.45, v_num=0]
Epoch 2:  45%|####5     | 10/22 [00:03<00:04,  2.77it/s, loss=5.45, v_num=0]
Epoch 2:  45%|####5     | 10/22 [00:03<00:04,  2.72it/s, loss=5.44, v_num=0]
Epoch 2:  50%|#####     | 11/22 [00:03<00:03,  2.92it/s, loss=5.44, v_num=0]
Epoch 2:  50%|#####     | 11/22 [00:03<00:03,  2.88it/s, loss=5.44, v_num=0]
Epoch 2:  55%|#####4    | 12/22 [00:03<00:03,  3.07it/s, loss=5.44, v_num=0]
Epoch 2:  55%|#####4    | 12/22 [00:03<00:03,  3.02it/s, loss=5.43, v_num=0]
Epoch 2:  59%|#####9    | 13/22 [00:04<00:02,  3.21it/s, loss=5.43, v_num=0]
Epoch 2:  59%|#####9    | 13/22 [00:04<00:02,  3.16it/s, loss=5.42, v_num=0]
Epoch 2:  64%|######3   | 14/22 [00:04<00:02,  3.33it/s, loss=5.42, v_num=0]
Epoch 2:  64%|######3   | 14/22 [00:04<00:02,  3.28it/s, loss=5.42, v_num=0]
Epoch 2:  68%|######8   | 15/22 [00:04<00:02,  3.45it/s, loss=5.42, v_num=0]
Epoch 2:  68%|######8   | 15/22 [00:04<00:02,  3.40it/s, loss=5.42, v_num=0]
Epoch 2:  73%|#######2  | 16/22 [00:04<00:01,  3.56it/s, loss=5.42, v_num=0]
Epoch 2:  73%|#######2  | 16/22 [00:04<00:01,  3.51it/s, loss=5.41, v_num=0]
Epoch 2:  77%|#######7  | 17/22 [00:04<00:01,  3.56it/s, loss=5.41, v_num=0]
Epoch 2:  77%|#######7  | 17/22 [00:04<00:01,  3.52it/s, loss=5.41, v_num=0]
Epoch 2:  82%|########1 | 18/22 [00:04<00:01,  3.66it/s, loss=5.41, v_num=0]
Epoch 2:  82%|########1 | 18/22 [00:04<00:01,  3.61it/s, loss=5.4, v_num=0]
Epoch 2:  86%|########6 | 19/22 [00:05<00:00,  3.75it/s, loss=5.4, v_num=0]
Epoch 2:  86%|########6 | 19/22 [00:05<00:00,  3.70it/s, loss=5.39, v_num=0]
Epoch 2:  91%|######### | 20/22 [00:05<00:00,  3.83it/s, loss=5.39, v_num=0]
Epoch 2:  91%|######### | 20/22 [00:05<00:00,  3.79it/s, loss=5.39, v_num=0]
Epoch 2:  95%|#########5| 21/22 [00:05<00:00,  3.92it/s, loss=5.39, v_num=0]
Epoch 2:  95%|#########5| 21/22 [00:05<00:00,  3.87it/s, loss=5.38, v_num=0]
Epoch 2: 100%|##########| 22/22 [00:05<00:00,  3.99it/s, loss=5.38, v_num=0]
Epoch 2: 100%|##########| 22/22 [00:05<00:00,  3.95it/s, loss=5.38, v_num=0]
Epoch 2: 100%|##########| 22/22 [00:05<00:00,  3.95it/s, loss=5.38, v_num=0]
Epoch 2:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.38, v_num=0]
Epoch 3:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.38, v_num=0]
Epoch 3:   5%|4         | 1/22 [00:02<00:44,  2.13s/it, loss=5.38, v_num=0]
Epoch 3:   5%|4         | 1/22 [00:02<00:46,  2.19s/it, loss=5.38, v_num=0]
Epoch 3:   9%|9         | 2/22 [00:02<00:22,  1.15s/it, loss=5.38, v_num=0]
Epoch 3:   9%|9         | 2/22 [00:02<00:23,  1.18s/it, loss=5.38, v_num=0]
Epoch 3:  14%|#3        | 3/22 [00:02<00:15,  1.23it/s, loss=5.38, v_num=0]
Epoch 3:  14%|#3        | 3/22 [00:02<00:15,  1.20it/s, loss=5.37, v_num=0]
Epoch 3:  18%|#8        | 4/22 [00:02<00:11,  1.54it/s, loss=5.37, v_num=0]
Epoch 3:  18%|#8        | 4/22 [00:02<00:11,  1.51it/s, loss=5.38, v_num=0]
Epoch 3:  23%|##2       | 5/22 [00:02<00:09,  1.82it/s, loss=5.38, v_num=0]
Epoch 3:  23%|##2       | 5/22 [00:02<00:09,  1.78it/s, loss=5.37, v_num=0]
Epoch 3:  27%|##7       | 6/22 [00:02<00:07,  2.08it/s, loss=5.37, v_num=0]
Epoch 3:  27%|##7       | 6/22 [00:02<00:07,  2.03it/s, loss=5.37, v_num=0]
Epoch 3:  32%|###1      | 7/22 [00:03<00:06,  2.30it/s, loss=5.37, v_num=0]
Epoch 3:  32%|###1      | 7/22 [00:03<00:06,  2.26it/s, loss=5.37, v_num=0]
Epoch 3:  36%|###6      | 8/22 [00:03<00:05,  2.50it/s, loss=5.37, v_num=0]
Epoch 3:  36%|###6      | 8/22 [00:03<00:05,  2.46it/s, loss=5.37, v_num=0]
Epoch 3:  41%|####      | 9/22 [00:03<00:05,  2.57it/s, loss=5.37, v_num=0]
Epoch 3:  41%|####      | 9/22 [00:03<00:05,  2.53it/s, loss=5.36, v_num=0]
Epoch 3:  45%|####5     | 10/22 [00:03<00:04,  2.74it/s, loss=5.36, v_num=0]
Epoch 3:  45%|####5     | 10/22 [00:03<00:04,  2.69it/s, loss=5.36, v_num=0]
Epoch 3:  50%|#####     | 11/22 [00:03<00:03,  2.90it/s, loss=5.36, v_num=0]
Epoch 3:  50%|#####     | 11/22 [00:03<00:03,  2.85it/s, loss=5.36, v_num=0]
Epoch 3:  55%|#####4    | 12/22 [00:03<00:03,  3.04it/s, loss=5.36, v_num=0]
Epoch 3:  55%|#####4    | 12/22 [00:04<00:03,  2.99it/s, loss=5.36, v_num=0]
Epoch 3:  59%|#####9    | 13/22 [00:04<00:02,  3.17it/s, loss=5.36, v_num=0]
Epoch 3:  59%|#####9    | 13/22 [00:04<00:02,  3.13it/s, loss=5.35, v_num=0]
Epoch 3:  64%|######3   | 14/22 [00:04<00:02,  3.30it/s, loss=5.35, v_num=0]
Epoch 3:  64%|######3   | 14/22 [00:04<00:02,  3.25it/s, loss=5.35, v_num=0]
Epoch 3:  68%|######8   | 15/22 [00:04<00:02,  3.42it/s, loss=5.35, v_num=0]
Epoch 3:  68%|######8   | 15/22 [00:04<00:02,  3.37it/s, loss=5.34, v_num=0]
Epoch 3:  73%|#######2  | 16/22 [00:04<00:01,  3.52it/s, loss=5.34, v_num=0]
Epoch 3:  73%|#######2  | 16/22 [00:04<00:01,  3.48it/s, loss=5.35, v_num=0]
Epoch 3:  77%|#######7  | 17/22 [00:04<00:01,  3.53it/s, loss=5.35, v_num=0]
Epoch 3:  77%|#######7  | 17/22 [00:04<00:01,  3.49it/s, loss=5.34, v_num=0]
Epoch 3:  82%|########1 | 18/22 [00:04<00:01,  3.63it/s, loss=5.34, v_num=0]
Epoch 3:  82%|########1 | 18/22 [00:05<00:01,  3.58it/s, loss=5.34, v_num=0]
Epoch 3:  86%|########6 | 19/22 [00:05<00:00,  3.72it/s, loss=5.34, v_num=0]
Epoch 3:  86%|########6 | 19/22 [00:05<00:00,  3.67it/s, loss=5.34, v_num=0]
Epoch 3:  91%|######### | 20/22 [00:05<00:00,  3.80it/s, loss=5.34, v_num=0]
Epoch 3:  91%|######### | 20/22 [00:05<00:00,  3.76it/s, loss=5.33, v_num=0]
Epoch 3:  95%|#########5| 21/22 [00:05<00:00,  3.88it/s, loss=5.33, v_num=0]
Epoch 3:  95%|#########5| 21/22 [00:05<00:00,  3.84it/s, loss=5.33, v_num=0]
Epoch 3: 100%|##########| 22/22 [00:05<00:00,  3.96it/s, loss=5.33, v_num=0]
Epoch 3: 100%|##########| 22/22 [00:05<00:00,  3.91it/s, loss=5.32, v_num=0]
Epoch 3: 100%|##########| 22/22 [00:05<00:00,  3.91it/s, loss=5.32, v_num=0]
Epoch 3:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.32, v_num=0]
Epoch 4:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.32, v_num=0]
Epoch 4:   5%|4         | 1/22 [00:02<00:45,  2.15s/it, loss=5.32, v_num=0]
Epoch 4:   5%|4         | 1/22 [00:02<00:46,  2.21s/it, loss=5.32, v_num=0]
Epoch 4:   9%|9         | 2/22 [00:02<00:23,  1.16s/it, loss=5.32, v_num=0]
Epoch 4:   9%|9         | 2/22 [00:02<00:23,  1.19s/it, loss=5.31, v_num=0]
Epoch 4:  14%|#3        | 3/22 [00:02<00:15,  1.22it/s, loss=5.31, v_num=0]
Epoch 4:  14%|#3        | 3/22 [00:02<00:15,  1.19it/s, loss=5.31, v_num=0]
Epoch 4:  18%|#8        | 4/22 [00:02<00:11,  1.53it/s, loss=5.31, v_num=0]
Epoch 4:  18%|#8        | 4/22 [00:02<00:12,  1.50it/s, loss=5.3, v_num=0]
Epoch 4:  23%|##2       | 5/22 [00:02<00:09,  1.81it/s, loss=5.3, v_num=0]
Epoch 4:  23%|##2       | 5/22 [00:02<00:09,  1.77it/s, loss=5.31, v_num=0]
Epoch 4:  27%|##7       | 6/22 [00:02<00:07,  2.06it/s, loss=5.31, v_num=0]
Epoch 4:  27%|##7       | 6/22 [00:02<00:07,  2.02it/s, loss=5.3, v_num=0]
Epoch 4:  32%|###1      | 7/22 [00:03<00:06,  2.29it/s, loss=5.3, v_num=0]
Epoch 4:  32%|###1      | 7/22 [00:03<00:06,  2.24it/s, loss=5.31, v_num=0]
Epoch 4:  36%|###6      | 8/22 [00:03<00:05,  2.49it/s, loss=5.31, v_num=0]
Epoch 4:  36%|###6      | 8/22 [00:03<00:05,  2.44it/s, loss=5.3, v_num=0]
Epoch 4:  41%|####      | 9/22 [00:03<00:05,  2.55it/s, loss=5.3, v_num=0]
Epoch 4:  41%|####      | 9/22 [00:03<00:05,  2.51it/s, loss=5.3, v_num=0]
Epoch 4:  45%|####5     | 10/22 [00:03<00:04,  2.72it/s, loss=5.3, v_num=0]
Epoch 4:  45%|####5     | 10/22 [00:03<00:04,  2.67it/s, loss=5.3, v_num=0]
Epoch 4:  50%|#####     | 11/22 [00:03<00:03,  2.87it/s, loss=5.3, v_num=0]
Epoch 4:  50%|#####     | 11/22 [00:03<00:03,  2.83it/s, loss=5.3, v_num=0]
Epoch 4:  55%|#####4    | 12/22 [00:03<00:03,  3.01it/s, loss=5.3, v_num=0]
Epoch 4:  55%|#####4    | 12/22 [00:04<00:03,  2.97it/s, loss=5.29, v_num=0]
Epoch 4:  59%|#####9    | 13/22 [00:04<00:02,  3.14it/s, loss=5.29, v_num=0]
Epoch 4:  59%|#####9    | 13/22 [00:04<00:02,  3.10it/s, loss=5.29, v_num=0]
Epoch 4:  64%|######3   | 14/22 [00:04<00:02,  3.27it/s, loss=5.29, v_num=0]
Epoch 4:  64%|######3   | 14/22 [00:04<00:02,  3.22it/s, loss=5.29, v_num=0]
Epoch 4:  68%|######8   | 15/22 [00:04<00:02,  3.38it/s, loss=5.29, v_num=0]
Epoch 4:  68%|######8   | 15/22 [00:04<00:02,  3.34it/s, loss=5.29, v_num=0]
Epoch 4:  73%|#######2  | 16/22 [00:04<00:01,  3.49it/s, loss=5.29, v_num=0]
Epoch 4:  73%|#######2  | 16/22 [00:04<00:01,  3.45it/s, loss=5.29, v_num=0]
Epoch 4:  77%|#######7  | 17/22 [00:04<00:01,  3.50it/s, loss=5.29, v_num=0]
Epoch 4:  77%|#######7  | 17/22 [00:04<00:01,  3.46it/s, loss=5.28, v_num=0]
Epoch 4:  82%|########1 | 18/22 [00:05<00:01,  3.59it/s, loss=5.28, v_num=0]
Epoch 4:  82%|########1 | 18/22 [00:05<00:01,  3.55it/s, loss=5.29, v_num=0]
Epoch 4:  86%|########6 | 19/22 [00:05<00:00,  3.68it/s, loss=5.29, v_num=0]
Epoch 4:  86%|########6 | 19/22 [00:05<00:00,  3.64it/s, loss=5.28, v_num=0]
Epoch 4:  91%|######### | 20/22 [00:05<00:00,  3.77it/s, loss=5.28, v_num=0]
Epoch 4:  91%|######### | 20/22 [00:05<00:00,  3.73it/s, loss=5.28, v_num=0]
Epoch 4:  95%|#########5| 21/22 [00:05<00:00,  3.85it/s, loss=5.28, v_num=0]
Epoch 4:  95%|#########5| 21/22 [00:05<00:00,  3.81it/s, loss=5.28, v_num=0]
Epoch 4: 100%|##########| 22/22 [00:05<00:00,  3.93it/s, loss=5.28, v_num=0]
Epoch 4: 100%|##########| 22/22 [00:05<00:00,  3.89it/s, loss=5.28, v_num=0]
Epoch 4: 100%|##########| 22/22 [00:05<00:00,  3.89it/s, loss=5.28, v_num=0]
Epoch 4:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.28, v_num=0]
Epoch 5:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.28, v_num=0]
Epoch 5:   5%|4         | 1/22 [00:02<00:43,  2.07s/it, loss=5.28, v_num=0]
Epoch 5:   5%|4         | 1/22 [00:02<00:44,  2.13s/it, loss=5.28, v_num=0]
Epoch 5:   9%|9         | 2/22 [00:02<00:22,  1.11s/it, loss=5.28, v_num=0]
Epoch 5:   9%|9         | 2/22 [00:02<00:22,  1.14s/it, loss=5.28, v_num=0]
Epoch 5:  14%|#3        | 3/22 [00:02<00:15,  1.26it/s, loss=5.28, v_num=0]
Epoch 5:  14%|#3        | 3/22 [00:02<00:15,  1.23it/s, loss=5.28, v_num=0]
Epoch 5:  18%|#8        | 4/22 [00:02<00:11,  1.58it/s, loss=5.28, v_num=0]
Epoch 5:  18%|#8        | 4/22 [00:02<00:11,  1.55it/s, loss=5.28, v_num=0]
Epoch 5:  23%|##2       | 5/22 [00:02<00:09,  1.87it/s, loss=5.28, v_num=0]
Epoch 5:  23%|##2       | 5/22 [00:02<00:09,  1.83it/s, loss=5.27, v_num=0]
Epoch 5:  27%|##7       | 6/22 [00:02<00:07,  2.12it/s, loss=5.27, v_num=0]
Epoch 5:  27%|##7       | 6/22 [00:02<00:07,  2.08it/s, loss=5.28, v_num=0]
Epoch 5:  32%|###1      | 7/22 [00:02<00:06,  2.35it/s, loss=5.28, v_num=0]
Epoch 5:  32%|###1      | 7/22 [00:03<00:06,  2.30it/s, loss=5.27, v_num=0]
Epoch 5:  36%|###6      | 8/22 [00:03<00:05,  2.55it/s, loss=5.27, v_num=0]
Epoch 5:  36%|###6      | 8/22 [00:03<00:05,  2.51it/s, loss=5.28, v_num=0]
Epoch 5:  41%|####      | 9/22 [00:03<00:04,  2.63it/s, loss=5.28, v_num=0]
Epoch 5:  41%|####      | 9/22 [00:03<00:05,  2.59it/s, loss=5.27, v_num=0]
Epoch 5:  45%|####5     | 10/22 [00:03<00:04,  2.80it/s, loss=5.27, v_num=0]
Epoch 5:  45%|####5     | 10/22 [00:03<00:04,  2.75it/s, loss=5.26, v_num=0]
Epoch 5:  50%|#####     | 11/22 [00:03<00:03,  2.96it/s, loss=5.26, v_num=0]
Epoch 5:  50%|#####     | 11/22 [00:03<00:03,  2.91it/s, loss=5.26, v_num=0]
Epoch 5:  55%|#####4    | 12/22 [00:03<00:03,  3.10it/s, loss=5.26, v_num=0]
Epoch 5:  55%|#####4    | 12/22 [00:03<00:03,  3.05it/s, loss=5.26, v_num=0]
Epoch 5:  59%|#####9    | 13/22 [00:04<00:02,  3.24it/s, loss=5.26, v_num=0]
Epoch 5:  59%|#####9    | 13/22 [00:04<00:02,  3.19it/s, loss=5.26, v_num=0]
Epoch 5:  64%|######3   | 14/22 [00:04<00:02,  3.36it/s, loss=5.26, v_num=0]
Epoch 5:  64%|######3   | 14/22 [00:04<00:02,  3.31it/s, loss=5.26, v_num=0]
Epoch 5:  68%|######8   | 15/22 [00:04<00:02,  3.48it/s, loss=5.26, v_num=0]
Epoch 5:  68%|######8   | 15/22 [00:04<00:02,  3.43it/s, loss=5.26, v_num=0]
Epoch 5:  73%|#######2  | 16/22 [00:04<00:01,  3.59it/s, loss=5.26, v_num=0]
Epoch 5:  73%|#######2  | 16/22 [00:04<00:01,  3.54it/s, loss=5.26, v_num=0]
Epoch 5:  77%|#######7  | 17/22 [00:04<00:01,  3.60it/s, loss=5.26, v_num=0]
Epoch 5:  77%|#######7  | 17/22 [00:04<00:01,  3.56it/s, loss=5.26, v_num=0]
Epoch 5:  82%|########1 | 18/22 [00:04<00:01,  3.70it/s, loss=5.26, v_num=0]
Epoch 5:  82%|########1 | 18/22 [00:04<00:01,  3.65it/s, loss=5.26, v_num=0]
Epoch 5:  86%|########6 | 19/22 [00:05<00:00,  3.78it/s, loss=5.26, v_num=0]
Epoch 5:  86%|########6 | 19/22 [00:05<00:00,  3.74it/s, loss=5.25, v_num=0]
Epoch 5:  91%|######### | 20/22 [00:05<00:00,  3.87it/s, loss=5.25, v_num=0]
Epoch 5:  91%|######### | 20/22 [00:05<00:00,  3.82it/s, loss=5.25, v_num=0]
Epoch 5:  95%|#########5| 21/22 [00:05<00:00,  3.95it/s, loss=5.25, v_num=0]
Epoch 5:  95%|#########5| 21/22 [00:05<00:00,  3.91it/s, loss=5.25, v_num=0]
Epoch 5: 100%|##########| 22/22 [00:05<00:00,  4.03it/s, loss=5.25, v_num=0]
Epoch 5: 100%|##########| 22/22 [00:05<00:00,  3.98it/s, loss=5.25, v_num=0]
Epoch 5: 100%|##########| 22/22 [00:05<00:00,  3.98it/s, loss=5.25, v_num=0]
Epoch 5:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.25, v_num=0]
Epoch 6:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.25, v_num=0]
Epoch 6:   5%|4         | 1/22 [00:02<00:45,  2.16s/it, loss=5.25, v_num=0]
Epoch 6:   5%|4         | 1/22 [00:02<00:46,  2.22s/it, loss=5.24, v_num=0]
Epoch 6:   9%|9         | 2/22 [00:02<00:23,  1.16s/it, loss=5.24, v_num=0]
Epoch 6:   9%|9         | 2/22 [00:02<00:23,  1.19s/it, loss=5.24, v_num=0]
Epoch 6:  14%|#3        | 3/22 [00:02<00:15,  1.21it/s, loss=5.24, v_num=0]
Epoch 6:  14%|#3        | 3/22 [00:02<00:16,  1.19it/s, loss=5.23, v_num=0]
Epoch 6:  18%|#8        | 4/22 [00:02<00:11,  1.52it/s, loss=5.23, v_num=0]
Epoch 6:  18%|#8        | 4/22 [00:02<00:12,  1.49it/s, loss=5.23, v_num=0]
Epoch 6:  23%|##2       | 5/22 [00:02<00:09,  1.80it/s, loss=5.23, v_num=0]
Epoch 6:  23%|##2       | 5/22 [00:02<00:09,  1.76it/s, loss=5.23, v_num=0]
Epoch 6:  27%|##7       | 6/22 [00:02<00:07,  2.05it/s, loss=5.23, v_num=0]
Epoch 6:  27%|##7       | 6/22 [00:02<00:07,  2.01it/s, loss=5.22, v_num=0]
Epoch 6:  32%|###1      | 7/22 [00:03<00:06,  2.28it/s, loss=5.22, v_num=0]
Epoch 6:  32%|###1      | 7/22 [00:03<00:06,  2.23it/s, loss=5.22, v_num=0]
Epoch 6:  36%|###6      | 8/22 [00:03<00:05,  2.48it/s, loss=5.22, v_num=0]
Epoch 6:  36%|###6      | 8/22 [00:03<00:05,  2.44it/s, loss=5.22, v_num=0]
Epoch 6:  41%|####      | 9/22 [00:03<00:05,  2.57it/s, loss=5.22, v_num=0]
Epoch 6:  41%|####      | 9/22 [00:03<00:05,  2.53it/s, loss=5.22, v_num=0]
Epoch 6:  45%|####5     | 10/22 [00:03<00:04,  2.73it/s, loss=5.22, v_num=0]
Epoch 6:  45%|####5     | 10/22 [00:03<00:04,  2.69it/s, loss=5.22, v_num=0]
Epoch 6:  50%|#####     | 11/22 [00:03<00:03,  2.89it/s, loss=5.22, v_num=0]
Epoch 6:  50%|#####     | 11/22 [00:03<00:03,  2.84it/s, loss=5.21, v_num=0]
Epoch 6:  55%|#####4    | 12/22 [00:03<00:03,  3.03it/s, loss=5.21, v_num=0]
Epoch 6:  55%|#####4    | 12/22 [00:04<00:03,  2.99it/s, loss=5.21, v_num=0]
Epoch 6:  59%|#####9    | 13/22 [00:04<00:02,  3.17it/s, loss=5.21, v_num=0]
Epoch 6:  59%|#####9    | 13/22 [00:04<00:02,  3.12it/s, loss=5.21, v_num=0]
Epoch 6:  64%|######3   | 14/22 [00:04<00:02,  3.29it/s, loss=5.21, v_num=0]
Epoch 6:  64%|######3   | 14/22 [00:04<00:02,  3.24it/s, loss=5.22, v_num=0]
Epoch 6:  68%|######8   | 15/22 [00:04<00:02,  3.40it/s, loss=5.22, v_num=0]
Epoch 6:  68%|######8   | 15/22 [00:04<00:02,  3.36it/s, loss=5.21, v_num=0]
Epoch 6:  73%|#######2  | 16/22 [00:04<00:01,  3.51it/s, loss=5.21, v_num=0]
Epoch 6:  73%|#######2  | 16/22 [00:04<00:01,  3.47it/s, loss=5.21, v_num=0]
Epoch 6:  77%|#######7  | 17/22 [00:04<00:01,  3.54it/s, loss=5.21, v_num=0]
Epoch 6:  77%|#######7  | 17/22 [00:04<00:01,  3.49it/s, loss=5.21, v_num=0]
Epoch 6:  82%|########1 | 18/22 [00:04<00:01,  3.63it/s, loss=5.21, v_num=0]
Epoch 6:  82%|########1 | 18/22 [00:05<00:01,  3.59it/s, loss=5.2, v_num=0]
Epoch 6:  86%|########6 | 19/22 [00:05<00:00,  3.72it/s, loss=5.2, v_num=0]
Epoch 6:  86%|########6 | 19/22 [00:05<00:00,  3.68it/s, loss=5.2, v_num=0]
Epoch 6:  91%|######### | 20/22 [00:05<00:00,  3.80it/s, loss=5.2, v_num=0]
Epoch 6:  91%|######### | 20/22 [00:05<00:00,  3.76it/s, loss=5.21, v_num=0]
Epoch 6:  95%|#########5| 21/22 [00:05<00:00,  3.88it/s, loss=5.21, v_num=0]
Epoch 6:  95%|#########5| 21/22 [00:05<00:00,  3.84it/s, loss=5.2, v_num=0]
Epoch 6: 100%|##########| 22/22 [00:05<00:00,  3.96it/s, loss=5.2, v_num=0]
Epoch 6: 100%|##########| 22/22 [00:05<00:00,  3.91it/s, loss=5.2, v_num=0]
Epoch 6: 100%|##########| 22/22 [00:05<00:00,  3.91it/s, loss=5.2, v_num=0]
Epoch 6:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.2, v_num=0]
Epoch 7:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.2, v_num=0]
Epoch 7:   5%|4         | 1/22 [00:02<00:45,  2.17s/it, loss=5.2, v_num=0]
Epoch 7:   5%|4         | 1/22 [00:02<00:46,  2.23s/it, loss=5.21, v_num=0]
Epoch 7:   9%|9         | 2/22 [00:02<00:23,  1.16s/it, loss=5.21, v_num=0]
Epoch 7:   9%|9         | 2/22 [00:02<00:23,  1.19s/it, loss=5.21, v_num=0]
Epoch 7:  14%|#3        | 3/22 [00:02<00:15,  1.21it/s, loss=5.21, v_num=0]
Epoch 7:  14%|#3        | 3/22 [00:02<00:16,  1.18it/s, loss=5.2, v_num=0]
Epoch 7:  18%|#8        | 4/22 [00:02<00:11,  1.52it/s, loss=5.2, v_num=0]
Epoch 7:  18%|#8        | 4/22 [00:02<00:12,  1.49it/s, loss=5.2, v_num=0]
Epoch 7:  23%|##2       | 5/22 [00:02<00:09,  1.80it/s, loss=5.2, v_num=0]
Epoch 7:  23%|##2       | 5/22 [00:02<00:09,  1.76it/s, loss=5.19, v_num=0]
Epoch 7:  27%|##7       | 6/22 [00:02<00:07,  2.04it/s, loss=5.19, v_num=0]
Epoch 7:  27%|##7       | 6/22 [00:02<00:07,  2.00it/s, loss=5.19, v_num=0]
Epoch 7:  32%|###1      | 7/22 [00:03<00:06,  2.27it/s, loss=5.19, v_num=0]
Epoch 7:  32%|###1      | 7/22 [00:03<00:06,  2.23it/s, loss=5.19, v_num=0]
Epoch 7:  36%|###6      | 8/22 [00:03<00:05,  2.47it/s, loss=5.19, v_num=0]
Epoch 7:  36%|###6      | 8/22 [00:03<00:05,  2.43it/s, loss=5.2, v_num=0]
Epoch 7:  41%|####      | 9/22 [00:03<00:05,  2.51it/s, loss=5.2, v_num=0]
Epoch 7:  41%|####      | 9/22 [00:03<00:05,  2.47it/s, loss=5.2, v_num=0]
Epoch 7:  45%|####5     | 10/22 [00:03<00:04,  2.67it/s, loss=5.2, v_num=0]
Epoch 7:  45%|####5     | 10/22 [00:03<00:04,  2.63it/s, loss=5.2, v_num=0]
Epoch 7:  50%|#####     | 11/22 [00:03<00:03,  2.83it/s, loss=5.2, v_num=0]
Epoch 7:  50%|#####     | 11/22 [00:03<00:03,  2.79it/s, loss=5.19, v_num=0]
Epoch 7:  55%|#####4    | 12/22 [00:04<00:03,  2.97it/s, loss=5.19, v_num=0]
Epoch 7:  55%|#####4    | 12/22 [00:04<00:03,  2.93it/s, loss=5.19, v_num=0]
Epoch 7:  59%|#####9    | 13/22 [00:04<00:02,  3.10it/s, loss=5.19, v_num=0]
Epoch 7:  59%|#####9    | 13/22 [00:04<00:02,  3.06it/s, loss=5.19, v_num=0]
Epoch 7:  64%|######3   | 14/22 [00:04<00:02,  3.22it/s, loss=5.19, v_num=0]
Epoch 7:  64%|######3   | 14/22 [00:04<00:02,  3.18it/s, loss=5.18, v_num=0]
Epoch 7:  68%|######8   | 15/22 [00:04<00:02,  3.34it/s, loss=5.18, v_num=0]
Epoch 7:  68%|######8   | 15/22 [00:04<00:02,  3.30it/s, loss=5.18, v_num=0]
Epoch 7:  73%|#######2  | 16/22 [00:04<00:01,  3.45it/s, loss=5.18, v_num=0]
Epoch 7:  73%|#######2  | 16/22 [00:04<00:01,  3.40it/s, loss=5.18, v_num=0]
Epoch 7:  77%|#######7  | 17/22 [00:04<00:01,  3.40it/s, loss=5.18, v_num=0]
Epoch 7:  77%|#######7  | 17/22 [00:05<00:01,  3.36it/s, loss=5.18, v_num=0]
Epoch 7:  82%|########1 | 18/22 [00:05<00:01,  3.50it/s, loss=5.18, v_num=0]
Epoch 7:  82%|########1 | 18/22 [00:05<00:01,  3.46it/s, loss=5.18, v_num=0]
Epoch 7:  86%|########6 | 19/22 [00:05<00:00,  3.59it/s, loss=5.18, v_num=0]
Epoch 7:  86%|########6 | 19/22 [00:05<00:00,  3.55it/s, loss=5.18, v_num=0]
Epoch 7:  91%|######### | 20/22 [00:05<00:00,  3.68it/s, loss=5.18, v_num=0]
Epoch 7:  91%|######### | 20/22 [00:05<00:00,  3.64it/s, loss=5.18, v_num=0]
Epoch 7:  95%|#########5| 21/22 [00:05<00:00,  3.76it/s, loss=5.18, v_num=0]
Epoch 7:  95%|#########5| 21/22 [00:05<00:00,  3.72it/s, loss=5.18, v_num=0]
Epoch 7: 100%|##########| 22/22 [00:05<00:00,  3.83it/s, loss=5.18, v_num=0]
Epoch 7: 100%|##########| 22/22 [00:05<00:00,  3.80it/s, loss=5.17, v_num=0]
Epoch 7: 100%|##########| 22/22 [00:05<00:00,  3.79it/s, loss=5.17, v_num=0]
Epoch 7:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.17, v_num=0]
Epoch 8:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.17, v_num=0]
Epoch 8:   5%|4         | 1/22 [00:02<00:45,  2.16s/it, loss=5.17, v_num=0]
Epoch 8:   5%|4         | 1/22 [00:02<00:46,  2.22s/it, loss=5.18, v_num=0]
Epoch 8:   9%|9         | 2/22 [00:02<00:23,  1.16s/it, loss=5.18, v_num=0]
Epoch 8:   9%|9         | 2/22 [00:02<00:23,  1.19s/it, loss=5.18, v_num=0]
Epoch 8:  14%|#3        | 3/22 [00:02<00:15,  1.21it/s, loss=5.18, v_num=0]
Epoch 8:  14%|#3        | 3/22 [00:02<00:16,  1.18it/s, loss=5.18, v_num=0]
Epoch 8:  18%|#8        | 4/22 [00:02<00:11,  1.52it/s, loss=5.18, v_num=0]
Epoch 8:  18%|#8        | 4/22 [00:02<00:12,  1.49it/s, loss=5.18, v_num=0]
Epoch 8:  23%|##2       | 5/22 [00:02<00:09,  1.80it/s, loss=5.18, v_num=0]
Epoch 8:  23%|##2       | 5/22 [00:02<00:09,  1.76it/s, loss=5.18, v_num=0]
Epoch 8:  27%|##7       | 6/22 [00:02<00:07,  2.04it/s, loss=5.18, v_num=0]
Epoch 8:  27%|##7       | 6/22 [00:02<00:07,  2.00it/s, loss=5.17, v_num=0]
Epoch 8:  32%|###1      | 7/22 [00:03<00:06,  2.26it/s, loss=5.17, v_num=0]
Epoch 8:  32%|###1      | 7/22 [00:03<00:06,  2.22it/s, loss=5.17, v_num=0]
Epoch 8:  36%|###6      | 8/22 [00:03<00:05,  2.47it/s, loss=5.17, v_num=0]
Epoch 8:  36%|###6      | 8/22 [00:03<00:05,  2.42it/s, loss=5.17, v_num=0]
Epoch 8:  41%|####      | 9/22 [00:03<00:05,  2.56it/s, loss=5.17, v_num=0]
Epoch 8:  41%|####      | 9/22 [00:03<00:05,  2.52it/s, loss=5.16, v_num=0]
Epoch 8:  45%|####5     | 10/22 [00:03<00:04,  2.72it/s, loss=5.16, v_num=0]
Epoch 8:  45%|####5     | 10/22 [00:03<00:04,  2.68it/s, loss=5.16, v_num=0]
Epoch 8:  50%|#####     | 11/22 [00:03<00:03,  2.87it/s, loss=5.16, v_num=0]
Epoch 8:  50%|#####     | 11/22 [00:03<00:03,  2.83it/s, loss=5.16, v_num=0]
Epoch 8:  55%|#####4    | 12/22 [00:03<00:03,  3.01it/s, loss=5.16, v_num=0]
Epoch 8:  55%|#####4    | 12/22 [00:04<00:03,  2.97it/s, loss=5.16, v_num=0]
Epoch 8:  59%|#####9    | 13/22 [00:04<00:02,  3.15it/s, loss=5.16, v_num=0]
Epoch 8:  59%|#####9    | 13/22 [00:04<00:02,  3.10it/s, loss=5.16, v_num=0]
Epoch 8:  64%|######3   | 14/22 [00:04<00:02,  3.27it/s, loss=5.16, v_num=0]
Epoch 8:  64%|######3   | 14/22 [00:04<00:02,  3.23it/s, loss=5.16, v_num=0]
Epoch 8:  68%|######8   | 15/22 [00:04<00:02,  3.39it/s, loss=5.16, v_num=0]
Epoch 8:  68%|######8   | 15/22 [00:04<00:02,  3.34it/s, loss=5.16, v_num=0]
Epoch 8:  73%|#######2  | 16/22 [00:04<00:01,  3.49it/s, loss=5.16, v_num=0]
Epoch 8:  73%|#######2  | 16/22 [00:04<00:01,  3.45it/s, loss=5.15, v_num=0]
Epoch 8:  77%|#######7  | 17/22 [00:04<00:01,  3.50it/s, loss=5.15, v_num=0]
Epoch 8:  77%|#######7  | 17/22 [00:04<00:01,  3.46it/s, loss=5.15, v_num=0]
Epoch 8:  82%|########1 | 18/22 [00:05<00:01,  3.60it/s, loss=5.15, v_num=0]
Epoch 8:  82%|########1 | 18/22 [00:05<00:01,  3.55it/s, loss=5.15, v_num=0]
Epoch 8:  86%|########6 | 19/22 [00:05<00:00,  3.69it/s, loss=5.15, v_num=0]
Epoch 8:  86%|########6 | 19/22 [00:05<00:00,  3.64it/s, loss=5.14, v_num=0]
Epoch 8:  91%|######### | 20/22 [00:05<00:00,  3.77it/s, loss=5.14, v_num=0]
Epoch 8:  91%|######### | 20/22 [00:05<00:00,  3.73it/s, loss=5.14, v_num=0]
Epoch 8:  95%|#########5| 21/22 [00:05<00:00,  3.85it/s, loss=5.14, v_num=0]
Epoch 8:  95%|#########5| 21/22 [00:05<00:00,  3.81it/s, loss=5.14, v_num=0]
Epoch 8: 100%|##########| 22/22 [00:05<00:00,  3.93it/s, loss=5.14, v_num=0]
Epoch 8: 100%|##########| 22/22 [00:05<00:00,  3.89it/s, loss=5.14, v_num=0]
Epoch 8: 100%|##########| 22/22 [00:05<00:00,  3.88it/s, loss=5.14, v_num=0]
Epoch 8:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.14, v_num=0]
Epoch 9:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.14, v_num=0]
Epoch 9:   5%|4         | 1/22 [00:02<00:43,  2.08s/it, loss=5.14, v_num=0]
Epoch 9:   5%|4         | 1/22 [00:02<00:44,  2.14s/it, loss=5.14, v_num=0]
Epoch 9:   9%|9         | 2/22 [00:02<00:22,  1.12s/it, loss=5.14, v_num=0]
Epoch 9:   9%|9         | 2/22 [00:02<00:22,  1.15s/it, loss=5.14, v_num=0]
Epoch 9:  14%|#3        | 3/22 [00:02<00:15,  1.25it/s, loss=5.14, v_num=0]
Epoch 9:  14%|#3        | 3/22 [00:02<00:15,  1.22it/s, loss=5.14, v_num=0]
Epoch 9:  18%|#8        | 4/22 [00:02<00:11,  1.56it/s, loss=5.14, v_num=0]
Epoch 9:  18%|#8        | 4/22 [00:02<00:11,  1.53it/s, loss=5.14, v_num=0]
Epoch 9:  23%|##2       | 5/22 [00:02<00:09,  1.85it/s, loss=5.14, v_num=0]
Epoch 9:  23%|##2       | 5/22 [00:02<00:09,  1.81it/s, loss=5.14, v_num=0]
Epoch 9:  27%|##7       | 6/22 [00:02<00:07,  2.10it/s, loss=5.14, v_num=0]
Epoch 9:  27%|##7       | 6/22 [00:02<00:07,  2.06it/s, loss=5.14, v_num=0]
Epoch 9:  32%|###1      | 7/22 [00:03<00:06,  2.33it/s, loss=5.14, v_num=0]
Epoch 9:  32%|###1      | 7/22 [00:03<00:06,  2.28it/s, loss=5.14, v_num=0]
Epoch 9:  36%|###6      | 8/22 [00:03<00:05,  2.53it/s, loss=5.14, v_num=0]
Epoch 9:  36%|###6      | 8/22 [00:03<00:05,  2.48it/s, loss=5.14, v_num=0]
Epoch 9:  41%|####      | 9/22 [00:03<00:04,  2.61it/s, loss=5.14, v_num=0]
Epoch 9:  41%|####      | 9/22 [00:03<00:05,  2.56it/s, loss=5.15, v_num=0]
Epoch 9:  45%|####5     | 10/22 [00:03<00:04,  2.77it/s, loss=5.15, v_num=0]
Epoch 9:  45%|####5     | 10/22 [00:03<00:04,  2.72it/s, loss=5.14, v_num=0]
Epoch 9:  50%|#####     | 11/22 [00:03<00:03,  2.92it/s, loss=5.14, v_num=0]
Epoch 9:  50%|#####     | 11/22 [00:03<00:03,  2.88it/s, loss=5.14, v_num=0]
Epoch 9:  55%|#####4    | 12/22 [00:03<00:03,  3.06it/s, loss=5.14, v_num=0]
Epoch 9:  55%|#####4    | 12/22 [00:03<00:03,  3.01it/s, loss=5.14, v_num=0]
Epoch 9:  59%|#####9    | 13/22 [00:04<00:02,  3.19it/s, loss=5.14, v_num=0]
Epoch 9:  59%|#####9    | 13/22 [00:04<00:02,  3.14it/s, loss=5.14, v_num=0]
Epoch 9:  64%|######3   | 14/22 [00:04<00:02,  3.31it/s, loss=5.14, v_num=0]
Epoch 9:  64%|######3   | 14/22 [00:04<00:02,  3.27it/s, loss=5.14, v_num=0]
Epoch 9:  68%|######8   | 15/22 [00:04<00:02,  3.43it/s, loss=5.14, v_num=0]
Epoch 9:  68%|######8   | 15/22 [00:04<00:02,  3.38it/s, loss=5.13, v_num=0]
Epoch 9:  73%|#######2  | 16/22 [00:04<00:01,  3.54it/s, loss=5.13, v_num=0]
Epoch 9:  73%|#######2  | 16/22 [00:04<00:01,  3.49it/s, loss=5.13, v_num=0]
Epoch 9:  77%|#######7  | 17/22 [00:04<00:01,  3.54it/s, loss=5.13, v_num=0]
Epoch 9:  77%|#######7  | 17/22 [00:04<00:01,  3.50it/s, loss=5.13, v_num=0]
Epoch 9:  82%|########1 | 18/22 [00:04<00:01,  3.63it/s, loss=5.13, v_num=0]
Epoch 9:  82%|########1 | 18/22 [00:05<00:01,  3.59it/s, loss=5.13, v_num=0]
Epoch 9:  86%|########6 | 19/22 [00:05<00:00,  3.72it/s, loss=5.13, v_num=0]
Epoch 9:  86%|########6 | 19/22 [00:05<00:00,  3.68it/s, loss=5.13, v_num=0]
Epoch 9:  91%|######### | 20/22 [00:05<00:00,  3.80it/s, loss=5.13, v_num=0]
Epoch 9:  91%|######### | 20/22 [00:05<00:00,  3.76it/s, loss=5.12, v_num=0]
Epoch 9:  95%|#########5| 21/22 [00:05<00:00,  3.89it/s, loss=5.12, v_num=0]
Epoch 9:  95%|#########5| 21/22 [00:05<00:00,  3.84it/s, loss=5.12, v_num=0]
Epoch 9: 100%|##########| 22/22 [00:05<00:00,  3.96it/s, loss=5.12, v_num=0]
Epoch 9: 100%|##########| 22/22 [00:05<00:00,  3.92it/s, loss=5.12, v_num=0]
Epoch 9: 100%|##########| 22/22 [00:05<00:00,  3.92it/s, loss=5.12, v_num=0]
Epoch 9:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.12, v_num=0]
Epoch 10:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.12, v_num=0]
Epoch 10:   5%|4         | 1/22 [00:02<00:46,  2.21s/it, loss=5.12, v_num=0]
Epoch 10:   5%|4         | 1/22 [00:02<00:47,  2.27s/it, loss=5.12, v_num=0]
Epoch 10:   9%|9         | 2/22 [00:02<00:23,  1.18s/it, loss=5.12, v_num=0]
Epoch 10:   9%|9         | 2/22 [00:02<00:24,  1.21s/it, loss=5.12, v_num=0]
Epoch 10:  14%|#3        | 3/22 [00:02<00:15,  1.19it/s, loss=5.12, v_num=0]
Epoch 10:  14%|#3        | 3/22 [00:02<00:16,  1.17it/s, loss=5.11, v_num=0]
Epoch 10:  18%|#8        | 4/22 [00:02<00:11,  1.50it/s, loss=5.11, v_num=0]
Epoch 10:  18%|#8        | 4/22 [00:02<00:12,  1.47it/s, loss=5.11, v_num=0]
Epoch 10:  23%|##2       | 5/22 [00:02<00:09,  1.77it/s, loss=5.11, v_num=0]
Epoch 10:  23%|##2       | 5/22 [00:02<00:09,  1.74it/s, loss=5.11, v_num=0]
Epoch 10:  27%|##7       | 6/22 [00:02<00:07,  2.02it/s, loss=5.11, v_num=0]
Epoch 10:  27%|##7       | 6/22 [00:03<00:08,  1.98it/s, loss=5.11, v_num=0]
Epoch 10:  32%|###1      | 7/22 [00:03<00:06,  2.24it/s, loss=5.11, v_num=0]
Epoch 10:  32%|###1      | 7/22 [00:03<00:06,  2.20it/s, loss=5.11, v_num=0]
Epoch 10:  36%|###6      | 8/22 [00:03<00:05,  2.45it/s, loss=5.11, v_num=0]
Epoch 10:  36%|###6      | 8/22 [00:03<00:05,  2.40it/s, loss=5.11, v_num=0]
Epoch 10:  41%|####      | 9/22 [00:03<00:05,  2.52it/s, loss=5.11, v_num=0]
Epoch 10:  41%|####      | 9/22 [00:03<00:05,  2.48it/s, loss=5.11, v_num=0]
Epoch 10:  45%|####5     | 10/22 [00:03<00:04,  2.68it/s, loss=5.11, v_num=0]
Epoch 10:  45%|####5     | 10/22 [00:03<00:04,  2.64it/s, loss=5.11, v_num=0]
Epoch 10:  50%|#####     | 11/22 [00:03<00:03,  2.84it/s, loss=5.11, v_num=0]
Epoch 10:  50%|#####     | 11/22 [00:03<00:03,  2.80it/s, loss=5.11, v_num=0]
Epoch 10:  55%|#####4    | 12/22 [00:04<00:03,  2.98it/s, loss=5.11, v_num=0]
Epoch 10:  55%|#####4    | 12/22 [00:04<00:03,  2.94it/s, loss=5.11, v_num=0]
Epoch 10:  59%|#####9    | 13/22 [00:04<00:02,  3.11it/s, loss=5.11, v_num=0]
Epoch 10:  59%|#####9    | 13/22 [00:04<00:02,  3.07it/s, loss=5.11, v_num=0]
Epoch 10:  64%|######3   | 14/22 [00:04<00:02,  3.24it/s, loss=5.11, v_num=0]
Epoch 10:  64%|######3   | 14/22 [00:04<00:02,  3.19it/s, loss=5.12, v_num=0]
Epoch 10:  68%|######8   | 15/22 [00:04<00:02,  3.35it/s, loss=5.12, v_num=0]
Epoch 10:  68%|######8   | 15/22 [00:04<00:02,  3.31it/s, loss=5.12, v_num=0]
Epoch 10:  73%|#######2  | 16/22 [00:04<00:01,  3.46it/s, loss=5.12, v_num=0]
Epoch 10:  73%|#######2  | 16/22 [00:04<00:01,  3.42it/s, loss=5.12, v_num=0]
Epoch 10:  77%|#######7  | 17/22 [00:04<00:01,  3.46it/s, loss=5.12, v_num=0]
Epoch 10:  77%|#######7  | 17/22 [00:04<00:01,  3.42it/s, loss=5.12, v_num=0]
Epoch 10:  82%|########1 | 18/22 [00:05<00:01,  3.55it/s, loss=5.12, v_num=0]
Epoch 10:  82%|########1 | 18/22 [00:05<00:01,  3.51it/s, loss=5.12, v_num=0]
Epoch 10:  86%|########6 | 19/22 [00:05<00:00,  3.64it/s, loss=5.12, v_num=0]
Epoch 10:  86%|########6 | 19/22 [00:05<00:00,  3.60it/s, loss=5.12, v_num=0]
Epoch 10:  91%|######### | 20/22 [00:05<00:00,  3.73it/s, loss=5.12, v_num=0]
Epoch 10:  91%|######### | 20/22 [00:05<00:00,  3.69it/s, loss=5.12, v_num=0]
Epoch 10:  95%|#########5| 21/22 [00:05<00:00,  3.81it/s, loss=5.12, v_num=0]
Epoch 10:  95%|#########5| 21/22 [00:05<00:00,  3.77it/s, loss=5.12, v_num=0]
Epoch 10: 100%|##########| 22/22 [00:05<00:00,  3.88it/s, loss=5.12, v_num=0]
Epoch 10: 100%|##########| 22/22 [00:05<00:00,  3.84it/s, loss=5.12, v_num=0]
Epoch 10: 100%|##########| 22/22 [00:05<00:00,  3.84it/s, loss=5.12, v_num=0]
Epoch 10:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.12, v_num=0]
Epoch 11:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.12, v_num=0]
Epoch 11:   5%|4         | 1/22 [00:02<00:44,  2.11s/it, loss=5.12, v_num=0]
Epoch 11:   5%|4         | 1/22 [00:02<00:45,  2.17s/it, loss=5.12, v_num=0]
Epoch 11:   9%|9         | 2/22 [00:02<00:22,  1.13s/it, loss=5.12, v_num=0]
Epoch 11:   9%|9         | 2/22 [00:02<00:23,  1.16s/it, loss=5.12, v_num=0]
Epoch 11:  14%|#3        | 3/22 [00:02<00:15,  1.24it/s, loss=5.12, v_num=0]
Epoch 11:  14%|#3        | 3/22 [00:02<00:15,  1.21it/s, loss=5.12, v_num=0]
Epoch 11:  18%|#8        | 4/22 [00:02<00:11,  1.55it/s, loss=5.12, v_num=0]
Epoch 11:  18%|#8        | 4/22 [00:02<00:11,  1.52it/s, loss=5.11, v_num=0]
Epoch 11:  23%|##2       | 5/22 [00:02<00:09,  1.84it/s, loss=5.11, v_num=0]
Epoch 11:  23%|##2       | 5/22 [00:02<00:09,  1.80it/s, loss=5.11, v_num=0]
Epoch 11:  27%|##7       | 6/22 [00:02<00:07,  2.09it/s, loss=5.11, v_num=0]
Epoch 11:  27%|##7       | 6/22 [00:02<00:07,  2.05it/s, loss=5.11, v_num=0]
Epoch 11:  32%|###1      | 7/22 [00:03<00:06,  2.32it/s, loss=5.11, v_num=0]
Epoch 11:  32%|###1      | 7/22 [00:03<00:06,  2.27it/s, loss=5.11, v_num=0]
Epoch 11:  36%|###6      | 8/22 [00:03<00:05,  2.52it/s, loss=5.11, v_num=0]
Epoch 11:  36%|###6      | 8/22 [00:03<00:05,  2.48it/s, loss=5.1, v_num=0]
Epoch 11:  41%|####      | 9/22 [00:03<00:04,  2.61it/s, loss=5.1, v_num=0]
Epoch 11:  41%|####      | 9/22 [00:03<00:05,  2.57it/s, loss=5.1, v_num=0]
Epoch 11:  45%|####5     | 10/22 [00:03<00:04,  2.78it/s, loss=5.1, v_num=0]
Epoch 11:  45%|####5     | 10/22 [00:03<00:04,  2.74it/s, loss=5.1, v_num=0]
Epoch 11:  50%|#####     | 11/22 [00:03<00:03,  2.94it/s, loss=5.1, v_num=0]
Epoch 11:  50%|#####     | 11/22 [00:03<00:03,  2.89it/s, loss=5.1, v_num=0]
Epoch 11:  55%|#####4    | 12/22 [00:03<00:03,  3.08it/s, loss=5.1, v_num=0]
Epoch 11:  55%|#####4    | 12/22 [00:03<00:03,  3.03it/s, loss=5.1, v_num=0]
Epoch 11:  59%|#####9    | 13/22 [00:04<00:02,  3.21it/s, loss=5.1, v_num=0]
Epoch 11:  59%|#####9    | 13/22 [00:04<00:02,  3.17it/s, loss=5.1, v_num=0]
Epoch 11:  64%|######3   | 14/22 [00:04<00:02,  3.34it/s, loss=5.1, v_num=0]
Epoch 11:  64%|######3   | 14/22 [00:04<00:02,  3.29it/s, loss=5.1, v_num=0]
Epoch 11:  68%|######8   | 15/22 [00:04<00:02,  3.45it/s, loss=5.1, v_num=0]
Epoch 11:  68%|######8   | 15/22 [00:04<00:02,  3.41it/s, loss=5.1, v_num=0]
Epoch 11:  73%|#######2  | 16/22 [00:04<00:01,  3.56it/s, loss=5.1, v_num=0]
Epoch 11:  73%|#######2  | 16/22 [00:04<00:01,  3.51it/s, loss=5.1, v_num=0]
Epoch 11:  77%|#######7  | 17/22 [00:04<00:01,  3.58it/s, loss=5.1, v_num=0]
Epoch 11:  77%|#######7  | 17/22 [00:04<00:01,  3.54it/s, loss=5.09, v_num=0]
Epoch 11:  82%|########1 | 18/22 [00:04<00:01,  3.68it/s, loss=5.09, v_num=0]
Epoch 11:  82%|########1 | 18/22 [00:04<00:01,  3.63it/s, loss=5.09, v_num=0]
Epoch 11:  86%|########6 | 19/22 [00:05<00:00,  3.77it/s, loss=5.09, v_num=0]
Epoch 11:  86%|########6 | 19/22 [00:05<00:00,  3.72it/s, loss=5.09, v_num=0]
Epoch 11:  91%|######### | 20/22 [00:05<00:00,  3.85it/s, loss=5.09, v_num=0]
Epoch 11:  91%|######### | 20/22 [00:05<00:00,  3.80it/s, loss=5.09, v_num=0]
Epoch 11:  95%|#########5| 21/22 [00:05<00:00,  3.93it/s, loss=5.09, v_num=0]
Epoch 11:  95%|#########5| 21/22 [00:05<00:00,  3.88it/s, loss=5.09, v_num=0]
Epoch 11: 100%|##########| 22/22 [00:05<00:00,  4.00it/s, loss=5.09, v_num=0]
Epoch 11: 100%|##########| 22/22 [00:05<00:00,  3.96it/s, loss=5.09, v_num=0]
Epoch 11: 100%|##########| 22/22 [00:05<00:00,  3.96it/s, loss=5.09, v_num=0]
Epoch 11:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.09, v_num=0]
Epoch 12:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.09, v_num=0]
Epoch 12:   5%|4         | 1/22 [00:02<00:45,  2.15s/it, loss=5.09, v_num=0]
Epoch 12:   5%|4         | 1/22 [00:02<00:46,  2.21s/it, loss=5.08, v_num=0]
Epoch 12:   9%|9         | 2/22 [00:02<00:23,  1.16s/it, loss=5.08, v_num=0]
Epoch 12:   9%|9         | 2/22 [00:02<00:23,  1.19s/it, loss=5.09, v_num=0]
Epoch 12:  14%|#3        | 3/22 [00:02<00:15,  1.22it/s, loss=5.09, v_num=0]
Epoch 12:  14%|#3        | 3/22 [00:02<00:15,  1.19it/s, loss=5.08, v_num=0]
Epoch 12:  18%|#8        | 4/22 [00:02<00:11,  1.53it/s, loss=5.08, v_num=0]
Epoch 12:  18%|#8        | 4/22 [00:02<00:12,  1.49it/s, loss=5.08, v_num=0]
Epoch 12:  23%|##2       | 5/22 [00:02<00:09,  1.80it/s, loss=5.08, v_num=0]
Epoch 12:  23%|##2       | 5/22 [00:02<00:09,  1.77it/s, loss=5.08, v_num=0]
Epoch 12:  27%|##7       | 6/22 [00:02<00:07,  2.05it/s, loss=5.08, v_num=0]
Epoch 12:  27%|##7       | 6/22 [00:02<00:07,  2.01it/s, loss=5.08, v_num=0]
Epoch 12:  32%|###1      | 7/22 [00:03<00:06,  2.27it/s, loss=5.08, v_num=0]
Epoch 12:  32%|###1      | 7/22 [00:03<00:06,  2.23it/s, loss=5.08, v_num=0]
Epoch 12:  36%|###6      | 8/22 [00:03<00:05,  2.47it/s, loss=5.08, v_num=0]
Epoch 12:  36%|###6      | 8/22 [00:03<00:05,  2.43it/s, loss=5.08, v_num=0]
Epoch 12:  41%|####      | 9/22 [00:03<00:05,  2.58it/s, loss=5.08, v_num=0]
Epoch 12:  41%|####      | 9/22 [00:03<00:05,  2.54it/s, loss=5.08, v_num=0]
Epoch 12:  45%|####5     | 10/22 [00:03<00:04,  2.73it/s, loss=5.08, v_num=0]
Epoch 12:  45%|####5     | 10/22 [00:03<00:04,  2.68it/s, loss=5.08, v_num=0]
Epoch 12:  50%|#####     | 11/22 [00:03<00:03,  2.88it/s, loss=5.08, v_num=0]
Epoch 12:  50%|#####     | 11/22 [00:03<00:03,  2.84it/s, loss=5.08, v_num=0]
Epoch 12:  55%|#####4    | 12/22 [00:03<00:03,  3.03it/s, loss=5.08, v_num=0]
Epoch 12:  55%|#####4    | 12/22 [00:04<00:03,  2.98it/s, loss=5.08, v_num=0]
Epoch 12:  59%|#####9    | 13/22 [00:04<00:02,  3.16it/s, loss=5.08, v_num=0]
Epoch 12:  59%|#####9    | 13/22 [00:04<00:02,  3.11it/s, loss=5.07, v_num=0]
Epoch 12:  64%|######3   | 14/22 [00:04<00:02,  3.28it/s, loss=5.07, v_num=0]
Epoch 12:  64%|######3   | 14/22 [00:04<00:02,  3.23it/s, loss=5.07, v_num=0]
Epoch 12:  68%|######8   | 15/22 [00:04<00:02,  3.39it/s, loss=5.07, v_num=0]
Epoch 12:  68%|######8   | 15/22 [00:04<00:02,  3.35it/s, loss=5.07, v_num=0]
Epoch 12:  73%|#######2  | 16/22 [00:04<00:01,  3.50it/s, loss=5.07, v_num=0]
Epoch 12:  73%|#######2  | 16/22 [00:04<00:01,  3.45it/s, loss=5.07, v_num=0]
Epoch 12:  77%|#######7  | 17/22 [00:04<00:01,  3.45it/s, loss=5.07, v_num=0]
Epoch 12:  77%|#######7  | 17/22 [00:04<00:01,  3.41it/s, loss=5.07, v_num=0]
Epoch 12:  82%|########1 | 18/22 [00:05<00:01,  3.54it/s, loss=5.07, v_num=0]
Epoch 12:  82%|########1 | 18/22 [00:05<00:01,  3.50it/s, loss=5.07, v_num=0]
Epoch 12:  86%|########6 | 19/22 [00:05<00:00,  3.63it/s, loss=5.07, v_num=0]
Epoch 12:  86%|########6 | 19/22 [00:05<00:00,  3.59it/s, loss=5.07, v_num=0]
Epoch 12:  91%|######### | 20/22 [00:05<00:00,  3.71it/s, loss=5.07, v_num=0]
Epoch 12:  91%|######### | 20/22 [00:05<00:00,  3.67it/s, loss=5.07, v_num=0]
Epoch 12:  95%|#########5| 21/22 [00:05<00:00,  3.79it/s, loss=5.07, v_num=0]
Epoch 12:  95%|#########5| 21/22 [00:05<00:00,  3.75it/s, loss=5.07, v_num=0]
Epoch 12: 100%|##########| 22/22 [00:05<00:00,  3.86it/s, loss=5.07, v_num=0]
Epoch 12: 100%|##########| 22/22 [00:05<00:00,  3.82it/s, loss=5.07, v_num=0]
Epoch 12: 100%|##########| 22/22 [00:05<00:00,  3.82it/s, loss=5.07, v_num=0]
Epoch 12:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.07, v_num=0]
Epoch 13:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.07, v_num=0]
Epoch 13:   5%|4         | 1/22 [00:02<00:45,  2.15s/it, loss=5.07, v_num=0]
Epoch 13:   5%|4         | 1/22 [00:02<00:46,  2.22s/it, loss=5.07, v_num=0]
Epoch 13:   9%|9         | 2/22 [00:02<00:23,  1.16s/it, loss=5.07, v_num=0]
Epoch 13:   9%|9         | 2/22 [00:02<00:23,  1.19s/it, loss=5.07, v_num=0]
Epoch 13:  14%|#3        | 3/22 [00:02<00:15,  1.22it/s, loss=5.07, v_num=0]
Epoch 13:  14%|#3        | 3/22 [00:02<00:16,  1.19it/s, loss=5.07, v_num=0]
Epoch 13:  18%|#8        | 4/22 [00:02<00:11,  1.53it/s, loss=5.07, v_num=0]
Epoch 13:  18%|#8        | 4/22 [00:02<00:12,  1.49it/s, loss=5.07, v_num=0]
Epoch 13:  23%|##2       | 5/22 [00:02<00:09,  1.80it/s, loss=5.07, v_num=0]
Epoch 13:  23%|##2       | 5/22 [00:02<00:09,  1.77it/s, loss=5.07, v_num=0]
Epoch 13:  27%|##7       | 6/22 [00:02<00:07,  2.05it/s, loss=5.07, v_num=0]
Epoch 13:  27%|##7       | 6/22 [00:02<00:07,  2.01it/s, loss=5.07, v_num=0]
Epoch 13:  32%|###1      | 7/22 [00:03<00:06,  2.28it/s, loss=5.07, v_num=0]
Epoch 13:  32%|###1      | 7/22 [00:03<00:06,  2.23it/s, loss=5.06, v_num=0]
Epoch 13:  36%|###6      | 8/22 [00:03<00:05,  2.47it/s, loss=5.06, v_num=0]
Epoch 13:  36%|###6      | 8/22 [00:03<00:05,  2.42it/s, loss=5.06, v_num=0]
Epoch 13:  41%|####      | 9/22 [00:03<00:05,  2.52it/s, loss=5.06, v_num=0]
Epoch 13:  41%|####      | 9/22 [00:03<00:05,  2.48it/s, loss=5.07, v_num=0]
Epoch 13:  45%|####5     | 10/22 [00:03<00:04,  2.69it/s, loss=5.07, v_num=0]
Epoch 13:  45%|####5     | 10/22 [00:03<00:04,  2.64it/s, loss=5.06, v_num=0]
Epoch 13:  50%|#####     | 11/22 [00:03<00:03,  2.84it/s, loss=5.06, v_num=0]
Epoch 13:  50%|#####     | 11/22 [00:03<00:03,  2.80it/s, loss=5.06, v_num=0]
Epoch 13:  55%|#####4    | 12/22 [00:04<00:03,  2.98it/s, loss=5.06, v_num=0]
Epoch 13:  55%|#####4    | 12/22 [00:04<00:03,  2.94it/s, loss=5.06, v_num=0]
Epoch 13:  59%|#####9    | 13/22 [00:04<00:02,  3.12it/s, loss=5.06, v_num=0]
Epoch 13:  59%|#####9    | 13/22 [00:04<00:02,  3.07it/s, loss=5.06, v_num=0]
Epoch 13:  64%|######3   | 14/22 [00:04<00:02,  3.24it/s, loss=5.06, v_num=0]
Epoch 13:  64%|######3   | 14/22 [00:04<00:02,  3.19it/s, loss=5.06, v_num=0]
Epoch 13:  68%|######8   | 15/22 [00:04<00:02,  3.35it/s, loss=5.06, v_num=0]
Epoch 13:  68%|######8   | 15/22 [00:04<00:02,  3.30it/s, loss=5.06, v_num=0]
Epoch 13:  73%|#######2  | 16/22 [00:04<00:01,  3.45it/s, loss=5.06, v_num=0]
Epoch 13:  73%|#######2  | 16/22 [00:04<00:01,  3.41it/s, loss=5.07, v_num=0]
Epoch 13:  77%|#######7  | 17/22 [00:04<00:01,  3.45it/s, loss=5.07, v_num=0]
Epoch 13:  77%|#######7  | 17/22 [00:04<00:01,  3.41it/s, loss=5.07, v_num=0]
Epoch 13:  82%|########1 | 18/22 [00:05<00:01,  3.55it/s, loss=5.07, v_num=0]
Epoch 13:  82%|########1 | 18/22 [00:05<00:01,  3.50it/s, loss=5.06, v_num=0]
Epoch 13:  86%|########6 | 19/22 [00:05<00:00,  3.63it/s, loss=5.06, v_num=0]
Epoch 13:  86%|########6 | 19/22 [00:05<00:00,  3.59it/s, loss=5.07, v_num=0]
Epoch 13:  91%|######### | 20/22 [00:05<00:00,  3.72it/s, loss=5.07, v_num=0]
Epoch 13:  91%|######### | 20/22 [00:05<00:00,  3.68it/s, loss=5.06, v_num=0]
Epoch 13:  95%|#########5| 21/22 [00:05<00:00,  3.80it/s, loss=5.06, v_num=0]
Epoch 13:  95%|#########5| 21/22 [00:05<00:00,  3.76it/s, loss=5.06, v_num=0]
Epoch 13: 100%|##########| 22/22 [00:05<00:00,  3.88it/s, loss=5.06, v_num=0]
Epoch 13: 100%|##########| 22/22 [00:05<00:00,  3.83it/s, loss=5.06, v_num=0]
Epoch 13: 100%|##########| 22/22 [00:05<00:00,  3.83it/s, loss=5.06, v_num=0]
Epoch 13:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.06, v_num=0]
Epoch 14:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.06, v_num=0]
Epoch 14:   5%|4         | 1/22 [00:02<00:46,  2.21s/it, loss=5.06, v_num=0]
Epoch 14:   5%|4         | 1/22 [00:02<00:47,  2.27s/it, loss=5.06, v_num=0]
Epoch 14:   9%|9         | 2/22 [00:02<00:23,  1.18s/it, loss=5.06, v_num=0]
Epoch 14:   9%|9         | 2/22 [00:02<00:24,  1.21s/it, loss=5.07, v_num=0]
Epoch 14:  14%|#3        | 3/22 [00:02<00:15,  1.19it/s, loss=5.07, v_num=0]
Epoch 14:  14%|#3        | 3/22 [00:02<00:16,  1.16it/s, loss=5.07, v_num=0]
Epoch 14:  18%|#8        | 4/22 [00:02<00:12,  1.50it/s, loss=5.07, v_num=0]
Epoch 14:  18%|#8        | 4/22 [00:02<00:12,  1.46it/s, loss=5.07, v_num=0]
Epoch 14:  23%|##2       | 5/22 [00:02<00:09,  1.77it/s, loss=5.07, v_num=0]
Epoch 14:  23%|##2       | 5/22 [00:02<00:09,  1.73it/s, loss=5.07, v_num=0]
Epoch 14:  27%|##7       | 6/22 [00:02<00:07,  2.01it/s, loss=5.07, v_num=0]
Epoch 14:  27%|##7       | 6/22 [00:03<00:08,  1.97it/s, loss=5.07, v_num=0]
Epoch 14:  32%|###1      | 7/22 [00:03<00:06,  2.24it/s, loss=5.07, v_num=0]
Epoch 14:  32%|###1      | 7/22 [00:03<00:06,  2.19it/s, loss=5.07, v_num=0]
Epoch 14:  36%|###6      | 8/22 [00:03<00:05,  2.44it/s, loss=5.07, v_num=0]
Epoch 14:  36%|###6      | 8/22 [00:03<00:05,  2.39it/s, loss=5.06, v_num=0]
Epoch 14:  41%|####      | 9/22 [00:03<00:05,  2.49it/s, loss=5.06, v_num=0]
Epoch 14:  41%|####      | 9/22 [00:03<00:05,  2.44it/s, loss=5.06, v_num=0]
Epoch 14:  45%|####5     | 10/22 [00:03<00:04,  2.65it/s, loss=5.06, v_num=0]
Epoch 14:  45%|####5     | 10/22 [00:03<00:04,  2.61it/s, loss=5.06, v_num=0]
Epoch 14:  50%|#####     | 11/22 [00:03<00:03,  2.80it/s, loss=5.06, v_num=0]
Epoch 14:  50%|#####     | 11/22 [00:03<00:03,  2.76it/s, loss=5.06, v_num=0]
Epoch 14:  55%|#####4    | 12/22 [00:04<00:03,  2.94it/s, loss=5.06, v_num=0]
Epoch 14:  55%|#####4    | 12/22 [00:04<00:03,  2.90it/s, loss=5.06, v_num=0]
Epoch 14:  59%|#####9    | 13/22 [00:04<00:02,  3.08it/s, loss=5.06, v_num=0]
Epoch 14:  59%|#####9    | 13/22 [00:04<00:02,  3.03it/s, loss=5.06, v_num=0]
Epoch 14:  64%|######3   | 14/22 [00:04<00:02,  3.20it/s, loss=5.06, v_num=0]
Epoch 14:  64%|######3   | 14/22 [00:04<00:02,  3.15it/s, loss=5.06, v_num=0]
Epoch 14:  68%|######8   | 15/22 [00:04<00:02,  3.32it/s, loss=5.06, v_num=0]
Epoch 14:  68%|######8   | 15/22 [00:04<00:02,  3.27it/s, loss=5.06, v_num=0]
Epoch 14:  73%|#######2  | 16/22 [00:04<00:01,  3.42it/s, loss=5.06, v_num=0]
Epoch 14:  73%|#######2  | 16/22 [00:04<00:01,  3.38it/s, loss=5.06, v_num=0]
Epoch 14:  77%|#######7  | 17/22 [00:04<00:01,  3.42it/s, loss=5.06, v_num=0]
Epoch 14:  77%|#######7  | 17/22 [00:05<00:01,  3.38it/s, loss=5.05, v_num=0]
Epoch 14:  82%|########1 | 18/22 [00:05<00:01,  3.52it/s, loss=5.05, v_num=0]
Epoch 14:  82%|########1 | 18/22 [00:05<00:01,  3.48it/s, loss=5.05, v_num=0]
Epoch 14:  86%|########6 | 19/22 [00:05<00:00,  3.61it/s, loss=5.05, v_num=0]
Epoch 14:  86%|########6 | 19/22 [00:05<00:00,  3.57it/s, loss=5.05, v_num=0]
Epoch 14:  91%|######### | 20/22 [00:05<00:00,  3.69it/s, loss=5.05, v_num=0]
Epoch 14:  91%|######### | 20/22 [00:05<00:00,  3.65it/s, loss=5.05, v_num=0]
Epoch 14:  95%|#########5| 21/22 [00:05<00:00,  3.78it/s, loss=5.05, v_num=0]
Epoch 14:  95%|#########5| 21/22 [00:05<00:00,  3.73it/s, loss=5.05, v_num=0]
Epoch 14: 100%|##########| 22/22 [00:05<00:00,  3.85it/s, loss=5.05, v_num=0]
Epoch 14: 100%|##########| 22/22 [00:05<00:00,  3.81it/s, loss=5.05, v_num=0]
Epoch 14: 100%|##########| 22/22 [00:05<00:00,  3.81it/s, loss=5.05, v_num=0]
Epoch 14:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.05, v_num=0]
Epoch 15:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.05, v_num=0]
Epoch 15:   5%|4         | 1/22 [00:02<00:45,  2.17s/it, loss=5.05, v_num=0]
Epoch 15:   5%|4         | 1/22 [00:02<00:46,  2.23s/it, loss=5.05, v_num=0]
Epoch 15:   9%|9         | 2/22 [00:02<00:23,  1.16s/it, loss=5.05, v_num=0]
Epoch 15:   9%|9         | 2/22 [00:02<00:23,  1.19s/it, loss=5.05, v_num=0]
Epoch 15:  14%|#3        | 3/22 [00:02<00:15,  1.21it/s, loss=5.05, v_num=0]
Epoch 15:  14%|#3        | 3/22 [00:02<00:16,  1.18it/s, loss=5.05, v_num=0]
Epoch 15:  18%|#8        | 4/22 [00:02<00:11,  1.52it/s, loss=5.05, v_num=0]
Epoch 15:  18%|#8        | 4/22 [00:02<00:12,  1.49it/s, loss=5.06, v_num=0]
Epoch 15:  23%|##2       | 5/22 [00:02<00:09,  1.80it/s, loss=5.06, v_num=0]
Epoch 15:  23%|##2       | 5/22 [00:02<00:09,  1.76it/s, loss=5.05, v_num=0]
Epoch 15:  27%|##7       | 6/22 [00:02<00:07,  2.04it/s, loss=5.05, v_num=0]
Epoch 15:  27%|##7       | 6/22 [00:02<00:07,  2.00it/s, loss=5.05, v_num=0]
Epoch 15:  32%|###1      | 7/22 [00:03<00:06,  2.27it/s, loss=5.05, v_num=0]
Epoch 15:  32%|###1      | 7/22 [00:03<00:06,  2.22it/s, loss=5.05, v_num=0]
Epoch 15:  36%|###6      | 8/22 [00:03<00:05,  2.47it/s, loss=5.05, v_num=0]
Epoch 15:  36%|###6      | 8/22 [00:03<00:05,  2.42it/s, loss=5.05, v_num=0]
Epoch 15:  41%|####      | 9/22 [00:03<00:05,  2.55it/s, loss=5.05, v_num=0]
Epoch 15:  41%|####      | 9/22 [00:03<00:05,  2.50it/s, loss=5.05, v_num=0]
Epoch 15:  45%|####5     | 10/22 [00:03<00:04,  2.71it/s, loss=5.05, v_num=0]
Epoch 15:  45%|####5     | 10/22 [00:03<00:04,  2.67it/s, loss=5.05, v_num=0]
Epoch 15:  50%|#####     | 11/22 [00:03<00:03,  2.87it/s, loss=5.05, v_num=0]
Epoch 15:  50%|#####     | 11/22 [00:03<00:03,  2.82it/s, loss=5.05, v_num=0]
Epoch 15:  55%|#####4    | 12/22 [00:03<00:03,  3.01it/s, loss=5.05, v_num=0]
Epoch 15:  55%|#####4    | 12/22 [00:04<00:03,  2.96it/s, loss=5.05, v_num=0]
Epoch 15:  59%|#####9    | 13/22 [00:04<00:02,  3.14it/s, loss=5.05, v_num=0]
Epoch 15:  59%|#####9    | 13/22 [00:04<00:02,  3.09it/s, loss=5.05, v_num=0]
Epoch 15:  64%|######3   | 14/22 [00:04<00:02,  3.26it/s, loss=5.05, v_num=0]
Epoch 15:  64%|######3   | 14/22 [00:04<00:02,  3.22it/s, loss=5.05, v_num=0]
Epoch 15:  68%|######8   | 15/22 [00:04<00:02,  3.38it/s, loss=5.05, v_num=0]
Epoch 15:  68%|######8   | 15/22 [00:04<00:02,  3.33it/s, loss=5.06, v_num=0]
Epoch 15:  73%|#######2  | 16/22 [00:04<00:01,  3.49it/s, loss=5.06, v_num=0]
Epoch 15:  73%|#######2  | 16/22 [00:04<00:01,  3.44it/s, loss=5.05, v_num=0]
Epoch 15:  77%|#######7  | 17/22 [00:04<00:01,  3.48it/s, loss=5.05, v_num=0]
Epoch 15:  77%|#######7  | 17/22 [00:04<00:01,  3.44it/s, loss=5.06, v_num=0]
Epoch 15:  82%|########1 | 18/22 [00:05<00:01,  3.58it/s, loss=5.06, v_num=0]
Epoch 15:  82%|########1 | 18/22 [00:05<00:01,  3.53it/s, loss=5.05, v_num=0]
Epoch 15:  86%|########6 | 19/22 [00:05<00:00,  3.67it/s, loss=5.05, v_num=0]
Epoch 15:  86%|########6 | 19/22 [00:05<00:00,  3.62it/s, loss=5.05, v_num=0]
Epoch 15:  91%|######### | 20/22 [00:05<00:00,  3.75it/s, loss=5.05, v_num=0]
Epoch 15:  91%|######### | 20/22 [00:05<00:00,  3.70it/s, loss=5.05, v_num=0]
Epoch 15:  95%|#########5| 21/22 [00:05<00:00,  3.83it/s, loss=5.05, v_num=0]
Epoch 15:  95%|#########5| 21/22 [00:05<00:00,  3.79it/s, loss=5.05, v_num=0]
Epoch 15: 100%|##########| 22/22 [00:05<00:00,  3.90it/s, loss=5.05, v_num=0]
Epoch 15: 100%|##########| 22/22 [00:05<00:00,  3.86it/s, loss=5.05, v_num=0]
Epoch 15: 100%|##########| 22/22 [00:05<00:00,  3.86it/s, loss=5.05, v_num=0]
Epoch 15:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.05, v_num=0]
Epoch 16:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.05, v_num=0]
Epoch 16:   5%|4         | 1/22 [00:02<00:44,  2.12s/it, loss=5.05, v_num=0]
Epoch 16:   5%|4         | 1/22 [00:02<00:45,  2.18s/it, loss=5.05, v_num=0]
Epoch 16:   9%|9         | 2/22 [00:02<00:22,  1.14s/it, loss=5.05, v_num=0]
Epoch 16:   9%|9         | 2/22 [00:02<00:23,  1.17s/it, loss=5.05, v_num=0]
Epoch 16:  14%|#3        | 3/22 [00:02<00:15,  1.23it/s, loss=5.05, v_num=0]
Epoch 16:  14%|#3        | 3/22 [00:02<00:15,  1.20it/s, loss=5.05, v_num=0]
Epoch 16:  18%|#8        | 4/22 [00:02<00:11,  1.55it/s, loss=5.05, v_num=0]
Epoch 16:  18%|#8        | 4/22 [00:02<00:11,  1.51it/s, loss=5.05, v_num=0]
Epoch 16:  23%|##2       | 5/22 [00:02<00:09,  1.83it/s, loss=5.05, v_num=0]
Epoch 16:  23%|##2       | 5/22 [00:02<00:09,  1.79it/s, loss=5.05, v_num=0]
Epoch 16:  27%|##7       | 6/22 [00:02<00:07,  2.08it/s, loss=5.05, v_num=0]
Epoch 16:  27%|##7       | 6/22 [00:02<00:07,  2.03it/s, loss=5.05, v_num=0]
Epoch 16:  32%|###1      | 7/22 [00:03<00:06,  2.30it/s, loss=5.05, v_num=0]
Epoch 16:  32%|###1      | 7/22 [00:03<00:06,  2.26it/s, loss=5.05, v_num=0]
Epoch 16:  36%|###6      | 8/22 [00:03<00:05,  2.51it/s, loss=5.05, v_num=0]
Epoch 16:  36%|###6      | 8/22 [00:03<00:05,  2.46it/s, loss=5.05, v_num=0]
Epoch 16:  41%|####      | 9/22 [00:03<00:04,  2.60it/s, loss=5.05, v_num=0]
Epoch 16:  41%|####      | 9/22 [00:03<00:05,  2.56it/s, loss=5.05, v_num=0]
Epoch 16:  45%|####5     | 10/22 [00:03<00:04,  2.77it/s, loss=5.05, v_num=0]
Epoch 16:  45%|####5     | 10/22 [00:03<00:04,  2.72it/s, loss=5.05, v_num=0]
Epoch 16:  50%|#####     | 11/22 [00:03<00:03,  2.93it/s, loss=5.05, v_num=0]
Epoch 16:  50%|#####     | 11/22 [00:03<00:03,  2.88it/s, loss=5.05, v_num=0]
Epoch 16:  55%|#####4    | 12/22 [00:03<00:03,  3.06it/s, loss=5.05, v_num=0]
Epoch 16:  55%|#####4    | 12/22 [00:03<00:03,  3.02it/s, loss=5.05, v_num=0]
Epoch 16:  59%|#####9    | 13/22 [00:04<00:02,  3.20it/s, loss=5.05, v_num=0]
Epoch 16:  59%|#####9    | 13/22 [00:04<00:02,  3.15it/s, loss=5.05, v_num=0]
Epoch 16:  64%|######3   | 14/22 [00:04<00:02,  3.32it/s, loss=5.05, v_num=0]
Epoch 16:  64%|######3   | 14/22 [00:04<00:02,  3.27it/s, loss=5.05, v_num=0]
Epoch 16:  68%|######8   | 15/22 [00:04<00:02,  3.43it/s, loss=5.05, v_num=0]
Epoch 16:  68%|######8   | 15/22 [00:04<00:02,  3.38it/s, loss=5.05, v_num=0]
Epoch 16:  73%|#######2  | 16/22 [00:04<00:01,  3.54it/s, loss=5.05, v_num=0]
Epoch 16:  73%|#######2  | 16/22 [00:04<00:01,  3.49it/s, loss=5.05, v_num=0]
Epoch 16:  77%|#######7  | 17/22 [00:04<00:01,  3.59it/s, loss=5.05, v_num=0]
Epoch 16:  77%|#######7  | 17/22 [00:04<00:01,  3.54it/s, loss=5.04, v_num=0]
Epoch 16:  82%|########1 | 18/22 [00:04<00:01,  3.68it/s, loss=5.04, v_num=0]
Epoch 16:  82%|########1 | 18/22 [00:04<00:01,  3.63it/s, loss=5.05, v_num=0]
Epoch 16:  86%|########6 | 19/22 [00:05<00:00,  3.77it/s, loss=5.05, v_num=0]
Epoch 16:  86%|########6 | 19/22 [00:05<00:00,  3.72it/s, loss=5.04, v_num=0]
Epoch 16:  91%|######### | 20/22 [00:05<00:00,  3.86it/s, loss=5.04, v_num=0]
Epoch 16:  91%|######### | 20/22 [00:05<00:00,  3.81it/s, loss=5.04, v_num=0]
Epoch 16:  95%|#########5| 21/22 [00:05<00:00,  3.93it/s, loss=5.04, v_num=0]
Epoch 16:  95%|#########5| 21/22 [00:05<00:00,  3.89it/s, loss=5.04, v_num=0]
Epoch 16: 100%|##########| 22/22 [00:05<00:00,  4.01it/s, loss=5.04, v_num=0]
Epoch 16: 100%|##########| 22/22 [00:05<00:00,  3.96it/s, loss=5.04, v_num=0]
Epoch 16: 100%|##########| 22/22 [00:05<00:00,  3.96it/s, loss=5.04, v_num=0]
Epoch 16:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.04, v_num=0]
Epoch 17:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.04, v_num=0]
Epoch 17:   5%|4         | 1/22 [00:02<00:45,  2.17s/it, loss=5.04, v_num=0]
Epoch 17:   5%|4         | 1/22 [00:02<00:46,  2.23s/it, loss=5.04, v_num=0]
Epoch 17:   9%|9         | 2/22 [00:02<00:23,  1.17s/it, loss=5.04, v_num=0]
Epoch 17:   9%|9         | 2/22 [00:02<00:23,  1.19s/it, loss=5.04, v_num=0]
Epoch 17:  14%|#3        | 3/22 [00:02<00:15,  1.21it/s, loss=5.04, v_num=0]
Epoch 17:  14%|#3        | 3/22 [00:02<00:16,  1.18it/s, loss=5.04, v_num=0]
Epoch 17:  18%|#8        | 4/22 [00:02<00:11,  1.52it/s, loss=5.04, v_num=0]
Epoch 17:  18%|#8        | 4/22 [00:02<00:12,  1.48it/s, loss=5.04, v_num=0]
Epoch 17:  23%|##2       | 5/22 [00:02<00:09,  1.79it/s, loss=5.04, v_num=0]
Epoch 17:  23%|##2       | 5/22 [00:02<00:09,  1.76it/s, loss=5.04, v_num=0]
Epoch 17:  27%|##7       | 6/22 [00:02<00:07,  2.04it/s, loss=5.04, v_num=0]
Epoch 17:  27%|##7       | 6/22 [00:02<00:07,  2.00it/s, loss=5.04, v_num=0]
Epoch 17:  32%|###1      | 7/22 [00:03<00:06,  2.27it/s, loss=5.04, v_num=0]
Epoch 17:  32%|###1      | 7/22 [00:03<00:06,  2.22it/s, loss=5.04, v_num=0]
Epoch 17:  36%|###6      | 8/22 [00:03<00:05,  2.47it/s, loss=5.04, v_num=0]
Epoch 17:  36%|###6      | 8/22 [00:03<00:05,  2.42it/s, loss=5.04, v_num=0]
Epoch 17:  41%|####      | 9/22 [00:03<00:05,  2.55it/s, loss=5.04, v_num=0]
Epoch 17:  41%|####      | 9/22 [00:03<00:05,  2.51it/s, loss=5.04, v_num=0]
Epoch 17:  45%|####5     | 10/22 [00:03<00:04,  2.72it/s, loss=5.04, v_num=0]
Epoch 17:  45%|####5     | 10/22 [00:03<00:04,  2.67it/s, loss=5.04, v_num=0]
Epoch 17:  50%|#####     | 11/22 [00:03<00:03,  2.87it/s, loss=5.04, v_num=0]
Epoch 17:  50%|#####     | 11/22 [00:03<00:03,  2.83it/s, loss=5.04, v_num=0]
Epoch 17:  55%|#####4    | 12/22 [00:03<00:03,  3.02it/s, loss=5.04, v_num=0]
Epoch 17:  55%|#####4    | 12/22 [00:04<00:03,  2.97it/s, loss=5.04, v_num=0]
Epoch 17:  59%|#####9    | 13/22 [00:04<00:02,  3.14it/s, loss=5.04, v_num=0]
Epoch 17:  59%|#####9    | 13/22 [00:04<00:02,  3.10it/s, loss=5.04, v_num=0]
Epoch 17:  64%|######3   | 14/22 [00:04<00:02,  3.27it/s, loss=5.04, v_num=0]
Epoch 17:  64%|######3   | 14/22 [00:04<00:02,  3.22it/s, loss=5.03, v_num=0]
Epoch 17:  68%|######8   | 15/22 [00:04<00:02,  3.38it/s, loss=5.03, v_num=0]
Epoch 17:  68%|######8   | 15/22 [00:04<00:02,  3.34it/s, loss=5.04, v_num=0]
Epoch 17:  73%|#######2  | 16/22 [00:04<00:01,  3.49it/s, loss=5.04, v_num=0]
Epoch 17:  73%|#######2  | 16/22 [00:04<00:01,  3.44it/s, loss=5.03, v_num=0]
Epoch 17:  77%|#######7  | 17/22 [00:04<00:01,  3.51it/s, loss=5.03, v_num=0]
Epoch 17:  77%|#######7  | 17/22 [00:04<00:01,  3.47it/s, loss=5.03, v_num=0]
Epoch 17:  82%|########1 | 18/22 [00:04<00:01,  3.60it/s, loss=5.03, v_num=0]
Epoch 17:  82%|########1 | 18/22 [00:05<00:01,  3.56it/s, loss=5.04, v_num=0]
Epoch 17:  86%|########6 | 19/22 [00:05<00:00,  3.69it/s, loss=5.04, v_num=0]
Epoch 17:  86%|########6 | 19/22 [00:05<00:00,  3.65it/s, loss=5.04, v_num=0]
Epoch 17:  91%|######### | 20/22 [00:05<00:00,  3.78it/s, loss=5.04, v_num=0]
Epoch 17:  91%|######### | 20/22 [00:05<00:00,  3.73it/s, loss=5.04, v_num=0]
Epoch 17:  95%|#########5| 21/22 [00:05<00:00,  3.86it/s, loss=5.04, v_num=0]
Epoch 17:  95%|#########5| 21/22 [00:05<00:00,  3.81it/s, loss=5.03, v_num=0]
Epoch 17: 100%|##########| 22/22 [00:05<00:00,  3.93it/s, loss=5.03, v_num=0]
Epoch 17: 100%|##########| 22/22 [00:05<00:00,  3.89it/s, loss=5.03, v_num=0]
Epoch 17: 100%|##########| 22/22 [00:05<00:00,  3.89it/s, loss=5.03, v_num=0]
Epoch 17:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.03, v_num=0]
Epoch 18:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.03, v_num=0]
Epoch 18:   5%|4         | 1/22 [00:02<00:44,  2.12s/it, loss=5.03, v_num=0]
Epoch 18:   5%|4         | 1/22 [00:02<00:45,  2.19s/it, loss=5.03, v_num=0]
Epoch 18:   9%|9         | 2/22 [00:02<00:22,  1.14s/it, loss=5.03, v_num=0]
Epoch 18:   9%|9         | 2/22 [00:02<00:23,  1.17s/it, loss=5.03, v_num=0]
Epoch 18:  14%|#3        | 3/22 [00:02<00:15,  1.23it/s, loss=5.03, v_num=0]
Epoch 18:  14%|#3        | 3/22 [00:02<00:15,  1.20it/s, loss=5.03, v_num=0]
Epoch 18:  18%|#8        | 4/22 [00:02<00:11,  1.54it/s, loss=5.03, v_num=0]
Epoch 18:  18%|#8        | 4/22 [00:02<00:11,  1.51it/s, loss=5.03, v_num=0]
Epoch 18:  23%|##2       | 5/22 [00:02<00:09,  1.82it/s, loss=5.03, v_num=0]
Epoch 18:  23%|##2       | 5/22 [00:02<00:09,  1.78it/s, loss=5.03, v_num=0]
Epoch 18:  27%|##7       | 6/22 [00:02<00:07,  2.08it/s, loss=5.03, v_num=0]
Epoch 18:  27%|##7       | 6/22 [00:02<00:07,  2.03it/s, loss=5.03, v_num=0]
Epoch 18:  32%|###1      | 7/22 [00:03<00:06,  2.30it/s, loss=5.03, v_num=0]
Epoch 18:  32%|###1      | 7/22 [00:03<00:06,  2.26it/s, loss=5.03, v_num=0]
Epoch 18:  36%|###6      | 8/22 [00:03<00:05,  2.51it/s, loss=5.03, v_num=0]
Epoch 18:  36%|###6      | 8/22 [00:03<00:05,  2.46it/s, loss=5.02, v_num=0]
Epoch 18:  41%|####      | 9/22 [00:03<00:05,  2.56it/s, loss=5.02, v_num=0]
Epoch 18:  41%|####      | 9/22 [00:03<00:05,  2.52it/s, loss=5.02, v_num=0]
Epoch 18:  45%|####5     | 10/22 [00:03<00:04,  2.73it/s, loss=5.02, v_num=0]
Epoch 18:  45%|####5     | 10/22 [00:03<00:04,  2.68it/s, loss=5.02, v_num=0]
Epoch 18:  50%|#####     | 11/22 [00:03<00:03,  2.88it/s, loss=5.02, v_num=0]
Epoch 18:  50%|#####     | 11/22 [00:03<00:03,  2.84it/s, loss=5.03, v_num=0]
Epoch 18:  55%|#####4    | 12/22 [00:03<00:03,  3.02it/s, loss=5.03, v_num=0]
Epoch 18:  55%|#####4    | 12/22 [00:04<00:03,  2.98it/s, loss=5.03, v_num=0]
Epoch 18:  59%|#####9    | 13/22 [00:04<00:02,  3.16it/s, loss=5.03, v_num=0]
Epoch 18:  59%|#####9    | 13/22 [00:04<00:02,  3.11it/s, loss=5.03, v_num=0]
Epoch 18:  64%|######3   | 14/22 [00:04<00:02,  3.27it/s, loss=5.03, v_num=0]
Epoch 18:  64%|######3   | 14/22 [00:04<00:02,  3.23it/s, loss=5.03, v_num=0]
Epoch 18:  68%|######8   | 15/22 [00:04<00:02,  3.39it/s, loss=5.03, v_num=0]
Epoch 18:  68%|######8   | 15/22 [00:04<00:02,  3.34it/s, loss=5.03, v_num=0]
Epoch 18:  73%|#######2  | 16/22 [00:04<00:01,  3.49it/s, loss=5.03, v_num=0]
Epoch 18:  73%|#######2  | 16/22 [00:04<00:01,  3.44it/s, loss=5.03, v_num=0]
Epoch 18:  77%|#######7  | 17/22 [00:04<00:01,  3.48it/s, loss=5.03, v_num=0]
Epoch 18:  77%|#######7  | 17/22 [00:04<00:01,  3.43it/s, loss=5.02, v_num=0]
Epoch 18:  82%|########1 | 18/22 [00:05<00:01,  3.57it/s, loss=5.02, v_num=0]
Epoch 18:  82%|########1 | 18/22 [00:05<00:01,  3.53it/s, loss=5.03, v_num=0]
Epoch 18:  86%|########6 | 19/22 [00:05<00:00,  3.66it/s, loss=5.03, v_num=0]
Epoch 18:  86%|########6 | 19/22 [00:05<00:00,  3.62it/s, loss=5.03, v_num=0]
Epoch 18:  91%|######### | 20/22 [00:05<00:00,  3.74it/s, loss=5.03, v_num=0]
Epoch 18:  91%|######### | 20/22 [00:05<00:00,  3.70it/s, loss=5.03, v_num=0]
Epoch 18:  95%|#########5| 21/22 [00:05<00:00,  3.82it/s, loss=5.03, v_num=0]
Epoch 18:  95%|#########5| 21/22 [00:05<00:00,  3.78it/s, loss=5.03, v_num=0]
Epoch 18: 100%|##########| 22/22 [00:05<00:00,  3.90it/s, loss=5.03, v_num=0]
Epoch 18: 100%|##########| 22/22 [00:05<00:00,  3.86it/s, loss=5.03, v_num=0]
Epoch 18: 100%|##########| 22/22 [00:05<00:00,  3.86it/s, loss=5.03, v_num=0]
Epoch 18:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.03, v_num=0]
Epoch 19:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.03, v_num=0]
Epoch 19:   5%|4         | 1/22 [00:02<00:45,  2.16s/it, loss=5.03, v_num=0]
Epoch 19:   5%|4         | 1/22 [00:02<00:46,  2.22s/it, loss=5.02, v_num=0]
Epoch 19:   9%|9         | 2/22 [00:02<00:23,  1.16s/it, loss=5.02, v_num=0]
Epoch 19:   9%|9         | 2/22 [00:02<00:23,  1.19s/it, loss=5.02, v_num=0]
Epoch 19:  14%|#3        | 3/22 [00:02<00:15,  1.22it/s, loss=5.02, v_num=0]
Epoch 19:  14%|#3        | 3/22 [00:02<00:16,  1.19it/s, loss=5.02, v_num=0]
Epoch 19:  18%|#8        | 4/22 [00:02<00:11,  1.53it/s, loss=5.02, v_num=0]
Epoch 19:  18%|#8        | 4/22 [00:02<00:12,  1.49it/s, loss=5.02, v_num=0]
Epoch 19:  23%|##2       | 5/22 [00:02<00:09,  1.81it/s, loss=5.02, v_num=0]
Epoch 19:  23%|##2       | 5/22 [00:02<00:09,  1.77it/s, loss=5.02, v_num=0]
Epoch 19:  27%|##7       | 6/22 [00:02<00:07,  2.06it/s, loss=5.02, v_num=0]
Epoch 19:  27%|##7       | 6/22 [00:02<00:07,  2.01it/s, loss=5.02, v_num=0]
Epoch 19:  32%|###1      | 7/22 [00:03<00:06,  2.28it/s, loss=5.02, v_num=0]
Epoch 19:  32%|###1      | 7/22 [00:03<00:06,  2.24it/s, loss=5.02, v_num=0]
Epoch 19:  36%|###6      | 8/22 [00:03<00:05,  2.48it/s, loss=5.02, v_num=0]
Epoch 19:  36%|###6      | 8/22 [00:03<00:05,  2.44it/s, loss=5.02, v_num=0]
Epoch 19:  41%|####      | 9/22 [00:03<00:05,  2.57it/s, loss=5.02, v_num=0]
Epoch 19:  41%|####      | 9/22 [00:03<00:05,  2.53it/s, loss=5.02, v_num=0]
Epoch 19:  45%|####5     | 10/22 [00:03<00:04,  2.74it/s, loss=5.02, v_num=0]
Epoch 19:  45%|####5     | 10/22 [00:03<00:04,  2.69it/s, loss=5.02, v_num=0]
Epoch 19:  50%|#####     | 11/22 [00:03<00:03,  2.89it/s, loss=5.02, v_num=0]
Epoch 19:  50%|#####     | 11/22 [00:03<00:03,  2.84it/s, loss=5.02, v_num=0]
Epoch 19:  55%|#####4    | 12/22 [00:03<00:03,  3.03it/s, loss=5.02, v_num=0]
Epoch 19:  55%|#####4    | 12/22 [00:04<00:03,  2.99it/s, loss=5.02, v_num=0]
Epoch 19:  59%|#####9    | 13/22 [00:04<00:02,  3.17it/s, loss=5.02, v_num=0]
Epoch 19:  59%|#####9    | 13/22 [00:04<00:02,  3.12it/s, loss=5.02, v_num=0]
Epoch 19:  64%|######3   | 14/22 [00:04<00:02,  3.29it/s, loss=5.02, v_num=0]
Epoch 19:  64%|######3   | 14/22 [00:04<00:02,  3.24it/s, loss=5.02, v_num=0]
Epoch 19:  68%|######8   | 15/22 [00:04<00:02,  3.41it/s, loss=5.02, v_num=0]
Epoch 19:  68%|######8   | 15/22 [00:04<00:02,  3.36it/s, loss=5.02, v_num=0]
Epoch 19:  73%|#######2  | 16/22 [00:04<00:01,  3.52it/s, loss=5.02, v_num=0]
Epoch 19:  73%|#######2  | 16/22 [00:04<00:01,  3.47it/s, loss=5.01, v_num=0]
Epoch 19:  77%|#######7  | 17/22 [00:04<00:01,  3.52it/s, loss=5.01, v_num=0]
Epoch 19:  77%|#######7  | 17/22 [00:04<00:01,  3.47it/s, loss=5.02, v_num=0]
Epoch 19:  82%|########1 | 18/22 [00:04<00:01,  3.61it/s, loss=5.02, v_num=0]
Epoch 19:  82%|########1 | 18/22 [00:05<00:01,  3.57it/s, loss=5.02, v_num=0]
Epoch 19:  86%|########6 | 19/22 [00:05<00:00,  3.70it/s, loss=5.02, v_num=0]
Epoch 19:  86%|########6 | 19/22 [00:05<00:00,  3.66it/s, loss=5.02, v_num=0]
Epoch 19:  91%|######### | 20/22 [00:05<00:00,  3.78it/s, loss=5.02, v_num=0]
Epoch 19:  91%|######### | 20/22 [00:05<00:00,  3.74it/s, loss=5.02, v_num=0]
Epoch 19:  95%|#########5| 21/22 [00:05<00:00,  3.86it/s, loss=5.02, v_num=0]
Epoch 19:  95%|#########5| 21/22 [00:05<00:00,  3.82it/s, loss=5.02, v_num=0]
Epoch 19: 100%|##########| 22/22 [00:05<00:00,  3.94it/s, loss=5.02, v_num=0]
Epoch 19: 100%|##########| 22/22 [00:05<00:00,  3.90it/s, loss=5.03, v_num=0]
Epoch 19: 100%|##########| 22/22 [00:05<00:00,  3.90it/s, loss=5.03, v_num=0]
Epoch 19: 100%|##########| 22/22 [00:05<00:00,  3.80it/s, loss=5.03, v_num=0]

Next we create a helper function to generate embeddings from our test images using the model we just trained. Note that only the backbone is needed to generate embeddings, the projection head is only required for the training. Make sure to put the model into eval mode for this part!

def generate_embeddings(model, dataloader):
    """Generates representations for all images in the dataloader with
    the given model
    """

    embeddings = []
    filenames = []
    with torch.no_grad():
        for img, label, fnames in dataloader:
            img = img.to(model.device)
            emb = model.backbone(img).flatten(start_dim=1)
            embeddings.append(emb)
            filenames.extend(fnames)

    embeddings = torch.cat(embeddings, 0)
    embeddings = normalize(embeddings)
    return embeddings, filenames


model.eval()
embeddings, filenames = generate_embeddings(model, dataloader_test)

Visualize Nearest Neighbors

Let’s look at the trained embedding and visualize the nearest neighbors for a few random samples.

We create some helper functions to simplify the work

def get_image_as_np_array(filename: str):
    """Returns an image as an numpy array"""
    img = Image.open(filename)
    return np.asarray(img)


def plot_knn_examples(embeddings, filenames, n_neighbors=3, num_examples=6):
    """Plots multiple rows of random images with their nearest neighbors"""
    # lets look at the nearest neighbors for some samples
    # we use the sklearn library
    nbrs = NearestNeighbors(n_neighbors=n_neighbors).fit(embeddings)
    distances, indices = nbrs.kneighbors(embeddings)

    # get 5 random samples
    samples_idx = np.random.choice(len(indices), size=num_examples, replace=False)

    # loop through our randomly picked samples
    for idx in samples_idx:
        fig = plt.figure()
        # loop through their nearest neighbors
        for plot_x_offset, neighbor_idx in enumerate(indices[idx]):
            # add the subplot
            ax = fig.add_subplot(1, len(indices[idx]), plot_x_offset + 1)
            # get the correponding filename for the current index
            fname = os.path.join(path_to_data, filenames[neighbor_idx])
            # plot the image
            plt.imshow(get_image_as_np_array(fname))
            # set the title to the distance of the neighbor
            ax.set_title(f"d={distances[idx][plot_x_offset]:.3f}")
            # let's disable the axis
            plt.axis("off")

Let’s do the plot of the images. The leftmost image is the query image whereas the ones next to it on the same row are the nearest neighbors. In the title we see the distance of the neigbor.

plot_knn_examples(embeddings, filenames)
  • d=0.000, d=0.154, d=0.182
  • d=0.000, d=0.149, d=0.154
  • d=0.000, d=0.222, d=0.225
  • d=0.000, d=0.214, d=0.214
  • d=0.000, d=0.173, d=0.222
  • d=0.000, d=0.223, d=0.228

Color Invariance

Let’s train again without color augmentation. This will force our model to respect the colors in the images.

# Set color jitter and gray scale probability to 0
new_collate_fn = SimCLRCollateFunction(
    input_size=input_size, vf_prob=0.5, rr_prob=0.5, cj_prob=0.0, random_gray_scale=0.0
)

# let's update our collate method and reuse our dataloader
dataloader_train_simclr.collate_fn = new_collate_fn

# then train a new model
model = SimCLRModel()
trainer = pl.Trainer(max_epochs=max_epochs, devices=1, accelerator="gpu")
trainer.fit(model, dataloader_train_simclr)

# and generate again embeddings from the test set
model.eval()
embeddings, filenames = generate_embeddings(model, dataloader_test)
/opt/runner_04/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1555: PossibleUserWarning: The number of training batches (22) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
  rank_zero_warn(

Training: 0it [00:00, ?it/s]
Training:   0%|          | 0/22 [00:00<?, ?it/s]
Epoch 0:   0%|          | 0/22 [00:00<?, ?it/s]
Epoch 0:   5%|4         | 1/22 [00:01<00:32,  1.54s/it]
Epoch 0:   5%|4         | 1/22 [00:01<00:33,  1.59s/it, loss=6.17, v_num=1]
Epoch 0:   9%|9         | 2/22 [00:01<00:16,  1.18it/s, loss=6.17, v_num=1]
Epoch 0:   9%|9         | 2/22 [00:01<00:17,  1.14it/s, loss=6.15, v_num=1]
Epoch 0:  14%|#3        | 3/22 [00:01<00:11,  1.63it/s, loss=6.15, v_num=1]
Epoch 0:  14%|#3        | 3/22 [00:01<00:12,  1.57it/s, loss=6.06, v_num=1]
Epoch 0:  18%|#8        | 4/22 [00:01<00:08,  2.00it/s, loss=6.06, v_num=1]
Epoch 0:  18%|#8        | 4/22 [00:02<00:09,  1.94it/s, loss=5.97, v_num=1]
Epoch 0:  23%|##2       | 5/22 [00:02<00:07,  2.33it/s, loss=5.97, v_num=1]
Epoch 0:  23%|##2       | 5/22 [00:02<00:07,  2.26it/s, loss=5.88, v_num=1]
Epoch 0:  27%|##7       | 6/22 [00:02<00:06,  2.61it/s, loss=5.88, v_num=1]
Epoch 0:  27%|##7       | 6/22 [00:02<00:06,  2.54it/s, loss=5.79, v_num=1]
Epoch 0:  32%|###1      | 7/22 [00:02<00:05,  2.85it/s, loss=5.79, v_num=1]
Epoch 0:  32%|###1      | 7/22 [00:02<00:05,  2.78it/s, loss=5.72, v_num=1]
Epoch 0:  36%|###6      | 8/22 [00:02<00:04,  3.07it/s, loss=5.72, v_num=1]
Epoch 0:  36%|###6      | 8/22 [00:02<00:04,  3.00it/s, loss=5.65, v_num=1]
Epoch 0:  41%|####      | 9/22 [00:02<00:03,  3.26it/s, loss=5.65, v_num=1]
Epoch 0:  41%|####      | 9/22 [00:02<00:04,  3.19it/s, loss=5.59, v_num=1]
Epoch 0:  45%|####5     | 10/22 [00:02<00:03,  3.44it/s, loss=5.59, v_num=1]
Epoch 0:  45%|####5     | 10/22 [00:02<00:03,  3.37it/s, loss=5.54, v_num=1]
Epoch 0:  50%|#####     | 11/22 [00:03<00:03,  3.59it/s, loss=5.54, v_num=1]
Epoch 0:  50%|#####     | 11/22 [00:03<00:03,  3.52it/s, loss=5.49, v_num=1]
Epoch 0:  55%|#####4    | 12/22 [00:03<00:02,  3.73it/s, loss=5.49, v_num=1]
Epoch 0:  55%|#####4    | 12/22 [00:03<00:02,  3.66it/s, loss=5.45, v_num=1]
Epoch 0:  59%|#####9    | 13/22 [00:03<00:02,  3.86it/s, loss=5.45, v_num=1]
Epoch 0:  59%|#####9    | 13/22 [00:03<00:02,  3.79it/s, loss=5.41, v_num=1]
Epoch 0:  64%|######3   | 14/22 [00:03<00:02,  3.98it/s, loss=5.41, v_num=1]
Epoch 0:  64%|######3   | 14/22 [00:03<00:02,  3.91it/s, loss=5.38, v_num=1]
Epoch 0:  68%|######8   | 15/22 [00:03<00:01,  4.09it/s, loss=5.38, v_num=1]
Epoch 0:  68%|######8   | 15/22 [00:03<00:01,  4.02it/s, loss=5.35, v_num=1]
Epoch 0:  73%|#######2  | 16/22 [00:03<00:01,  4.19it/s, loss=5.35, v_num=1]
Epoch 0:  73%|#######2  | 16/22 [00:03<00:01,  4.13it/s, loss=5.32, v_num=1]
Epoch 0:  77%|#######7  | 17/22 [00:03<00:01,  4.29it/s, loss=5.32, v_num=1]
Epoch 0:  77%|#######7  | 17/22 [00:04<00:01,  4.22it/s, loss=5.3, v_num=1]
Epoch 0:  82%|########1 | 18/22 [00:04<00:00,  4.38it/s, loss=5.3, v_num=1]
Epoch 0:  82%|########1 | 18/22 [00:04<00:00,  4.31it/s, loss=5.28, v_num=1]
Epoch 0:  86%|########6 | 19/22 [00:04<00:00,  4.46it/s, loss=5.28, v_num=1]
Epoch 0:  86%|########6 | 19/22 [00:04<00:00,  4.39it/s, loss=5.25, v_num=1]
Epoch 0:  91%|######### | 20/22 [00:04<00:00,  4.54it/s, loss=5.25, v_num=1]
Epoch 0:  91%|######### | 20/22 [00:04<00:00,  4.47it/s, loss=5.23, v_num=1]
Epoch 0:  95%|#########5| 21/22 [00:04<00:00,  4.61it/s, loss=5.23, v_num=1]
Epoch 0:  95%|#########5| 21/22 [00:04<00:00,  4.55it/s, loss=5.17, v_num=1]
Epoch 0: 100%|##########| 22/22 [00:04<00:00,  4.68it/s, loss=5.17, v_num=1]
Epoch 0: 100%|##########| 22/22 [00:04<00:00,  4.62it/s, loss=5.1, v_num=1]
Epoch 0: 100%|##########| 22/22 [00:04<00:00,  4.62it/s, loss=5.1, v_num=1]
Epoch 0:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.1, v_num=1]
Epoch 1:   0%|          | 0/22 [00:00<?, ?it/s, loss=5.1, v_num=1]
Epoch 1:   5%|4         | 1/22 [00:01<00:36,  1.74s/it, loss=5.1, v_num=1]
Epoch 1:   5%|4         | 1/22 [00:01<00:37,  1.80s/it, loss=5.05, v_num=1]
Epoch 1:   9%|9         | 2/22 [00:01<00:18,  1.06it/s, loss=5.05, v_num=1]
Epoch 1:   9%|9         | 2/22 [00:01<00:19,  1.02it/s, loss=5, v_num=1]
Epoch 1:  14%|#3        | 3/22 [00:02<00:12,  1.47it/s, loss=5, v_num=1]
Epoch 1:  14%|#3        | 3/22 [00:02<00:13,  1.43it/s, loss=4.97, v_num=1]
Epoch 1:  18%|#8        | 4/22 [00:02<00:09,  1.82it/s, loss=4.97, v_num=1]
Epoch 1:  18%|#8        | 4/22 [00:02<00:10,  1.77it/s, loss=4.94, v_num=1]
Epoch 1:  23%|##2       | 5/22 [00:02<00:08,  2.12it/s, loss=4.94, v_num=1]
Epoch 1:  23%|##2       | 5/22 [00:02<00:08,  2.07it/s, loss=4.92, v_num=1]
Epoch 1:  27%|##7       | 6/22 [00:02<00:06,  2.39it/s, loss=4.92, v_num=1]
Epoch 1:  27%|##7       | 6/22 [00:02<00:06,  2.33it/s, loss=4.9, v_num=1]
Epoch 1:  32%|###1      | 7/22 [00:02<00:05,  2.63it/s, loss=4.9, v_num=1]
Epoch 1:  32%|###1      | 7/22 [00:02<00:05,  2.57it/s, loss=4.88, v_num=1]
Epoch 1:  36%|###6      | 8/22 [00:02<00:04,  2.84it/s, loss=4.88, v_num=1]
Epoch 1:  36%|###6      | 8/22 [00:02<00:05,  2.78it/s, loss=4.87, v_num=1]
Epoch 1:  41%|####      | 9/22 [00:02<00:04,  3.03it/s, loss=4.87, v_num=1]
Epoch 1:  41%|####      | 9/22 [00:03<00:04,  2.97it/s, loss=4.86, v_num=1]
Epoch 1:  45%|####5     | 10/22 [00:03<00:03,  3.21it/s, loss=4.86, v_num=1]
Epoch 1:  45%|####5     | 10/22 [00:03<00:03,  3.15it/s, loss=4.85, v_num=1]
Epoch 1:  50%|#####     | 11/22 [00:03<00:03,  3.37it/s, loss=4.85, v_num=1]
Epoch 1:  50%|#####     | 11/22 [00:03<00:03,  3.30it/s, loss=4.84, v_num=1]
Epoch 1:  55%|#####4    | 12/22 [00:03<00:02,  3.50it/s, loss=4.84, v_num=1]
Epoch 1:  55%|#####4    | 12/22 [00:03<00:02,  3.44it/s, loss=4.83, v_num=1]
Epoch 1:  59%|#####9    | 13/22 [00:03<00:02,  3.63it/s, loss=4.83, v_num=1]
Epoch 1:  59%|#####9    | 13/22 [00:03<00:02,  3.57it/s, loss=4.82, v_num=1]
Epoch 1:  64%|######3   | 14/22 [00:03<00:02,  3.76it/s, loss=4.82, v_num=1]
Epoch 1:  64%|######3   | 14/22 [00:03<00:02,  3.69it/s, loss=4.82, v_num=1]
Epoch 1:  68%|######8   | 15/22 [00:03<00:01,  3.87it/s, loss=4.82, v_num=1]
Epoch 1:  68%|######8   | 15/22 [00:03<00:01,  3.81it/s, loss=4.81, v_num=1]
Epoch 1:  73%|#######2  | 16/22 [00:04<00:01,  3.98it/s, loss=4.81, v_num=1]
Epoch 1:  73%|#######2  | 16/22 [00:04<00:01,  3.92it/s, loss=4.8, v_num=1]
Epoch 1:  77%|#######7  | 17/22 [00:04<00:01,  4.07it/s, loss=4.8, v_num=1]
Epoch 1:  77%|#######7  | 17/22 [00:04<00:01,  4.01it/s, loss=4.8, v_num=1]
Epoch 1:  82%|########1 | 18/22 [00:04<00:00,  4.16it/s, loss=4.8, v_num=1]
Epoch 1:  82%|########1 | 18/22 [00:04<00:00,  4.10it/s, loss=4.79, v_num=1]
Epoch 1:  86%|########6 | 19/22 [00:04<00:00,  4.25it/s, loss=4.79, v_num=1]
Epoch 1:  86%|########6 | 19/22 [00:04<00:00,  4.19it/s, loss=4.79, v_num=1]
Epoch 1:  91%|######### | 20/22 [00:04<00:00,  4.32it/s, loss=4.79, v_num=1]
Epoch 1:  91%|######### | 20/22 [00:04<00:00,  4.27it/s, loss=4.78, v_num=1]
Epoch 1:  95%|#########5| 21/22 [00:04<00:00,  4.40it/s, loss=4.78, v_num=1]
Epoch 1:  95%|#########5| 21/22 [00:04<00:00,  4.34it/s, loss=4.78, v_num=1]
Epoch 1: 100%|##########| 22/22 [00:04<00:00,  4.47it/s, loss=4.78, v_num=1]
Epoch 1: 100%|##########| 22/22 [00:04<00:00,  4.41it/s, loss=4.78, v_num=1]
Epoch 1: 100%|##########| 22/22 [00:04<00:00,  4.41it/s, loss=4.78, v_num=1]
Epoch 1:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.78, v_num=1]
Epoch 2:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.78, v_num=1]
Epoch 2:   5%|4         | 1/22 [00:01<00:36,  1.74s/it, loss=4.78, v_num=1]
Epoch 2:   5%|4         | 1/22 [00:01<00:37,  1.80s/it, loss=4.77, v_num=1]
Epoch 2:   9%|9         | 2/22 [00:01<00:18,  1.05it/s, loss=4.77, v_num=1]
Epoch 2:   9%|9         | 2/22 [00:01<00:19,  1.02it/s, loss=4.77, v_num=1]
Epoch 2:  14%|#3        | 3/22 [00:02<00:13,  1.46it/s, loss=4.77, v_num=1]
Epoch 2:  14%|#3        | 3/22 [00:02<00:13,  1.42it/s, loss=4.77, v_num=1]
Epoch 2:  18%|#8        | 4/22 [00:02<00:09,  1.81it/s, loss=4.77, v_num=1]
Epoch 2:  18%|#8        | 4/22 [00:02<00:10,  1.77it/s, loss=4.76, v_num=1]
Epoch 2:  23%|##2       | 5/22 [00:02<00:07,  2.13it/s, loss=4.76, v_num=1]
Epoch 2:  23%|##2       | 5/22 [00:02<00:08,  2.07it/s, loss=4.76, v_num=1]
Epoch 2:  27%|##7       | 6/22 [00:02<00:06,  2.38it/s, loss=4.76, v_num=1]
Epoch 2:  27%|##7       | 6/22 [00:02<00:06,  2.32it/s, loss=4.75, v_num=1]
Epoch 2:  32%|###1      | 7/22 [00:02<00:05,  2.61it/s, loss=4.75, v_num=1]
Epoch 2:  32%|###1      | 7/22 [00:02<00:05,  2.56it/s, loss=4.75, v_num=1]
Epoch 2:  36%|###6      | 8/22 [00:02<00:04,  2.83it/s, loss=4.75, v_num=1]
Epoch 2:  36%|###6      | 8/22 [00:02<00:05,  2.77it/s, loss=4.75, v_num=1]
Epoch 2:  41%|####      | 9/22 [00:02<00:04,  3.01it/s, loss=4.75, v_num=1]
Epoch 2:  41%|####      | 9/22 [00:03<00:04,  2.96it/s, loss=4.75, v_num=1]
Epoch 2:  45%|####5     | 10/22 [00:03<00:03,  3.18it/s, loss=4.75, v_num=1]
Epoch 2:  45%|####5     | 10/22 [00:03<00:03,  3.12it/s, loss=4.74, v_num=1]
Epoch 2:  50%|#####     | 11/22 [00:03<00:03,  3.33it/s, loss=4.74, v_num=1]
Epoch 2:  50%|#####     | 11/22 [00:03<00:03,  3.27it/s, loss=4.74, v_num=1]
Epoch 2:  55%|#####4    | 12/22 [00:03<00:02,  3.45it/s, loss=4.74, v_num=1]
Epoch 2:  55%|#####4    | 12/22 [00:03<00:02,  3.39it/s, loss=4.73, v_num=1]
Epoch 2:  59%|#####9    | 13/22 [00:03<00:02,  3.58it/s, loss=4.73, v_num=1]
Epoch 2:  59%|#####9    | 13/22 [00:03<00:02,  3.52it/s, loss=4.72, v_num=1]
Epoch 2:  64%|######3   | 14/22 [00:03<00:02,  3.69it/s, loss=4.72, v_num=1]
Epoch 2:  64%|######3   | 14/22 [00:03<00:02,  3.64it/s, loss=4.72, v_num=1]
Epoch 2:  68%|######8   | 15/22 [00:03<00:01,  3.80it/s, loss=4.72, v_num=1]
Epoch 2:  68%|######8   | 15/22 [00:04<00:01,  3.75it/s, loss=4.72, v_num=1]
Epoch 2:  73%|#######2  | 16/22 [00:04<00:01,  3.91it/s, loss=4.72, v_num=1]
Epoch 2:  73%|#######2  | 16/22 [00:04<00:01,  3.85it/s, loss=4.72, v_num=1]
Epoch 2:  77%|#######7  | 17/22 [00:04<00:01,  4.00it/s, loss=4.72, v_num=1]
Epoch 2:  77%|#######7  | 17/22 [00:04<00:01,  3.95it/s, loss=4.71, v_num=1]
Epoch 2:  82%|########1 | 18/22 [00:04<00:00,  4.09it/s, loss=4.71, v_num=1]
Epoch 2:  82%|########1 | 18/22 [00:04<00:00,  4.04it/s, loss=4.71, v_num=1]
Epoch 2:  86%|########6 | 19/22 [00:04<00:00,  4.17it/s, loss=4.71, v_num=1]
Epoch 2:  86%|########6 | 19/22 [00:04<00:00,  4.12it/s, loss=4.71, v_num=1]
Epoch 2:  91%|######### | 20/22 [00:04<00:00,  4.24it/s, loss=4.71, v_num=1]
Epoch 2:  91%|######### | 20/22 [00:04<00:00,  4.20it/s, loss=4.71, v_num=1]
Epoch 2:  95%|#########5| 21/22 [00:04<00:00,  4.32it/s, loss=4.71, v_num=1]
Epoch 2:  95%|#########5| 21/22 [00:04<00:00,  4.27it/s, loss=4.7, v_num=1]
Epoch 2: 100%|##########| 22/22 [00:05<00:00,  4.39it/s, loss=4.7, v_num=1]
Epoch 2: 100%|##########| 22/22 [00:05<00:00,  4.34it/s, loss=4.7, v_num=1]
Epoch 2: 100%|##########| 22/22 [00:05<00:00,  4.34it/s, loss=4.7, v_num=1]
Epoch 2:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.7, v_num=1]
Epoch 3:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.7, v_num=1]
Epoch 3:   5%|4         | 1/22 [00:01<00:37,  1.81s/it, loss=4.7, v_num=1]
Epoch 3:   5%|4         | 1/22 [00:01<00:39,  1.87s/it, loss=4.7, v_num=1]
Epoch 3:   9%|9         | 2/22 [00:01<00:19,  1.02it/s, loss=4.7, v_num=1]
Epoch 3:   9%|9         | 2/22 [00:02<00:20,  1.01s/it, loss=4.69, v_num=1]
Epoch 3:  14%|#3        | 3/22 [00:02<00:13,  1.41it/s, loss=4.69, v_num=1]
Epoch 3:  14%|#3        | 3/22 [00:02<00:13,  1.37it/s, loss=4.69, v_num=1]
Epoch 3:  18%|#8        | 4/22 [00:02<00:10,  1.75it/s, loss=4.69, v_num=1]
Epoch 3:  18%|#8        | 4/22 [00:02<00:10,  1.71it/s, loss=4.69, v_num=1]
Epoch 3:  23%|##2       | 5/22 [00:02<00:08,  2.05it/s, loss=4.69, v_num=1]
Epoch 3:  23%|##2       | 5/22 [00:02<00:08,  2.00it/s, loss=4.69, v_num=1]
Epoch 3:  27%|##7       | 6/22 [00:02<00:06,  2.32it/s, loss=4.69, v_num=1]
Epoch 3:  27%|##7       | 6/22 [00:02<00:07,  2.26it/s, loss=4.69, v_num=1]
Epoch 3:  32%|###1      | 7/22 [00:02<00:05,  2.54it/s, loss=4.69, v_num=1]
Epoch 3:  32%|###1      | 7/22 [00:02<00:06,  2.49it/s, loss=4.69, v_num=1]
Epoch 3:  36%|###6      | 8/22 [00:02<00:05,  2.75it/s, loss=4.69, v_num=1]
Epoch 3:  36%|###6      | 8/22 [00:02<00:05,  2.70it/s, loss=4.69, v_num=1]
Epoch 3:  41%|####      | 9/22 [00:03<00:04,  2.94it/s, loss=4.69, v_num=1]
Epoch 3:  41%|####      | 9/22 [00:03<00:04,  2.88it/s, loss=4.69, v_num=1]
Epoch 3:  45%|####5     | 10/22 [00:03<00:03,  3.11it/s, loss=4.69, v_num=1]
Epoch 3:  45%|####5     | 10/22 [00:03<00:03,  3.06it/s, loss=4.69, v_num=1]
Epoch 3:  50%|#####     | 11/22 [00:03<00:03,  3.27it/s, loss=4.69, v_num=1]
Epoch 3:  50%|#####     | 11/22 [00:03<00:03,  3.21it/s, loss=4.69, v_num=1]
Epoch 3:  55%|#####4    | 12/22 [00:03<00:02,  3.40it/s, loss=4.69, v_num=1]
Epoch 3:  55%|#####4    | 12/22 [00:03<00:02,  3.35it/s, loss=4.68, v_num=1]
Epoch 3:  59%|#####9    | 13/22 [00:03<00:02,  3.53it/s, loss=4.68, v_num=1]
Epoch 3:  59%|#####9    | 13/22 [00:03<00:02,  3.48it/s, loss=4.68, v_num=1]
Epoch 3:  64%|######3   | 14/22 [00:03<00:02,  3.65it/s, loss=4.68, v_num=1]
Epoch 3:  64%|######3   | 14/22 [00:03<00:02,  3.60it/s, loss=4.68, v_num=1]
Epoch 3:  68%|######8   | 15/22 [00:03<00:01,  3.77it/s, loss=4.68, v_num=1]
Epoch 3:  68%|######8   | 15/22 [00:04<00:01,  3.71it/s, loss=4.68, v_num=1]
Epoch 3:  73%|#######2  | 16/22 [00:04<00:01,  3.88it/s, loss=4.68, v_num=1]
Epoch 3:  73%|#######2  | 16/22 [00:04<00:01,  3.82it/s, loss=4.68, v_num=1]
Epoch 3:  77%|#######7  | 17/22 [00:04<00:01,  3.97it/s, loss=4.68, v_num=1]
Epoch 3:  77%|#######7  | 17/22 [00:04<00:01,  3.92it/s, loss=4.67, v_num=1]
Epoch 3:  82%|########1 | 18/22 [00:04<00:00,  4.06it/s, loss=4.67, v_num=1]
Epoch 3:  82%|########1 | 18/22 [00:04<00:00,  4.01it/s, loss=4.67, v_num=1]
Epoch 3:  86%|########6 | 19/22 [00:04<00:00,  4.15it/s, loss=4.67, v_num=1]
Epoch 3:  86%|########6 | 19/22 [00:04<00:00,  4.09it/s, loss=4.67, v_num=1]
Epoch 3:  91%|######### | 20/22 [00:04<00:00,  4.23it/s, loss=4.67, v_num=1]
Epoch 3:  91%|######### | 20/22 [00:04<00:00,  4.17it/s, loss=4.67, v_num=1]
Epoch 3:  95%|#########5| 21/22 [00:04<00:00,  4.30it/s, loss=4.67, v_num=1]
Epoch 3:  95%|#########5| 21/22 [00:04<00:00,  4.25it/s, loss=4.67, v_num=1]
Epoch 3: 100%|##########| 22/22 [00:05<00:00,  4.38it/s, loss=4.67, v_num=1]
Epoch 3: 100%|##########| 22/22 [00:05<00:00,  4.32it/s, loss=4.67, v_num=1]
Epoch 3: 100%|##########| 22/22 [00:05<00:00,  4.32it/s, loss=4.67, v_num=1]
Epoch 3:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.67, v_num=1]
Epoch 4:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.67, v_num=1]
Epoch 4:   5%|4         | 1/22 [00:01<00:36,  1.74s/it, loss=4.67, v_num=1]
Epoch 4:   5%|4         | 1/22 [00:01<00:37,  1.80s/it, loss=4.67, v_num=1]
Epoch 4:   9%|9         | 2/22 [00:01<00:19,  1.05it/s, loss=4.67, v_num=1]
Epoch 4:   9%|9         | 2/22 [00:01<00:19,  1.01it/s, loss=4.66, v_num=1]
Epoch 4:  14%|#3        | 3/22 [00:02<00:13,  1.45it/s, loss=4.66, v_num=1]
Epoch 4:  14%|#3        | 3/22 [00:02<00:13,  1.41it/s, loss=4.66, v_num=1]
Epoch 4:  18%|#8        | 4/22 [00:02<00:10,  1.80it/s, loss=4.66, v_num=1]
Epoch 4:  18%|#8        | 4/22 [00:02<00:10,  1.75it/s, loss=4.65, v_num=1]
Epoch 4:  23%|##2       | 5/22 [00:02<00:08,  2.10it/s, loss=4.65, v_num=1]
Epoch 4:  23%|##2       | 5/22 [00:02<00:08,  2.05it/s, loss=4.65, v_num=1]
Epoch 4:  27%|##7       | 6/22 [00:02<00:06,  2.35it/s, loss=4.65, v_num=1]
Epoch 4:  27%|##7       | 6/22 [00:02<00:06,  2.30it/s, loss=4.65, v_num=1]
Epoch 4:  32%|###1      | 7/22 [00:02<00:05,  2.59it/s, loss=4.65, v_num=1]
Epoch 4:  32%|###1      | 7/22 [00:02<00:05,  2.53it/s, loss=4.65, v_num=1]
Epoch 4:  36%|###6      | 8/22 [00:02<00:04,  2.80it/s, loss=4.65, v_num=1]
Epoch 4:  36%|###6      | 8/22 [00:02<00:05,  2.74it/s, loss=4.65, v_num=1]
Epoch 4:  41%|####      | 9/22 [00:03<00:04,  2.99it/s, loss=4.65, v_num=1]
Epoch 4:  41%|####      | 9/22 [00:03<00:04,  2.93it/s, loss=4.64, v_num=1]
Epoch 4:  45%|####5     | 10/22 [00:03<00:03,  3.15it/s, loss=4.64, v_num=1]
Epoch 4:  45%|####5     | 10/22 [00:03<00:03,  3.10it/s, loss=4.64, v_num=1]
Epoch 4:  50%|#####     | 11/22 [00:03<00:03,  3.31it/s, loss=4.64, v_num=1]
Epoch 4:  50%|#####     | 11/22 [00:03<00:03,  3.25it/s, loss=4.64, v_num=1]
Epoch 4:  55%|#####4    | 12/22 [00:03<00:02,  3.46it/s, loss=4.64, v_num=1]
Epoch 4:  55%|#####4    | 12/22 [00:03<00:02,  3.39it/s, loss=4.64, v_num=1]
Epoch 4:  59%|#####9    | 13/22 [00:03<00:02,  3.59it/s, loss=4.64, v_num=1]
Epoch 4:  59%|#####9    | 13/22 [00:03<00:02,  3.53it/s, loss=4.64, v_num=1]
Epoch 4:  64%|######3   | 14/22 [00:03<00:02,  3.71it/s, loss=4.64, v_num=1]
Epoch 4:  64%|######3   | 14/22 [00:03<00:02,  3.65it/s, loss=4.64, v_num=1]
Epoch 4:  68%|######8   | 15/22 [00:03<00:01,  3.82it/s, loss=4.64, v_num=1]
Epoch 4:  68%|######8   | 15/22 [00:03<00:01,  3.76it/s, loss=4.64, v_num=1]
Epoch 4:  73%|#######2  | 16/22 [00:04<00:01,  3.93it/s, loss=4.64, v_num=1]
Epoch 4:  73%|#######2  | 16/22 [00:04<00:01,  3.87it/s, loss=4.64, v_num=1]
Epoch 4:  77%|#######7  | 17/22 [00:04<00:01,  4.02it/s, loss=4.64, v_num=1]
Epoch 4:  77%|#######7  | 17/22 [00:04<00:01,  3.97it/s, loss=4.64, v_num=1]
Epoch 4:  82%|########1 | 18/22 [00:04<00:00,  4.11it/s, loss=4.64, v_num=1]
Epoch 4:  82%|########1 | 18/22 [00:04<00:00,  4.06it/s, loss=4.64, v_num=1]
Epoch 4:  86%|########6 | 19/22 [00:04<00:00,  4.20it/s, loss=4.64, v_num=1]
Epoch 4:  86%|########6 | 19/22 [00:04<00:00,  4.14it/s, loss=4.64, v_num=1]
Epoch 4:  91%|######### | 20/22 [00:04<00:00,  4.28it/s, loss=4.64, v_num=1]
Epoch 4:  91%|######### | 20/22 [00:04<00:00,  4.22it/s, loss=4.64, v_num=1]
Epoch 4:  95%|#########5| 21/22 [00:04<00:00,  4.35it/s, loss=4.64, v_num=1]
Epoch 4:  95%|#########5| 21/22 [00:04<00:00,  4.30it/s, loss=4.64, v_num=1]
Epoch 4: 100%|##########| 22/22 [00:04<00:00,  4.42it/s, loss=4.64, v_num=1]
Epoch 4: 100%|##########| 22/22 [00:05<00:00,  4.37it/s, loss=4.64, v_num=1]
Epoch 4: 100%|##########| 22/22 [00:05<00:00,  4.37it/s, loss=4.64, v_num=1]
Epoch 4:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.64, v_num=1]
Epoch 5:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.64, v_num=1]
Epoch 5:   5%|4         | 1/22 [00:01<00:36,  1.74s/it, loss=4.64, v_num=1]
Epoch 5:   5%|4         | 1/22 [00:01<00:37,  1.80s/it, loss=4.64, v_num=1]
Epoch 5:   9%|9         | 2/22 [00:01<00:18,  1.05it/s, loss=4.64, v_num=1]
Epoch 5:   9%|9         | 2/22 [00:01<00:19,  1.02it/s, loss=4.64, v_num=1]
Epoch 5:  14%|#3        | 3/22 [00:02<00:12,  1.47it/s, loss=4.64, v_num=1]
Epoch 5:  14%|#3        | 3/22 [00:02<00:13,  1.42it/s, loss=4.64, v_num=1]
Epoch 5:  18%|#8        | 4/22 [00:02<00:09,  1.82it/s, loss=4.64, v_num=1]
Epoch 5:  18%|#8        | 4/22 [00:02<00:10,  1.77it/s, loss=4.64, v_num=1]
Epoch 5:  23%|##2       | 5/22 [00:02<00:08,  2.12it/s, loss=4.64, v_num=1]
Epoch 5:  23%|##2       | 5/22 [00:02<00:08,  2.07it/s, loss=4.64, v_num=1]
Epoch 5:  27%|##7       | 6/22 [00:02<00:06,  2.38it/s, loss=4.64, v_num=1]
Epoch 5:  27%|##7       | 6/22 [00:02<00:06,  2.33it/s, loss=4.64, v_num=1]
Epoch 5:  32%|###1      | 7/22 [00:02<00:05,  2.62it/s, loss=4.64, v_num=1]
Epoch 5:  32%|###1      | 7/22 [00:02<00:05,  2.56it/s, loss=4.64, v_num=1]
Epoch 5:  36%|###6      | 8/22 [00:02<00:04,  2.84it/s, loss=4.64, v_num=1]
Epoch 5:  36%|###6      | 8/22 [00:02<00:05,  2.78it/s, loss=4.64, v_num=1]
Epoch 5:  41%|####      | 9/22 [00:02<00:04,  3.03it/s, loss=4.64, v_num=1]
Epoch 5:  41%|####      | 9/22 [00:03<00:04,  2.97it/s, loss=4.63, v_num=1]
Epoch 5:  45%|####5     | 10/22 [00:03<00:03,  3.20it/s, loss=4.63, v_num=1]
Epoch 5:  45%|####5     | 10/22 [00:03<00:03,  3.14it/s, loss=4.63, v_num=1]
Epoch 5:  50%|#####     | 11/22 [00:03<00:03,  3.36it/s, loss=4.63, v_num=1]
Epoch 5:  50%|#####     | 11/22 [00:03<00:03,  3.30it/s, loss=4.63, v_num=1]
Epoch 5:  55%|#####4    | 12/22 [00:03<00:02,  3.45it/s, loss=4.63, v_num=1]
Epoch 5:  55%|#####4    | 12/22 [00:03<00:02,  3.38it/s, loss=4.63, v_num=1]
Epoch 5:  59%|#####9    | 13/22 [00:03<00:02,  3.57it/s, loss=4.63, v_num=1]
Epoch 5:  59%|#####9    | 13/22 [00:03<00:02,  3.52it/s, loss=4.63, v_num=1]
Epoch 5:  64%|######3   | 14/22 [00:03<00:02,  3.70it/s, loss=4.63, v_num=1]
Epoch 5:  64%|######3   | 14/22 [00:03<00:02,  3.64it/s, loss=4.63, v_num=1]
Epoch 5:  68%|######8   | 15/22 [00:03<00:01,  3.81it/s, loss=4.63, v_num=1]
Epoch 5:  68%|######8   | 15/22 [00:03<00:01,  3.75it/s, loss=4.63, v_num=1]
Epoch 5:  73%|#######2  | 16/22 [00:04<00:01,  3.92it/s, loss=4.63, v_num=1]
Epoch 5:  73%|#######2  | 16/22 [00:04<00:01,  3.86it/s, loss=4.63, v_num=1]
Epoch 5:  77%|#######7  | 17/22 [00:04<00:01,  4.01it/s, loss=4.63, v_num=1]
Epoch 5:  77%|#######7  | 17/22 [00:04<00:01,  3.96it/s, loss=4.63, v_num=1]
Epoch 5:  82%|########1 | 18/22 [00:04<00:00,  4.10it/s, loss=4.63, v_num=1]
Epoch 5:  82%|########1 | 18/22 [00:04<00:00,  4.05it/s, loss=4.63, v_num=1]
Epoch 5:  86%|########6 | 19/22 [00:04<00:00,  4.19it/s, loss=4.63, v_num=1]
Epoch 5:  86%|########6 | 19/22 [00:04<00:00,  4.13it/s, loss=4.63, v_num=1]
Epoch 5:  91%|######### | 20/22 [00:04<00:00,  4.27it/s, loss=4.63, v_num=1]
Epoch 5:  91%|######### | 20/22 [00:04<00:00,  4.21it/s, loss=4.63, v_num=1]
Epoch 5:  95%|#########5| 21/22 [00:04<00:00,  4.34it/s, loss=4.63, v_num=1]
Epoch 5:  95%|#########5| 21/22 [00:04<00:00,  4.29it/s, loss=4.63, v_num=1]
Epoch 5: 100%|##########| 22/22 [00:04<00:00,  4.41it/s, loss=4.63, v_num=1]
Epoch 5: 100%|##########| 22/22 [00:05<00:00,  4.36it/s, loss=4.63, v_num=1]
Epoch 5: 100%|##########| 22/22 [00:05<00:00,  4.36it/s, loss=4.63, v_num=1]
Epoch 5:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.63, v_num=1]
Epoch 6:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.63, v_num=1]
Epoch 6:   5%|4         | 1/22 [00:01<00:37,  1.79s/it, loss=4.63, v_num=1]
Epoch 6:   5%|4         | 1/22 [00:01<00:38,  1.85s/it, loss=4.63, v_num=1]
Epoch 6:   9%|9         | 2/22 [00:01<00:19,  1.03it/s, loss=4.63, v_num=1]
Epoch 6:   9%|9         | 2/22 [00:02<00:20,  1.00s/it, loss=4.63, v_num=1]
Epoch 6:  14%|#3        | 3/22 [00:02<00:13,  1.42it/s, loss=4.63, v_num=1]
Epoch 6:  14%|#3        | 3/22 [00:02<00:13,  1.38it/s, loss=4.63, v_num=1]
Epoch 6:  18%|#8        | 4/22 [00:02<00:10,  1.77it/s, loss=4.63, v_num=1]
Epoch 6:  18%|#8        | 4/22 [00:02<00:10,  1.72it/s, loss=4.63, v_num=1]
Epoch 6:  23%|##2       | 5/22 [00:02<00:08,  2.07it/s, loss=4.63, v_num=1]
Epoch 6:  23%|##2       | 5/22 [00:02<00:08,  2.02it/s, loss=4.62, v_num=1]
Epoch 6:  27%|##7       | 6/22 [00:02<00:06,  2.32it/s, loss=4.62, v_num=1]
Epoch 6:  27%|##7       | 6/22 [00:02<00:07,  2.27it/s, loss=4.62, v_num=1]
Epoch 6:  32%|###1      | 7/22 [00:02<00:05,  2.56it/s, loss=4.62, v_num=1]
Epoch 6:  32%|###1      | 7/22 [00:02<00:05,  2.50it/s, loss=4.62, v_num=1]
Epoch 6:  36%|###6      | 8/22 [00:02<00:05,  2.77it/s, loss=4.62, v_num=1]
Epoch 6:  36%|###6      | 8/22 [00:02<00:05,  2.71it/s, loss=4.62, v_num=1]
Epoch 6:  41%|####      | 9/22 [00:03<00:04,  2.95it/s, loss=4.62, v_num=1]
Epoch 6:  41%|####      | 9/22 [00:03<00:04,  2.90it/s, loss=4.62, v_num=1]
Epoch 6:  45%|####5     | 10/22 [00:03<00:03,  3.12it/s, loss=4.62, v_num=1]
Epoch 6:  45%|####5     | 10/22 [00:03<00:03,  3.07it/s, loss=4.62, v_num=1]
Epoch 6:  50%|#####     | 11/22 [00:03<00:03,  3.28it/s, loss=4.62, v_num=1]
Epoch 6:  50%|#####     | 11/22 [00:03<00:03,  3.22it/s, loss=4.62, v_num=1]
Epoch 6:  55%|#####4    | 12/22 [00:03<00:02,  3.42it/s, loss=4.62, v_num=1]
Epoch 6:  55%|#####4    | 12/22 [00:03<00:02,  3.36it/s, loss=4.62, v_num=1]
Epoch 6:  59%|#####9    | 13/22 [00:03<00:02,  3.55it/s, loss=4.62, v_num=1]
Epoch 6:  59%|#####9    | 13/22 [00:03<00:02,  3.49it/s, loss=4.62, v_num=1]
Epoch 6:  64%|######3   | 14/22 [00:03<00:02,  3.67it/s, loss=4.62, v_num=1]
Epoch 6:  64%|######3   | 14/22 [00:03<00:02,  3.61it/s, loss=4.62, v_num=1]
Epoch 6:  68%|######8   | 15/22 [00:03<00:01,  3.78it/s, loss=4.62, v_num=1]
Epoch 6:  68%|######8   | 15/22 [00:04<00:01,  3.73it/s, loss=4.62, v_num=1]
Epoch 6:  73%|#######2  | 16/22 [00:04<00:01,  3.89it/s, loss=4.62, v_num=1]
Epoch 6:  73%|#######2  | 16/22 [00:04<00:01,  3.83it/s, loss=4.61, v_num=1]
Epoch 6:  77%|#######7  | 17/22 [00:04<00:01,  3.98it/s, loss=4.61, v_num=1]
Epoch 6:  77%|#######7  | 17/22 [00:04<00:01,  3.92it/s, loss=4.61, v_num=1]
Epoch 6:  82%|########1 | 18/22 [00:04<00:00,  4.07it/s, loss=4.61, v_num=1]
Epoch 6:  82%|########1 | 18/22 [00:04<00:00,  4.01it/s, loss=4.61, v_num=1]
Epoch 6:  86%|########6 | 19/22 [00:04<00:00,  4.15it/s, loss=4.61, v_num=1]
Epoch 6:  86%|########6 | 19/22 [00:04<00:00,  4.09it/s, loss=4.61, v_num=1]
Epoch 6:  91%|######### | 20/22 [00:04<00:00,  4.23it/s, loss=4.61, v_num=1]
Epoch 6:  91%|######### | 20/22 [00:04<00:00,  4.17it/s, loss=4.61, v_num=1]
Epoch 6:  95%|#########5| 21/22 [00:04<00:00,  4.29it/s, loss=4.61, v_num=1]
Epoch 6:  95%|#########5| 21/22 [00:04<00:00,  4.24it/s, loss=4.61, v_num=1]
Epoch 6: 100%|##########| 22/22 [00:05<00:00,  4.36it/s, loss=4.61, v_num=1]
Epoch 6: 100%|##########| 22/22 [00:05<00:00,  4.31it/s, loss=4.61, v_num=1]
Epoch 6: 100%|##########| 22/22 [00:05<00:00,  4.31it/s, loss=4.61, v_num=1]
Epoch 6:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.61, v_num=1]
Epoch 7:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.61, v_num=1]
Epoch 7:   5%|4         | 1/22 [00:01<00:38,  1.82s/it, loss=4.61, v_num=1]
Epoch 7:   5%|4         | 1/22 [00:01<00:39,  1.88s/it, loss=4.61, v_num=1]
Epoch 7:   9%|9         | 2/22 [00:01<00:19,  1.01it/s, loss=4.61, v_num=1]
Epoch 7:   9%|9         | 2/22 [00:02<00:20,  1.02s/it, loss=4.61, v_num=1]
Epoch 7:  14%|#3        | 3/22 [00:02<00:13,  1.40it/s, loss=4.61, v_num=1]
Epoch 7:  14%|#3        | 3/22 [00:02<00:13,  1.37it/s, loss=4.61, v_num=1]
Epoch 7:  18%|#8        | 4/22 [00:02<00:10,  1.74it/s, loss=4.61, v_num=1]
Epoch 7:  18%|#8        | 4/22 [00:02<00:10,  1.70it/s, loss=4.61, v_num=1]
Epoch 7:  23%|##2       | 5/22 [00:02<00:08,  2.04it/s, loss=4.61, v_num=1]
Epoch 7:  23%|##2       | 5/22 [00:02<00:08,  1.99it/s, loss=4.6, v_num=1]
Epoch 7:  27%|##7       | 6/22 [00:02<00:06,  2.29it/s, loss=4.6, v_num=1]
Epoch 7:  27%|##7       | 6/22 [00:02<00:07,  2.24it/s, loss=4.61, v_num=1]
Epoch 7:  32%|###1      | 7/22 [00:02<00:05,  2.53it/s, loss=4.61, v_num=1]
Epoch 7:  32%|###1      | 7/22 [00:02<00:06,  2.47it/s, loss=4.61, v_num=1]
Epoch 7:  36%|###6      | 8/22 [00:02<00:05,  2.74it/s, loss=4.61, v_num=1]
Epoch 7:  36%|###6      | 8/22 [00:02<00:05,  2.68it/s, loss=4.6, v_num=1]
Epoch 7:  41%|####      | 9/22 [00:03<00:04,  2.92it/s, loss=4.6, v_num=1]
Epoch 7:  41%|####      | 9/22 [00:03<00:04,  2.87it/s, loss=4.6, v_num=1]
Epoch 7:  45%|####5     | 10/22 [00:03<00:03,  3.10it/s, loss=4.6, v_num=1]
Epoch 7:  45%|####5     | 10/22 [00:03<00:03,  3.04it/s, loss=4.6, v_num=1]
Epoch 7:  50%|#####     | 11/22 [00:03<00:03,  3.26it/s, loss=4.6, v_num=1]
Epoch 7:  50%|#####     | 11/22 [00:03<00:03,  3.19it/s, loss=4.6, v_num=1]
Epoch 7:  55%|#####4    | 12/22 [00:03<00:02,  3.39it/s, loss=4.6, v_num=1]
Epoch 7:  55%|#####4    | 12/22 [00:03<00:03,  3.33it/s, loss=4.6, v_num=1]
Epoch 7:  59%|#####9    | 13/22 [00:03<00:02,  3.51it/s, loss=4.6, v_num=1]
Epoch 7:  59%|#####9    | 13/22 [00:03<00:02,  3.46it/s, loss=4.6, v_num=1]
Epoch 7:  64%|######3   | 14/22 [00:03<00:02,  3.64it/s, loss=4.6, v_num=1]
Epoch 7:  64%|######3   | 14/22 [00:03<00:02,  3.58it/s, loss=4.6, v_num=1]
Epoch 7:  68%|######8   | 15/22 [00:03<00:01,  3.75it/s, loss=4.6, v_num=1]
Epoch 7:  68%|######8   | 15/22 [00:04<00:01,  3.70it/s, loss=4.6, v_num=1]
Epoch 7:  73%|#######2  | 16/22 [00:04<00:01,  3.86it/s, loss=4.6, v_num=1]
Epoch 7:  73%|#######2  | 16/22 [00:04<00:01,  3.80it/s, loss=4.6, v_num=1]
Epoch 7:  77%|#######7  | 17/22 [00:04<00:01,  3.95it/s, loss=4.6, v_num=1]
Epoch 7:  77%|#######7  | 17/22 [00:04<00:01,  3.90it/s, loss=4.6, v_num=1]
Epoch 7:  82%|########1 | 18/22 [00:04<00:00,  4.04it/s, loss=4.6, v_num=1]
Epoch 7:  82%|########1 | 18/22 [00:04<00:01,  3.99it/s, loss=4.6, v_num=1]
Epoch 7:  86%|########6 | 19/22 [00:04<00:00,  4.13it/s, loss=4.6, v_num=1]
Epoch 7:  86%|########6 | 19/22 [00:04<00:00,  4.08it/s, loss=4.6, v_num=1]
Epoch 7:  91%|######### | 20/22 [00:04<00:00,  4.20it/s, loss=4.6, v_num=1]
Epoch 7:  91%|######### | 20/22 [00:04<00:00,  4.15it/s, loss=4.6, v_num=1]
Epoch 7:  95%|#########5| 21/22 [00:04<00:00,  4.27it/s, loss=4.6, v_num=1]
Epoch 7:  95%|#########5| 21/22 [00:04<00:00,  4.22it/s, loss=4.6, v_num=1]
Epoch 7: 100%|##########| 22/22 [00:05<00:00,  4.34it/s, loss=4.6, v_num=1]
Epoch 7: 100%|##########| 22/22 [00:05<00:00,  4.29it/s, loss=4.6, v_num=1]
Epoch 7: 100%|##########| 22/22 [00:05<00:00,  4.29it/s, loss=4.6, v_num=1]
Epoch 7:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.6, v_num=1]
Epoch 8:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.6, v_num=1]
Epoch 8:   5%|4         | 1/22 [00:01<00:37,  1.78s/it, loss=4.6, v_num=1]
Epoch 8:   5%|4         | 1/22 [00:01<00:38,  1.84s/it, loss=4.6, v_num=1]
Epoch 8:   9%|9         | 2/22 [00:01<00:19,  1.03it/s, loss=4.6, v_num=1]
Epoch 8:   9%|9         | 2/22 [00:02<00:20,  1.00s/it, loss=4.6, v_num=1]
Epoch 8:  14%|#3        | 3/22 [00:02<00:13,  1.43it/s, loss=4.6, v_num=1]
Epoch 8:  14%|#3        | 3/22 [00:02<00:13,  1.39it/s, loss=4.6, v_num=1]
Epoch 8:  18%|#8        | 4/22 [00:02<00:10,  1.77it/s, loss=4.6, v_num=1]
Epoch 8:  18%|#8        | 4/22 [00:02<00:10,  1.72it/s, loss=4.59, v_num=1]
Epoch 8:  23%|##2       | 5/22 [00:02<00:08,  2.07it/s, loss=4.59, v_num=1]
Epoch 8:  23%|##2       | 5/22 [00:02<00:08,  2.02it/s, loss=4.59, v_num=1]
Epoch 8:  27%|##7       | 6/22 [00:02<00:06,  2.31it/s, loss=4.59, v_num=1]
Epoch 8:  27%|##7       | 6/22 [00:02<00:07,  2.26it/s, loss=4.59, v_num=1]
Epoch 8:  32%|###1      | 7/22 [00:02<00:05,  2.54it/s, loss=4.59, v_num=1]
Epoch 8:  32%|###1      | 7/22 [00:02<00:06,  2.48it/s, loss=4.59, v_num=1]
Epoch 8:  36%|###6      | 8/22 [00:02<00:05,  2.75it/s, loss=4.59, v_num=1]
Epoch 8:  36%|###6      | 8/22 [00:02<00:05,  2.69it/s, loss=4.59, v_num=1]
Epoch 8:  41%|####      | 9/22 [00:03<00:04,  2.93it/s, loss=4.59, v_num=1]
Epoch 8:  41%|####      | 9/22 [00:03<00:04,  2.88it/s, loss=4.59, v_num=1]
Epoch 8:  45%|####5     | 10/22 [00:03<00:03,  3.10it/s, loss=4.59, v_num=1]
Epoch 8:  45%|####5     | 10/22 [00:03<00:03,  3.05it/s, loss=4.59, v_num=1]
Epoch 8:  50%|#####     | 11/22 [00:03<00:03,  3.26it/s, loss=4.59, v_num=1]
Epoch 8:  50%|#####     | 11/22 [00:03<00:03,  3.20it/s, loss=4.59, v_num=1]
Epoch 8:  55%|#####4    | 12/22 [00:03<00:02,  3.38it/s, loss=4.59, v_num=1]
Epoch 8:  55%|#####4    | 12/22 [00:03<00:03,  3.33it/s, loss=4.59, v_num=1]
Epoch 8:  59%|#####9    | 13/22 [00:03<00:02,  3.51it/s, loss=4.59, v_num=1]
Epoch 8:  59%|#####9    | 13/22 [00:03<00:02,  3.46it/s, loss=4.58, v_num=1]
Epoch 8:  64%|######3   | 14/22 [00:03<00:02,  3.64it/s, loss=4.58, v_num=1]
Epoch 8:  64%|######3   | 14/22 [00:03<00:02,  3.58it/s, loss=4.59, v_num=1]
Epoch 8:  68%|######8   | 15/22 [00:04<00:01,  3.75it/s, loss=4.59, v_num=1]
Epoch 8:  68%|######8   | 15/22 [00:04<00:01,  3.69it/s, loss=4.59, v_num=1]
Epoch 8:  73%|#######2  | 16/22 [00:04<00:01,  3.85it/s, loss=4.59, v_num=1]
Epoch 8:  73%|#######2  | 16/22 [00:04<00:01,  3.79it/s, loss=4.59, v_num=1]
Epoch 8:  77%|#######7  | 17/22 [00:04<00:01,  3.94it/s, loss=4.59, v_num=1]
Epoch 8:  77%|#######7  | 17/22 [00:04<00:01,  3.89it/s, loss=4.58, v_num=1]
Epoch 8:  82%|########1 | 18/22 [00:04<00:00,  4.03it/s, loss=4.58, v_num=1]
Epoch 8:  82%|########1 | 18/22 [00:04<00:01,  3.98it/s, loss=4.58, v_num=1]
Epoch 8:  86%|########6 | 19/22 [00:04<00:00,  4.12it/s, loss=4.58, v_num=1]
Epoch 8:  86%|########6 | 19/22 [00:04<00:00,  4.06it/s, loss=4.58, v_num=1]
Epoch 8:  91%|######### | 20/22 [00:04<00:00,  4.20it/s, loss=4.58, v_num=1]
Epoch 8:  91%|######### | 20/22 [00:04<00:00,  4.14it/s, loss=4.58, v_num=1]
Epoch 8:  95%|#########5| 21/22 [00:04<00:00,  4.27it/s, loss=4.58, v_num=1]
Epoch 8:  95%|#########5| 21/22 [00:04<00:00,  4.22it/s, loss=4.58, v_num=1]
Epoch 8: 100%|##########| 22/22 [00:05<00:00,  4.34it/s, loss=4.58, v_num=1]
Epoch 8: 100%|##########| 22/22 [00:05<00:00,  4.29it/s, loss=4.58, v_num=1]
Epoch 8: 100%|##########| 22/22 [00:05<00:00,  4.29it/s, loss=4.58, v_num=1]
Epoch 8:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.58, v_num=1]
Epoch 9:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.58, v_num=1]
Epoch 9:   5%|4         | 1/22 [00:01<00:38,  1.82s/it, loss=4.58, v_num=1]
Epoch 9:   5%|4         | 1/22 [00:01<00:39,  1.88s/it, loss=4.58, v_num=1]
Epoch 9:   9%|9         | 2/22 [00:01<00:19,  1.01it/s, loss=4.58, v_num=1]
Epoch 9:   9%|9         | 2/22 [00:02<00:20,  1.02s/it, loss=4.58, v_num=1]
Epoch 9:  14%|#3        | 3/22 [00:02<00:13,  1.40it/s, loss=4.58, v_num=1]
Epoch 9:  14%|#3        | 3/22 [00:02<00:13,  1.37it/s, loss=4.58, v_num=1]
Epoch 9:  18%|#8        | 4/22 [00:02<00:10,  1.74it/s, loss=4.58, v_num=1]
Epoch 9:  18%|#8        | 4/22 [00:02<00:10,  1.70it/s, loss=4.58, v_num=1]
Epoch 9:  23%|##2       | 5/22 [00:02<00:08,  2.04it/s, loss=4.58, v_num=1]
Epoch 9:  23%|##2       | 5/22 [00:02<00:08,  1.99it/s, loss=4.58, v_num=1]
Epoch 9:  27%|##7       | 6/22 [00:02<00:06,  2.30it/s, loss=4.58, v_num=1]
Epoch 9:  27%|##7       | 6/22 [00:02<00:07,  2.24it/s, loss=4.58, v_num=1]
Epoch 9:  32%|###1      | 7/22 [00:02<00:05,  2.52it/s, loss=4.58, v_num=1]
Epoch 9:  32%|###1      | 7/22 [00:02<00:06,  2.47it/s, loss=4.58, v_num=1]
Epoch 9:  36%|###6      | 8/22 [00:02<00:05,  2.73it/s, loss=4.58, v_num=1]
Epoch 9:  36%|###6      | 8/22 [00:02<00:05,  2.68it/s, loss=4.58, v_num=1]
Epoch 9:  41%|####      | 9/22 [00:03<00:04,  2.92it/s, loss=4.58, v_num=1]
Epoch 9:  41%|####      | 9/22 [00:03<00:04,  2.86it/s, loss=4.58, v_num=1]
Epoch 9:  45%|####5     | 10/22 [00:03<00:03,  3.09it/s, loss=4.58, v_num=1]
Epoch 9:  45%|####5     | 10/22 [00:03<00:03,  3.03it/s, loss=4.58, v_num=1]
Epoch 9:  50%|#####     | 11/22 [00:03<00:03,  3.24it/s, loss=4.58, v_num=1]
Epoch 9:  50%|#####     | 11/22 [00:03<00:03,  3.18it/s, loss=4.58, v_num=1]
Epoch 9:  55%|#####4    | 12/22 [00:03<00:02,  3.35it/s, loss=4.58, v_num=1]
Epoch 9:  55%|#####4    | 12/22 [00:03<00:03,  3.29it/s, loss=4.57, v_num=1]
Epoch 9:  59%|#####9    | 13/22 [00:03<00:02,  3.48it/s, loss=4.57, v_num=1]
Epoch 9:  59%|#####9    | 13/22 [00:03<00:02,  3.42it/s, loss=4.57, v_num=1]
Epoch 9:  64%|######3   | 14/22 [00:03<00:02,  3.60it/s, loss=4.57, v_num=1]
Epoch 9:  64%|######3   | 14/22 [00:03<00:02,  3.54it/s, loss=4.57, v_num=1]
Epoch 9:  68%|######8   | 15/22 [00:04<00:01,  3.71it/s, loss=4.57, v_num=1]
Epoch 9:  68%|######8   | 15/22 [00:04<00:01,  3.65it/s, loss=4.58, v_num=1]
Epoch 9:  73%|#######2  | 16/22 [00:04<00:01,  3.81it/s, loss=4.58, v_num=1]
Epoch 9:  73%|#######2  | 16/22 [00:04<00:01,  3.76it/s, loss=4.58, v_num=1]
Epoch 9:  77%|#######7  | 17/22 [00:04<00:01,  3.91it/s, loss=4.58, v_num=1]
Epoch 9:  77%|#######7  | 17/22 [00:04<00:01,  3.86it/s, loss=4.58, v_num=1]
Epoch 9:  82%|########1 | 18/22 [00:04<00:00,  4.00it/s, loss=4.58, v_num=1]
Epoch 9:  82%|########1 | 18/22 [00:04<00:01,  3.95it/s, loss=4.58, v_num=1]
Epoch 9:  86%|########6 | 19/22 [00:04<00:00,  4.09it/s, loss=4.58, v_num=1]
Epoch 9:  86%|########6 | 19/22 [00:04<00:00,  4.03it/s, loss=4.58, v_num=1]
Epoch 9:  91%|######### | 20/22 [00:04<00:00,  4.17it/s, loss=4.58, v_num=1]
Epoch 9:  91%|######### | 20/22 [00:04<00:00,  4.12it/s, loss=4.58, v_num=1]
Epoch 9:  95%|#########5| 21/22 [00:04<00:00,  4.24it/s, loss=4.58, v_num=1]
Epoch 9:  95%|#########5| 21/22 [00:05<00:00,  4.19it/s, loss=4.58, v_num=1]
Epoch 9: 100%|##########| 22/22 [00:05<00:00,  4.32it/s, loss=4.58, v_num=1]
Epoch 9: 100%|##########| 22/22 [00:05<00:00,  4.27it/s, loss=4.58, v_num=1]
Epoch 9: 100%|##########| 22/22 [00:05<00:00,  4.27it/s, loss=4.58, v_num=1]
Epoch 9:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.58, v_num=1]
Epoch 10:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.58, v_num=1]
Epoch 10:   5%|4         | 1/22 [00:01<00:37,  1.79s/it, loss=4.58, v_num=1]
Epoch 10:   5%|4         | 1/22 [00:01<00:38,  1.84s/it, loss=4.58, v_num=1]
Epoch 10:   9%|9         | 2/22 [00:01<00:19,  1.03it/s, loss=4.58, v_num=1]
Epoch 10:   9%|9         | 2/22 [00:02<00:20,  1.00s/it, loss=4.58, v_num=1]
Epoch 10:  14%|#3        | 3/22 [00:02<00:13,  1.43it/s, loss=4.58, v_num=1]
Epoch 10:  14%|#3        | 3/22 [00:02<00:13,  1.39it/s, loss=4.58, v_num=1]
Epoch 10:  18%|#8        | 4/22 [00:02<00:10,  1.77it/s, loss=4.58, v_num=1]
Epoch 10:  18%|#8        | 4/22 [00:02<00:10,  1.73it/s, loss=4.58, v_num=1]
Epoch 10:  23%|##2       | 5/22 [00:02<00:08,  2.08it/s, loss=4.58, v_num=1]
Epoch 10:  23%|##2       | 5/22 [00:02<00:08,  2.03it/s, loss=4.57, v_num=1]
Epoch 10:  27%|##7       | 6/22 [00:02<00:06,  2.33it/s, loss=4.57, v_num=1]
Epoch 10:  27%|##7       | 6/22 [00:02<00:07,  2.28it/s, loss=4.57, v_num=1]
Epoch 10:  32%|###1      | 7/22 [00:02<00:05,  2.57it/s, loss=4.57, v_num=1]
Epoch 10:  32%|###1      | 7/22 [00:02<00:05,  2.51it/s, loss=4.57, v_num=1]
Epoch 10:  36%|###6      | 8/22 [00:02<00:05,  2.78it/s, loss=4.57, v_num=1]
Epoch 10:  36%|###6      | 8/22 [00:02<00:05,  2.72it/s, loss=4.57, v_num=1]
Epoch 10:  41%|####      | 9/22 [00:03<00:04,  2.96it/s, loss=4.57, v_num=1]
Epoch 10:  41%|####      | 9/22 [00:03<00:04,  2.90it/s, loss=4.57, v_num=1]
Epoch 10:  45%|####5     | 10/22 [00:03<00:03,  3.13it/s, loss=4.57, v_num=1]
Epoch 10:  45%|####5     | 10/22 [00:03<00:03,  3.07it/s, loss=4.57, v_num=1]
Epoch 10:  50%|#####     | 11/22 [00:03<00:03,  3.29it/s, loss=4.57, v_num=1]
Epoch 10:  50%|#####     | 11/22 [00:03<00:03,  3.23it/s, loss=4.57, v_num=1]
Epoch 10:  55%|#####4    | 12/22 [00:03<00:02,  3.41it/s, loss=4.57, v_num=1]
Epoch 10:  55%|#####4    | 12/22 [00:03<00:02,  3.36it/s, loss=4.57, v_num=1]
Epoch 10:  59%|#####9    | 13/22 [00:03<00:02,  3.55it/s, loss=4.57, v_num=1]
Epoch 10:  59%|#####9    | 13/22 [00:03<00:02,  3.49it/s, loss=4.57, v_num=1]
Epoch 10:  64%|######3   | 14/22 [00:03<00:02,  3.67it/s, loss=4.57, v_num=1]
Epoch 10:  64%|######3   | 14/22 [00:03<00:02,  3.61it/s, loss=4.57, v_num=1]
Epoch 10:  68%|######8   | 15/22 [00:03<00:01,  3.78it/s, loss=4.57, v_num=1]
Epoch 10:  68%|######8   | 15/22 [00:04<00:01,  3.73it/s, loss=4.56, v_num=1]
Epoch 10:  73%|#######2  | 16/22 [00:04<00:01,  3.89it/s, loss=4.56, v_num=1]
Epoch 10:  73%|#######2  | 16/22 [00:04<00:01,  3.83it/s, loss=4.57, v_num=1]
Epoch 10:  77%|#######7  | 17/22 [00:04<00:01,  3.98it/s, loss=4.57, v_num=1]
Epoch 10:  77%|#######7  | 17/22 [00:04<00:01,  3.92it/s, loss=4.56, v_num=1]
Epoch 10:  82%|########1 | 18/22 [00:04<00:00,  4.08it/s, loss=4.56, v_num=1]
Epoch 10:  82%|########1 | 18/22 [00:04<00:00,  4.02it/s, loss=4.56, v_num=1]
Epoch 10:  86%|########6 | 19/22 [00:04<00:00,  4.16it/s, loss=4.56, v_num=1]
Epoch 10:  86%|########6 | 19/22 [00:04<00:00,  4.10it/s, loss=4.56, v_num=1]
Epoch 10:  91%|######### | 20/22 [00:04<00:00,  4.24it/s, loss=4.56, v_num=1]
Epoch 10:  91%|######### | 20/22 [00:04<00:00,  4.18it/s, loss=4.57, v_num=1]
Epoch 10:  95%|#########5| 21/22 [00:04<00:00,  4.31it/s, loss=4.57, v_num=1]
Epoch 10:  95%|#########5| 21/22 [00:04<00:00,  4.26it/s, loss=4.57, v_num=1]
Epoch 10: 100%|##########| 22/22 [00:05<00:00,  4.39it/s, loss=4.57, v_num=1]
Epoch 10: 100%|##########| 22/22 [00:05<00:00,  4.33it/s, loss=4.57, v_num=1]
Epoch 10: 100%|##########| 22/22 [00:05<00:00,  4.33it/s, loss=4.57, v_num=1]
Epoch 10:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.57, v_num=1]
Epoch 11:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.57, v_num=1]
Epoch 11:   5%|4         | 1/22 [00:01<00:37,  1.80s/it, loss=4.57, v_num=1]
Epoch 11:   5%|4         | 1/22 [00:01<00:39,  1.86s/it, loss=4.57, v_num=1]
Epoch 11:   9%|9         | 2/22 [00:01<00:19,  1.02it/s, loss=4.57, v_num=1]
Epoch 11:   9%|9         | 2/22 [00:02<00:20,  1.01s/it, loss=4.57, v_num=1]
Epoch 11:  14%|#3        | 3/22 [00:02<00:13,  1.41it/s, loss=4.57, v_num=1]
Epoch 11:  14%|#3        | 3/22 [00:02<00:13,  1.38it/s, loss=4.57, v_num=1]
Epoch 11:  18%|#8        | 4/22 [00:02<00:10,  1.76it/s, loss=4.57, v_num=1]
Epoch 11:  18%|#8        | 4/22 [00:02<00:10,  1.72it/s, loss=4.57, v_num=1]
Epoch 11:  23%|##2       | 5/22 [00:02<00:08,  2.06it/s, loss=4.57, v_num=1]
Epoch 11:  23%|##2       | 5/22 [00:02<00:08,  2.01it/s, loss=4.57, v_num=1]
Epoch 11:  27%|##7       | 6/22 [00:02<00:06,  2.33it/s, loss=4.57, v_num=1]
Epoch 11:  27%|##7       | 6/22 [00:02<00:07,  2.27it/s, loss=4.57, v_num=1]
Epoch 11:  32%|###1      | 7/22 [00:02<00:05,  2.55it/s, loss=4.57, v_num=1]
Epoch 11:  32%|###1      | 7/22 [00:02<00:06,  2.49it/s, loss=4.57, v_num=1]
Epoch 11:  36%|###6      | 8/22 [00:02<00:05,  2.76it/s, loss=4.57, v_num=1]
Epoch 11:  36%|###6      | 8/22 [00:02<00:05,  2.70it/s, loss=4.57, v_num=1]
Epoch 11:  41%|####      | 9/22 [00:03<00:04,  2.94it/s, loss=4.57, v_num=1]
Epoch 11:  41%|####      | 9/22 [00:03<00:04,  2.89it/s, loss=4.57, v_num=1]
Epoch 11:  45%|####5     | 10/22 [00:03<00:03,  3.12it/s, loss=4.57, v_num=1]
Epoch 11:  45%|####5     | 10/22 [00:03<00:03,  3.06it/s, loss=4.57, v_num=1]
Epoch 11:  50%|#####     | 11/22 [00:03<00:03,  3.28it/s, loss=4.57, v_num=1]
Epoch 11:  50%|#####     | 11/22 [00:03<00:03,  3.22it/s, loss=4.57, v_num=1]
Epoch 11:  55%|#####4    | 12/22 [00:03<00:02,  3.41it/s, loss=4.57, v_num=1]
Epoch 11:  55%|#####4    | 12/22 [00:03<00:02,  3.36it/s, loss=4.57, v_num=1]
Epoch 11:  59%|#####9    | 13/22 [00:03<00:02,  3.52it/s, loss=4.57, v_num=1]
Epoch 11:  59%|#####9    | 13/22 [00:03<00:02,  3.46it/s, loss=4.57, v_num=1]
Epoch 11:  64%|######3   | 14/22 [00:03<00:02,  3.65it/s, loss=4.57, v_num=1]
Epoch 11:  64%|######3   | 14/22 [00:03<00:02,  3.59it/s, loss=4.57, v_num=1]
Epoch 11:  68%|######8   | 15/22 [00:03<00:01,  3.76it/s, loss=4.57, v_num=1]
Epoch 11:  68%|######8   | 15/22 [00:04<00:01,  3.70it/s, loss=4.57, v_num=1]
Epoch 11:  73%|#######2  | 16/22 [00:04<00:01,  3.86it/s, loss=4.57, v_num=1]
Epoch 11:  73%|#######2  | 16/22 [00:04<00:01,  3.81it/s, loss=4.57, v_num=1]
Epoch 11:  77%|#######7  | 17/22 [00:04<00:01,  3.96it/s, loss=4.57, v_num=1]
Epoch 11:  77%|#######7  | 17/22 [00:04<00:01,  3.90it/s, loss=4.57, v_num=1]
Epoch 11:  82%|########1 | 18/22 [00:04<00:00,  4.05it/s, loss=4.57, v_num=1]
Epoch 11:  82%|########1 | 18/22 [00:04<00:01,  3.99it/s, loss=4.57, v_num=1]
Epoch 11:  86%|########6 | 19/22 [00:04<00:00,  4.14it/s, loss=4.57, v_num=1]
Epoch 11:  86%|########6 | 19/22 [00:04<00:00,  4.08it/s, loss=4.57, v_num=1]
Epoch 11:  91%|######### | 20/22 [00:04<00:00,  4.22it/s, loss=4.57, v_num=1]
Epoch 11:  91%|######### | 20/22 [00:04<00:00,  4.16it/s, loss=4.56, v_num=1]
Epoch 11:  95%|#########5| 21/22 [00:04<00:00,  4.30it/s, loss=4.56, v_num=1]
Epoch 11:  95%|#########5| 21/22 [00:04<00:00,  4.24it/s, loss=4.56, v_num=1]
Epoch 11: 100%|##########| 22/22 [00:05<00:00,  4.36it/s, loss=4.56, v_num=1]
Epoch 11: 100%|##########| 22/22 [00:05<00:00,  4.31it/s, loss=4.56, v_num=1]
Epoch 11: 100%|##########| 22/22 [00:05<00:00,  4.31it/s, loss=4.56, v_num=1]
Epoch 11:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.56, v_num=1]
Epoch 12:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.56, v_num=1]
Epoch 12:   5%|4         | 1/22 [00:01<00:38,  1.85s/it, loss=4.56, v_num=1]
Epoch 12:   5%|4         | 1/22 [00:01<00:40,  1.91s/it, loss=4.56, v_num=1]
Epoch 12:   9%|9         | 2/22 [00:02<00:20,  1.01s/it, loss=4.56, v_num=1]
Epoch 12:   9%|9         | 2/22 [00:02<00:20,  1.04s/it, loss=4.56, v_num=1]
Epoch 12:  14%|#3        | 3/22 [00:02<00:13,  1.38it/s, loss=4.56, v_num=1]
Epoch 12:  14%|#3        | 3/22 [00:02<00:14,  1.34it/s, loss=4.56, v_num=1]
Epoch 12:  18%|#8        | 4/22 [00:02<00:10,  1.72it/s, loss=4.56, v_num=1]
Epoch 12:  18%|#8        | 4/22 [00:02<00:10,  1.68it/s, loss=4.56, v_num=1]
Epoch 12:  23%|##2       | 5/22 [00:02<00:08,  2.02it/s, loss=4.56, v_num=1]
Epoch 12:  23%|##2       | 5/22 [00:02<00:08,  1.97it/s, loss=4.56, v_num=1]
Epoch 12:  27%|##7       | 6/22 [00:02<00:07,  2.27it/s, loss=4.56, v_num=1]
Epoch 12:  27%|##7       | 6/22 [00:02<00:07,  2.22it/s, loss=4.56, v_num=1]
Epoch 12:  32%|###1      | 7/22 [00:02<00:06,  2.49it/s, loss=4.56, v_num=1]
Epoch 12:  32%|###1      | 7/22 [00:02<00:06,  2.44it/s, loss=4.56, v_num=1]
Epoch 12:  36%|###6      | 8/22 [00:02<00:05,  2.70it/s, loss=4.56, v_num=1]
Epoch 12:  36%|###6      | 8/22 [00:03<00:05,  2.65it/s, loss=4.56, v_num=1]
Epoch 12:  41%|####      | 9/22 [00:03<00:04,  2.88it/s, loss=4.56, v_num=1]
Epoch 12:  41%|####      | 9/22 [00:03<00:04,  2.82it/s, loss=4.56, v_num=1]
Epoch 12:  45%|####5     | 10/22 [00:03<00:03,  3.05it/s, loss=4.56, v_num=1]
Epoch 12:  45%|####5     | 10/22 [00:03<00:04,  2.99it/s, loss=4.56, v_num=1]
Epoch 12:  50%|#####     | 11/22 [00:03<00:03,  3.20it/s, loss=4.56, v_num=1]
Epoch 12:  50%|#####     | 11/22 [00:03<00:03,  3.15it/s, loss=4.56, v_num=1]
Epoch 12:  55%|#####4    | 12/22 [00:03<00:02,  3.34it/s, loss=4.56, v_num=1]
Epoch 12:  55%|#####4    | 12/22 [00:03<00:03,  3.29it/s, loss=4.56, v_num=1]
Epoch 12:  59%|#####9    | 13/22 [00:03<00:02,  3.47it/s, loss=4.56, v_num=1]
Epoch 12:  59%|#####9    | 13/22 [00:03<00:02,  3.42it/s, loss=4.56, v_num=1]
Epoch 12:  64%|######3   | 14/22 [00:03<00:02,  3.59it/s, loss=4.56, v_num=1]
Epoch 12:  64%|######3   | 14/22 [00:03<00:02,  3.54it/s, loss=4.56, v_num=1]
Epoch 12:  68%|######8   | 15/22 [00:04<00:01,  3.70it/s, loss=4.56, v_num=1]
Epoch 12:  68%|######8   | 15/22 [00:04<00:01,  3.65it/s, loss=4.56, v_num=1]
Epoch 12:  73%|#######2  | 16/22 [00:04<00:01,  3.81it/s, loss=4.56, v_num=1]
Epoch 12:  73%|#######2  | 16/22 [00:04<00:01,  3.75it/s, loss=4.56, v_num=1]
Epoch 12:  77%|#######7  | 17/22 [00:04<00:01,  3.90it/s, loss=4.56, v_num=1]
Epoch 12:  77%|#######7  | 17/22 [00:04<00:01,  3.84it/s, loss=4.56, v_num=1]
Epoch 12:  82%|########1 | 18/22 [00:04<00:01,  3.99it/s, loss=4.56, v_num=1]
Epoch 12:  82%|########1 | 18/22 [00:04<00:01,  3.93it/s, loss=4.56, v_num=1]
Epoch 12:  86%|########6 | 19/22 [00:04<00:00,  4.07it/s, loss=4.56, v_num=1]
Epoch 12:  86%|########6 | 19/22 [00:04<00:00,  4.02it/s, loss=4.56, v_num=1]
Epoch 12:  91%|######### | 20/22 [00:04<00:00,  4.16it/s, loss=4.56, v_num=1]
Epoch 12:  91%|######### | 20/22 [00:04<00:00,  4.10it/s, loss=4.56, v_num=1]
Epoch 12:  95%|#########5| 21/22 [00:04<00:00,  4.23it/s, loss=4.56, v_num=1]
Epoch 12:  95%|#########5| 21/22 [00:05<00:00,  4.18it/s, loss=4.56, v_num=1]
Epoch 12: 100%|##########| 22/22 [00:05<00:00,  4.30it/s, loss=4.56, v_num=1]
Epoch 12: 100%|##########| 22/22 [00:05<00:00,  4.25it/s, loss=4.55, v_num=1]
Epoch 12: 100%|##########| 22/22 [00:05<00:00,  4.25it/s, loss=4.55, v_num=1]
Epoch 12:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.55, v_num=1]
Epoch 13:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.55, v_num=1]
Epoch 13:   5%|4         | 1/22 [00:01<00:38,  1.82s/it, loss=4.55, v_num=1]
Epoch 13:   5%|4         | 1/22 [00:01<00:39,  1.88s/it, loss=4.55, v_num=1]
Epoch 13:   9%|9         | 2/22 [00:01<00:19,  1.00it/s, loss=4.55, v_num=1]
Epoch 13:   9%|9         | 2/22 [00:02<00:20,  1.03s/it, loss=4.55, v_num=1]
Epoch 13:  14%|#3        | 3/22 [00:02<00:13,  1.39it/s, loss=4.55, v_num=1]
Epoch 13:  14%|#3        | 3/22 [00:02<00:14,  1.35it/s, loss=4.55, v_num=1]
Epoch 13:  18%|#8        | 4/22 [00:02<00:10,  1.72it/s, loss=4.55, v_num=1]
Epoch 13:  18%|#8        | 4/22 [00:02<00:10,  1.68it/s, loss=4.55, v_num=1]
Epoch 13:  23%|##2       | 5/22 [00:02<00:08,  2.02it/s, loss=4.55, v_num=1]
Epoch 13:  23%|##2       | 5/22 [00:02<00:08,  1.97it/s, loss=4.55, v_num=1]
Epoch 13:  27%|##7       | 6/22 [00:02<00:07,  2.27it/s, loss=4.55, v_num=1]
Epoch 13:  27%|##7       | 6/22 [00:02<00:07,  2.23it/s, loss=4.55, v_num=1]
Epoch 13:  32%|###1      | 7/22 [00:02<00:05,  2.50it/s, loss=4.55, v_num=1]
Epoch 13:  32%|###1      | 7/22 [00:02<00:06,  2.45it/s, loss=4.55, v_num=1]
Epoch 13:  36%|###6      | 8/22 [00:02<00:05,  2.71it/s, loss=4.55, v_num=1]
Epoch 13:  36%|###6      | 8/22 [00:03<00:05,  2.65it/s, loss=4.55, v_num=1]
Epoch 13:  41%|####      | 9/22 [00:03<00:04,  2.89it/s, loss=4.55, v_num=1]
Epoch 13:  41%|####      | 9/22 [00:03<00:04,  2.84it/s, loss=4.55, v_num=1]
Epoch 13:  45%|####5     | 10/22 [00:03<00:03,  3.05it/s, loss=4.55, v_num=1]
Epoch 13:  45%|####5     | 10/22 [00:03<00:04,  3.00it/s, loss=4.55, v_num=1]
Epoch 13:  50%|#####     | 11/22 [00:03<00:03,  3.21it/s, loss=4.55, v_num=1]
Epoch 13:  50%|#####     | 11/22 [00:03<00:03,  3.15it/s, loss=4.55, v_num=1]
Epoch 13:  55%|#####4    | 12/22 [00:03<00:02,  3.34it/s, loss=4.55, v_num=1]
Epoch 13:  55%|#####4    | 12/22 [00:03<00:03,  3.29it/s, loss=4.54, v_num=1]
Epoch 13:  59%|#####9    | 13/22 [00:03<00:02,  3.47it/s, loss=4.54, v_num=1]
Epoch 13:  59%|#####9    | 13/22 [00:03<00:02,  3.42it/s, loss=4.54, v_num=1]
Epoch 13:  64%|######3   | 14/22 [00:03<00:02,  3.60it/s, loss=4.54, v_num=1]
Epoch 13:  64%|######3   | 14/22 [00:03<00:02,  3.54it/s, loss=4.54, v_num=1]
Epoch 13:  68%|######8   | 15/22 [00:04<00:01,  3.71it/s, loss=4.54, v_num=1]
Epoch 13:  68%|######8   | 15/22 [00:04<00:01,  3.65it/s, loss=4.54, v_num=1]
Epoch 13:  73%|#######2  | 16/22 [00:04<00:01,  3.82it/s, loss=4.54, v_num=1]
Epoch 13:  73%|#######2  | 16/22 [00:04<00:01,  3.76it/s, loss=4.54, v_num=1]
Epoch 13:  77%|#######7  | 17/22 [00:04<00:01,  3.91it/s, loss=4.54, v_num=1]
Epoch 13:  77%|#######7  | 17/22 [00:04<00:01,  3.85it/s, loss=4.54, v_num=1]
Epoch 13:  82%|########1 | 18/22 [00:04<00:01,  4.00it/s, loss=4.54, v_num=1]
Epoch 13:  82%|########1 | 18/22 [00:04<00:01,  3.94it/s, loss=4.54, v_num=1]
Epoch 13:  86%|########6 | 19/22 [00:04<00:00,  4.08it/s, loss=4.54, v_num=1]
Epoch 13:  86%|########6 | 19/22 [00:04<00:00,  4.02it/s, loss=4.55, v_num=1]
Epoch 13:  91%|######### | 20/22 [00:04<00:00,  4.16it/s, loss=4.55, v_num=1]
Epoch 13:  91%|######### | 20/22 [00:04<00:00,  4.11it/s, loss=4.55, v_num=1]
Epoch 13:  95%|#########5| 21/22 [00:04<00:00,  4.23it/s, loss=4.55, v_num=1]
Epoch 13:  95%|#########5| 21/22 [00:05<00:00,  4.18it/s, loss=4.55, v_num=1]
Epoch 13: 100%|##########| 22/22 [00:05<00:00,  4.30it/s, loss=4.55, v_num=1]
Epoch 13: 100%|##########| 22/22 [00:05<00:00,  4.25it/s, loss=4.54, v_num=1]
Epoch 13: 100%|##########| 22/22 [00:05<00:00,  4.25it/s, loss=4.54, v_num=1]
Epoch 13:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 14:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 14:   5%|4         | 1/22 [00:01<00:39,  1.88s/it, loss=4.54, v_num=1]
Epoch 14:   5%|4         | 1/22 [00:01<00:40,  1.94s/it, loss=4.54, v_num=1]
Epoch 14:   9%|9         | 2/22 [00:02<00:20,  1.02s/it, loss=4.54, v_num=1]
Epoch 14:   9%|9         | 2/22 [00:02<00:21,  1.05s/it, loss=4.54, v_num=1]
Epoch 14:  14%|#3        | 3/22 [00:02<00:14,  1.35it/s, loss=4.54, v_num=1]
Epoch 14:  14%|#3        | 3/22 [00:02<00:14,  1.32it/s, loss=4.54, v_num=1]
Epoch 14:  18%|#8        | 4/22 [00:02<00:10,  1.68it/s, loss=4.54, v_num=1]
Epoch 14:  18%|#8        | 4/22 [00:02<00:10,  1.64it/s, loss=4.54, v_num=1]
Epoch 14:  23%|##2       | 5/22 [00:02<00:08,  1.97it/s, loss=4.54, v_num=1]
Epoch 14:  23%|##2       | 5/22 [00:02<00:08,  1.92it/s, loss=4.54, v_num=1]
Epoch 14:  27%|##7       | 6/22 [00:02<00:07,  2.22it/s, loss=4.54, v_num=1]
Epoch 14:  27%|##7       | 6/22 [00:02<00:07,  2.17it/s, loss=4.54, v_num=1]
Epoch 14:  32%|###1      | 7/22 [00:02<00:06,  2.40it/s, loss=4.54, v_num=1]
Epoch 14:  32%|###1      | 7/22 [00:02<00:06,  2.35it/s, loss=4.54, v_num=1]
Epoch 14:  36%|###6      | 8/22 [00:03<00:05,  2.59it/s, loss=4.54, v_num=1]
Epoch 14:  36%|###6      | 8/22 [00:03<00:05,  2.54it/s, loss=4.54, v_num=1]
Epoch 14:  41%|####      | 9/22 [00:03<00:04,  2.77it/s, loss=4.54, v_num=1]
Epoch 14:  41%|####      | 9/22 [00:03<00:04,  2.72it/s, loss=4.54, v_num=1]
Epoch 14:  45%|####5     | 10/22 [00:03<00:04,  2.94it/s, loss=4.54, v_num=1]
Epoch 14:  45%|####5     | 10/22 [00:03<00:04,  2.89it/s, loss=4.54, v_num=1]
Epoch 14:  50%|#####     | 11/22 [00:03<00:03,  3.09it/s, loss=4.54, v_num=1]
Epoch 14:  50%|#####     | 11/22 [00:03<00:03,  3.04it/s, loss=4.54, v_num=1]
Epoch 14:  55%|#####4    | 12/22 [00:03<00:03,  3.22it/s, loss=4.54, v_num=1]
Epoch 14:  55%|#####4    | 12/22 [00:03<00:03,  3.17it/s, loss=4.54, v_num=1]
Epoch 14:  59%|#####9    | 13/22 [00:03<00:02,  3.32it/s, loss=4.54, v_num=1]
Epoch 14:  59%|#####9    | 13/22 [00:03<00:02,  3.27it/s, loss=4.54, v_num=1]
Epoch 14:  64%|######3   | 14/22 [00:04<00:02,  3.44it/s, loss=4.54, v_num=1]
Epoch 14:  64%|######3   | 14/22 [00:04<00:02,  3.39it/s, loss=4.54, v_num=1]
Epoch 14:  68%|######8   | 15/22 [00:04<00:01,  3.56it/s, loss=4.54, v_num=1]
Epoch 14:  68%|######8   | 15/22 [00:04<00:01,  3.50it/s, loss=4.54, v_num=1]
Epoch 14:  73%|#######2  | 16/22 [00:04<00:01,  3.66it/s, loss=4.54, v_num=1]
Epoch 14:  73%|#######2  | 16/22 [00:04<00:01,  3.61it/s, loss=4.54, v_num=1]
Epoch 14:  77%|#######7  | 17/22 [00:04<00:01,  3.75it/s, loss=4.54, v_num=1]
Epoch 14:  77%|#######7  | 17/22 [00:04<00:01,  3.70it/s, loss=4.54, v_num=1]
Epoch 14:  82%|########1 | 18/22 [00:04<00:01,  3.84it/s, loss=4.54, v_num=1]
Epoch 14:  82%|########1 | 18/22 [00:04<00:01,  3.79it/s, loss=4.54, v_num=1]
Epoch 14:  86%|########6 | 19/22 [00:04<00:00,  3.92it/s, loss=4.54, v_num=1]
Epoch 14:  86%|########6 | 19/22 [00:04<00:00,  3.87it/s, loss=4.54, v_num=1]
Epoch 14:  91%|######### | 20/22 [00:04<00:00,  4.00it/s, loss=4.54, v_num=1]
Epoch 14:  91%|######### | 20/22 [00:05<00:00,  3.96it/s, loss=4.54, v_num=1]
Epoch 14:  95%|#########5| 21/22 [00:05<00:00,  4.08it/s, loss=4.54, v_num=1]
Epoch 14:  95%|#########5| 21/22 [00:05<00:00,  4.03it/s, loss=4.54, v_num=1]
Epoch 14: 100%|##########| 22/22 [00:05<00:00,  4.15it/s, loss=4.54, v_num=1]
Epoch 14: 100%|##########| 22/22 [00:05<00:00,  4.10it/s, loss=4.54, v_num=1]
Epoch 14: 100%|##########| 22/22 [00:05<00:00,  4.10it/s, loss=4.54, v_num=1]
Epoch 14:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 15:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 15:   5%|4         | 1/22 [00:01<00:38,  1.85s/it, loss=4.54, v_num=1]
Epoch 15:   5%|4         | 1/22 [00:01<00:40,  1.91s/it, loss=4.54, v_num=1]
Epoch 15:   9%|9         | 2/22 [00:02<00:20,  1.00s/it, loss=4.54, v_num=1]
Epoch 15:   9%|9         | 2/22 [00:02<00:20,  1.03s/it, loss=4.54, v_num=1]
Epoch 15:  14%|#3        | 3/22 [00:02<00:13,  1.39it/s, loss=4.54, v_num=1]
Epoch 15:  14%|#3        | 3/22 [00:02<00:14,  1.35it/s, loss=4.54, v_num=1]
Epoch 15:  18%|#8        | 4/22 [00:02<00:10,  1.73it/s, loss=4.54, v_num=1]
Epoch 15:  18%|#8        | 4/22 [00:02<00:10,  1.68it/s, loss=4.54, v_num=1]
Epoch 15:  23%|##2       | 5/22 [00:02<00:08,  2.02it/s, loss=4.54, v_num=1]
Epoch 15:  23%|##2       | 5/22 [00:02<00:08,  1.97it/s, loss=4.54, v_num=1]
Epoch 15:  27%|##7       | 6/22 [00:02<00:07,  2.27it/s, loss=4.54, v_num=1]
Epoch 15:  27%|##7       | 6/22 [00:02<00:07,  2.22it/s, loss=4.54, v_num=1]
Epoch 15:  32%|###1      | 7/22 [00:02<00:06,  2.50it/s, loss=4.54, v_num=1]
Epoch 15:  32%|###1      | 7/22 [00:02<00:06,  2.44it/s, loss=4.54, v_num=1]
Epoch 15:  36%|###6      | 8/22 [00:02<00:05,  2.70it/s, loss=4.54, v_num=1]
Epoch 15:  36%|###6      | 8/22 [00:03<00:05,  2.65it/s, loss=4.54, v_num=1]
Epoch 15:  41%|####      | 9/22 [00:03<00:04,  2.88it/s, loss=4.54, v_num=1]
Epoch 15:  41%|####      | 9/22 [00:03<00:04,  2.82it/s, loss=4.54, v_num=1]
Epoch 15:  45%|####5     | 10/22 [00:03<00:03,  3.05it/s, loss=4.54, v_num=1]
Epoch 15:  45%|####5     | 10/22 [00:03<00:04,  2.99it/s, loss=4.54, v_num=1]
Epoch 15:  50%|#####     | 11/22 [00:03<00:03,  3.20it/s, loss=4.54, v_num=1]
Epoch 15:  50%|#####     | 11/22 [00:03<00:03,  3.15it/s, loss=4.54, v_num=1]
Epoch 15:  55%|#####4    | 12/22 [00:03<00:02,  3.34it/s, loss=4.54, v_num=1]
Epoch 15:  55%|#####4    | 12/22 [00:03<00:03,  3.28it/s, loss=4.54, v_num=1]
Epoch 15:  59%|#####9    | 13/22 [00:03<00:02,  3.47it/s, loss=4.54, v_num=1]
Epoch 15:  59%|#####9    | 13/22 [00:03<00:02,  3.41it/s, loss=4.54, v_num=1]
Epoch 15:  64%|######3   | 14/22 [00:03<00:02,  3.59it/s, loss=4.54, v_num=1]
Epoch 15:  64%|######3   | 14/22 [00:03<00:02,  3.53it/s, loss=4.54, v_num=1]
Epoch 15:  68%|######8   | 15/22 [00:04<00:01,  3.70it/s, loss=4.54, v_num=1]
Epoch 15:  68%|######8   | 15/22 [00:04<00:01,  3.65it/s, loss=4.54, v_num=1]
Epoch 15:  73%|#######2  | 16/22 [00:04<00:01,  3.80it/s, loss=4.54, v_num=1]
Epoch 15:  73%|#######2  | 16/22 [00:04<00:01,  3.75it/s, loss=4.54, v_num=1]
Epoch 15:  77%|#######7  | 17/22 [00:04<00:01,  3.90it/s, loss=4.54, v_num=1]
Epoch 15:  77%|#######7  | 17/22 [00:04<00:01,  3.84it/s, loss=4.54, v_num=1]
Epoch 15:  82%|########1 | 18/22 [00:04<00:01,  3.98it/s, loss=4.54, v_num=1]
Epoch 15:  82%|########1 | 18/22 [00:04<00:01,  3.93it/s, loss=4.54, v_num=1]
Epoch 15:  86%|########6 | 19/22 [00:04<00:00,  4.06it/s, loss=4.54, v_num=1]
Epoch 15:  86%|########6 | 19/22 [00:04<00:00,  4.01it/s, loss=4.54, v_num=1]
Epoch 15:  91%|######### | 20/22 [00:04<00:00,  4.14it/s, loss=4.54, v_num=1]
Epoch 15:  91%|######### | 20/22 [00:04<00:00,  4.08it/s, loss=4.54, v_num=1]
Epoch 15:  95%|#########5| 21/22 [00:04<00:00,  4.20it/s, loss=4.54, v_num=1]
Epoch 15:  95%|#########5| 21/22 [00:05<00:00,  4.16it/s, loss=4.54, v_num=1]
Epoch 15: 100%|##########| 22/22 [00:05<00:00,  4.28it/s, loss=4.54, v_num=1]
Epoch 15: 100%|##########| 22/22 [00:05<00:00,  4.23it/s, loss=4.54, v_num=1]
Epoch 15: 100%|##########| 22/22 [00:05<00:00,  4.23it/s, loss=4.54, v_num=1]
Epoch 15:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 16:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 16:   5%|4         | 1/22 [00:02<00:43,  2.06s/it, loss=4.54, v_num=1]
Epoch 16:   5%|4         | 1/22 [00:02<00:44,  2.12s/it, loss=4.54, v_num=1]
Epoch 16:   9%|9         | 2/22 [00:02<00:22,  1.12s/it, loss=4.54, v_num=1]
Epoch 16:   9%|9         | 2/22 [00:02<00:22,  1.14s/it, loss=4.54, v_num=1]
Epoch 16:  14%|#3        | 3/22 [00:02<00:15,  1.25it/s, loss=4.54, v_num=1]
Epoch 16:  14%|#3        | 3/22 [00:02<00:15,  1.22it/s, loss=4.54, v_num=1]
Epoch 16:  18%|#8        | 4/22 [00:02<00:11,  1.56it/s, loss=4.54, v_num=1]
Epoch 16:  18%|#8        | 4/22 [00:02<00:11,  1.52it/s, loss=4.54, v_num=1]
Epoch 16:  23%|##2       | 5/22 [00:02<00:09,  1.82it/s, loss=4.54, v_num=1]
Epoch 16:  23%|##2       | 5/22 [00:02<00:09,  1.79it/s, loss=4.54, v_num=1]
Epoch 16:  27%|##7       | 6/22 [00:02<00:07,  2.07it/s, loss=4.54, v_num=1]
Epoch 16:  27%|##7       | 6/22 [00:02<00:07,  2.02it/s, loss=4.54, v_num=1]
Epoch 16:  32%|###1      | 7/22 [00:03<00:06,  2.27it/s, loss=4.54, v_num=1]
Epoch 16:  32%|###1      | 7/22 [00:03<00:06,  2.23it/s, loss=4.54, v_num=1]
Epoch 16:  36%|###6      | 8/22 [00:03<00:05,  2.46it/s, loss=4.54, v_num=1]
Epoch 16:  36%|###6      | 8/22 [00:03<00:05,  2.42it/s, loss=4.54, v_num=1]
Epoch 16:  41%|####      | 9/22 [00:03<00:04,  2.63it/s, loss=4.54, v_num=1]
Epoch 16:  41%|####      | 9/22 [00:03<00:05,  2.59it/s, loss=4.54, v_num=1]
Epoch 16:  45%|####5     | 10/22 [00:03<00:04,  2.79it/s, loss=4.54, v_num=1]
Epoch 16:  45%|####5     | 10/22 [00:03<00:04,  2.75it/s, loss=4.54, v_num=1]
Epoch 16:  50%|#####     | 11/22 [00:03<00:03,  2.94it/s, loss=4.54, v_num=1]
Epoch 16:  50%|#####     | 11/22 [00:03<00:03,  2.89it/s, loss=4.54, v_num=1]
Epoch 16:  55%|#####4    | 12/22 [00:03<00:03,  3.07it/s, loss=4.54, v_num=1]
Epoch 16:  55%|#####4    | 12/22 [00:03<00:03,  3.03it/s, loss=4.54, v_num=1]
Epoch 16:  59%|#####9    | 13/22 [00:04<00:02,  3.19it/s, loss=4.54, v_num=1]
Epoch 16:  59%|#####9    | 13/22 [00:04<00:02,  3.15it/s, loss=4.54, v_num=1]
Epoch 16:  64%|######3   | 14/22 [00:04<00:02,  3.30it/s, loss=4.54, v_num=1]
Epoch 16:  64%|######3   | 14/22 [00:04<00:02,  3.26it/s, loss=4.54, v_num=1]
Epoch 16:  68%|######8   | 15/22 [00:04<00:02,  3.40it/s, loss=4.54, v_num=1]
Epoch 16:  68%|######8   | 15/22 [00:04<00:02,  3.36it/s, loss=4.54, v_num=1]
Epoch 16:  73%|#######2  | 16/22 [00:04<00:01,  3.51it/s, loss=4.54, v_num=1]
Epoch 16:  73%|#######2  | 16/22 [00:04<00:01,  3.46it/s, loss=4.54, v_num=1]
Epoch 16:  77%|#######7  | 17/22 [00:04<00:01,  3.60it/s, loss=4.54, v_num=1]
Epoch 16:  77%|#######7  | 17/22 [00:04<00:01,  3.56it/s, loss=4.54, v_num=1]
Epoch 16:  82%|########1 | 18/22 [00:04<00:01,  3.69it/s, loss=4.54, v_num=1]
Epoch 16:  82%|########1 | 18/22 [00:04<00:01,  3.64it/s, loss=4.54, v_num=1]
Epoch 16:  86%|########6 | 19/22 [00:05<00:00,  3.76it/s, loss=4.54, v_num=1]
Epoch 16:  86%|########6 | 19/22 [00:05<00:00,  3.72it/s, loss=4.54, v_num=1]
Epoch 16:  91%|######### | 20/22 [00:05<00:00,  3.84it/s, loss=4.54, v_num=1]
Epoch 16:  91%|######### | 20/22 [00:05<00:00,  3.80it/s, loss=4.54, v_num=1]
Epoch 16:  95%|#########5| 21/22 [00:05<00:00,  3.91it/s, loss=4.54, v_num=1]
Epoch 16:  95%|#########5| 21/22 [00:05<00:00,  3.87it/s, loss=4.54, v_num=1]
Epoch 16: 100%|##########| 22/22 [00:05<00:00,  3.98it/s, loss=4.54, v_num=1]
Epoch 16: 100%|##########| 22/22 [00:05<00:00,  3.94it/s, loss=4.54, v_num=1]
Epoch 16: 100%|##########| 22/22 [00:05<00:00,  3.94it/s, loss=4.54, v_num=1]
Epoch 16:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 17:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 17:   5%|4         | 1/22 [00:02<00:47,  2.25s/it, loss=4.54, v_num=1]
Epoch 17:   5%|4         | 1/22 [00:02<00:48,  2.31s/it, loss=4.54, v_num=1]
Epoch 17:   9%|9         | 2/22 [00:02<00:24,  1.22s/it, loss=4.54, v_num=1]
Epoch 17:   9%|9         | 2/22 [00:02<00:24,  1.25s/it, loss=4.54, v_num=1]
Epoch 17:  14%|#3        | 3/22 [00:02<00:16,  1.15it/s, loss=4.54, v_num=1]
Epoch 17:  14%|#3        | 3/22 [00:02<00:16,  1.13it/s, loss=4.54, v_num=1]
Epoch 17:  18%|#8        | 4/22 [00:02<00:12,  1.45it/s, loss=4.54, v_num=1]
Epoch 17:  18%|#8        | 4/22 [00:02<00:12,  1.42it/s, loss=4.54, v_num=1]
Epoch 17:  23%|##2       | 5/22 [00:02<00:09,  1.70it/s, loss=4.54, v_num=1]
Epoch 17:  23%|##2       | 5/22 [00:02<00:10,  1.67it/s, loss=4.54, v_num=1]
Epoch 17:  27%|##7       | 6/22 [00:03<00:08,  1.93it/s, loss=4.54, v_num=1]
Epoch 17:  27%|##7       | 6/22 [00:03<00:08,  1.89it/s, loss=4.54, v_num=1]
Epoch 17:  32%|###1      | 7/22 [00:03<00:07,  2.13it/s, loss=4.54, v_num=1]
Epoch 17:  32%|###1      | 7/22 [00:03<00:07,  2.09it/s, loss=4.54, v_num=1]
Epoch 17:  36%|###6      | 8/22 [00:03<00:06,  2.32it/s, loss=4.54, v_num=1]
Epoch 17:  36%|###6      | 8/22 [00:03<00:06,  2.28it/s, loss=4.54, v_num=1]
Epoch 17:  41%|####      | 9/22 [00:03<00:05,  2.48it/s, loss=4.54, v_num=1]
Epoch 17:  41%|####      | 9/22 [00:03<00:05,  2.44it/s, loss=4.54, v_num=1]
Epoch 17:  45%|####5     | 10/22 [00:03<00:04,  2.64it/s, loss=4.54, v_num=1]
Epoch 17:  45%|####5     | 10/22 [00:03<00:04,  2.60it/s, loss=4.54, v_num=1]
Epoch 17:  50%|#####     | 11/22 [00:03<00:03,  2.79it/s, loss=4.54, v_num=1]
Epoch 17:  50%|#####     | 11/22 [00:04<00:04,  2.74it/s, loss=4.54, v_num=1]
Epoch 17:  55%|#####4    | 12/22 [00:04<00:03,  2.92it/s, loss=4.54, v_num=1]
Epoch 17:  55%|#####4    | 12/22 [00:04<00:03,  2.88it/s, loss=4.54, v_num=1]
Epoch 17:  59%|#####9    | 13/22 [00:04<00:02,  3.05it/s, loss=4.54, v_num=1]
Epoch 17:  59%|#####9    | 13/22 [00:04<00:02,  3.00it/s, loss=4.54, v_num=1]
Epoch 17:  64%|######3   | 14/22 [00:04<00:02,  3.16it/s, loss=4.54, v_num=1]
Epoch 17:  64%|######3   | 14/22 [00:04<00:02,  3.12it/s, loss=4.54, v_num=1]
Epoch 17:  68%|######8   | 15/22 [00:04<00:02,  3.26it/s, loss=4.54, v_num=1]
Epoch 17:  68%|######8   | 15/22 [00:04<00:02,  3.22it/s, loss=4.54, v_num=1]
Epoch 17:  73%|#######2  | 16/22 [00:04<00:01,  3.36it/s, loss=4.54, v_num=1]
Epoch 17:  73%|#######2  | 16/22 [00:04<00:01,  3.31it/s, loss=4.54, v_num=1]
Epoch 17:  77%|#######7  | 17/22 [00:04<00:01,  3.45it/s, loss=4.54, v_num=1]
Epoch 17:  77%|#######7  | 17/22 [00:04<00:01,  3.41it/s, loss=4.53, v_num=1]
Epoch 17:  82%|########1 | 18/22 [00:05<00:01,  3.54it/s, loss=4.53, v_num=1]
Epoch 17:  82%|########1 | 18/22 [00:05<00:01,  3.50it/s, loss=4.53, v_num=1]
Epoch 17:  86%|########6 | 19/22 [00:05<00:00,  3.62it/s, loss=4.53, v_num=1]
Epoch 17:  86%|########6 | 19/22 [00:05<00:00,  3.58it/s, loss=4.53, v_num=1]
Epoch 17:  91%|######### | 20/22 [00:05<00:00,  3.70it/s, loss=4.53, v_num=1]
Epoch 17:  91%|######### | 20/22 [00:05<00:00,  3.66it/s, loss=4.53, v_num=1]
Epoch 17:  95%|#########5| 21/22 [00:05<00:00,  3.77it/s, loss=4.53, v_num=1]
Epoch 17:  95%|#########5| 21/22 [00:05<00:00,  3.73it/s, loss=4.53, v_num=1]
Epoch 17: 100%|##########| 22/22 [00:05<00:00,  3.84it/s, loss=4.53, v_num=1]
Epoch 17: 100%|##########| 22/22 [00:05<00:00,  3.80it/s, loss=4.53, v_num=1]
Epoch 17: 100%|##########| 22/22 [00:05<00:00,  3.80it/s, loss=4.53, v_num=1]
Epoch 17:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 18:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 18:   5%|4         | 1/22 [00:02<00:47,  2.28s/it, loss=4.53, v_num=1]
Epoch 18:   5%|4         | 1/22 [00:02<00:49,  2.33s/it, loss=4.53, v_num=1]
Epoch 18:   9%|9         | 2/22 [00:02<00:24,  1.22s/it, loss=4.53, v_num=1]
Epoch 18:   9%|9         | 2/22 [00:02<00:25,  1.25s/it, loss=4.53, v_num=1]
Epoch 18:  14%|#3        | 3/22 [00:02<00:16,  1.15it/s, loss=4.53, v_num=1]
Epoch 18:  14%|#3        | 3/22 [00:02<00:16,  1.12it/s, loss=4.53, v_num=1]
Epoch 18:  18%|#8        | 4/22 [00:02<00:12,  1.44it/s, loss=4.53, v_num=1]
Epoch 18:  18%|#8        | 4/22 [00:02<00:12,  1.41it/s, loss=4.53, v_num=1]
Epoch 18:  23%|##2       | 5/22 [00:02<00:09,  1.70it/s, loss=4.53, v_num=1]
Epoch 18:  23%|##2       | 5/22 [00:02<00:10,  1.67it/s, loss=4.53, v_num=1]
Epoch 18:  27%|##7       | 6/22 [00:03<00:08,  1.94it/s, loss=4.53, v_num=1]
Epoch 18:  27%|##7       | 6/22 [00:03<00:08,  1.90it/s, loss=4.53, v_num=1]
Epoch 18:  32%|###1      | 7/22 [00:03<00:07,  2.14it/s, loss=4.53, v_num=1]
Epoch 18:  32%|###1      | 7/22 [00:03<00:07,  2.10it/s, loss=4.52, v_num=1]
Epoch 18:  36%|###6      | 8/22 [00:03<00:06,  2.32it/s, loss=4.52, v_num=1]
Epoch 18:  36%|###6      | 8/22 [00:03<00:06,  2.28it/s, loss=4.52, v_num=1]
Epoch 18:  41%|####      | 9/22 [00:03<00:05,  2.48it/s, loss=4.52, v_num=1]
Epoch 18:  41%|####      | 9/22 [00:03<00:05,  2.44it/s, loss=4.53, v_num=1]
Epoch 18:  45%|####5     | 10/22 [00:03<00:04,  2.63it/s, loss=4.53, v_num=1]
Epoch 18:  45%|####5     | 10/22 [00:03<00:04,  2.59it/s, loss=4.53, v_num=1]
Epoch 18:  50%|#####     | 11/22 [00:03<00:03,  2.78it/s, loss=4.53, v_num=1]
Epoch 18:  50%|#####     | 11/22 [00:04<00:04,  2.74it/s, loss=4.53, v_num=1]
Epoch 18:  55%|#####4    | 12/22 [00:04<00:03,  2.91it/s, loss=4.53, v_num=1]
Epoch 18:  55%|#####4    | 12/22 [00:04<00:03,  2.86it/s, loss=4.53, v_num=1]
Epoch 18:  59%|#####9    | 13/22 [00:04<00:02,  3.03it/s, loss=4.53, v_num=1]
Epoch 18:  59%|#####9    | 13/22 [00:04<00:03,  2.98it/s, loss=4.53, v_num=1]
Epoch 18:  64%|######3   | 14/22 [00:04<00:02,  3.14it/s, loss=4.53, v_num=1]
Epoch 18:  64%|######3   | 14/22 [00:04<00:02,  3.10it/s, loss=4.53, v_num=1]
Epoch 18:  68%|######8   | 15/22 [00:04<00:02,  3.25it/s, loss=4.53, v_num=1]
Epoch 18:  68%|######8   | 15/22 [00:04<00:02,  3.20it/s, loss=4.53, v_num=1]
Epoch 18:  73%|#######2  | 16/22 [00:04<00:01,  3.35it/s, loss=4.53, v_num=1]
Epoch 18:  73%|#######2  | 16/22 [00:04<00:01,  3.31it/s, loss=4.53, v_num=1]
Epoch 18:  77%|#######7  | 17/22 [00:04<00:01,  3.45it/s, loss=4.53, v_num=1]
Epoch 18:  77%|#######7  | 17/22 [00:04<00:01,  3.40it/s, loss=4.53, v_num=1]
Epoch 18:  82%|########1 | 18/22 [00:05<00:01,  3.54it/s, loss=4.53, v_num=1]
Epoch 18:  82%|########1 | 18/22 [00:05<00:01,  3.49it/s, loss=4.53, v_num=1]
Epoch 18:  86%|########6 | 19/22 [00:05<00:00,  3.62it/s, loss=4.53, v_num=1]
Epoch 18:  86%|########6 | 19/22 [00:05<00:00,  3.58it/s, loss=4.53, v_num=1]
Epoch 18:  91%|######### | 20/22 [00:05<00:00,  3.70it/s, loss=4.53, v_num=1]
Epoch 18:  91%|######### | 20/22 [00:05<00:00,  3.65it/s, loss=4.53, v_num=1]
Epoch 18:  95%|#########5| 21/22 [00:05<00:00,  3.77it/s, loss=4.53, v_num=1]
Epoch 18:  95%|#########5| 21/22 [00:05<00:00,  3.73it/s, loss=4.53, v_num=1]
Epoch 18: 100%|##########| 22/22 [00:05<00:00,  3.83it/s, loss=4.53, v_num=1]
Epoch 18: 100%|##########| 22/22 [00:05<00:00,  3.80it/s, loss=4.53, v_num=1]
Epoch 18: 100%|##########| 22/22 [00:05<00:00,  3.79it/s, loss=4.53, v_num=1]
Epoch 18:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 19:   0%|          | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 19:   5%|4         | 1/22 [00:02<00:45,  2.18s/it, loss=4.53, v_num=1]
Epoch 19:   5%|4         | 1/22 [00:02<00:46,  2.24s/it, loss=4.53, v_num=1]
Epoch 19:   9%|9         | 2/22 [00:02<00:23,  1.18s/it, loss=4.53, v_num=1]
Epoch 19:   9%|9         | 2/22 [00:02<00:24,  1.21s/it, loss=4.53, v_num=1]
Epoch 19:  14%|#3        | 3/22 [00:02<00:16,  1.19it/s, loss=4.53, v_num=1]
Epoch 19:  14%|#3        | 3/22 [00:02<00:16,  1.16it/s, loss=4.53, v_num=1]
Epoch 19:  18%|#8        | 4/22 [00:02<00:12,  1.48it/s, loss=4.53, v_num=1]
Epoch 19:  18%|#8        | 4/22 [00:02<00:12,  1.45it/s, loss=4.53, v_num=1]
Epoch 19:  23%|##2       | 5/22 [00:02<00:09,  1.75it/s, loss=4.53, v_num=1]
Epoch 19:  23%|##2       | 5/22 [00:02<00:09,  1.71it/s, loss=4.54, v_num=1]
Epoch 19:  27%|##7       | 6/22 [00:03<00:08,  1.98it/s, loss=4.54, v_num=1]
Epoch 19:  27%|##7       | 6/22 [00:03<00:08,  1.94it/s, loss=4.54, v_num=1]
Epoch 19:  32%|###1      | 7/22 [00:03<00:06,  2.19it/s, loss=4.54, v_num=1]
Epoch 19:  32%|###1      | 7/22 [00:03<00:06,  2.15it/s, loss=4.54, v_num=1]
Epoch 19:  36%|###6      | 8/22 [00:03<00:06,  2.31it/s, loss=4.54, v_num=1]
Epoch 19:  36%|###6      | 8/22 [00:03<00:06,  2.27it/s, loss=4.54, v_num=1]
Epoch 19:  41%|####      | 9/22 [00:03<00:05,  2.47it/s, loss=4.54, v_num=1]
Epoch 19:  41%|####      | 9/22 [00:03<00:05,  2.43it/s, loss=4.54, v_num=1]
Epoch 19:  45%|####5     | 10/22 [00:03<00:04,  2.62it/s, loss=4.54, v_num=1]
Epoch 19:  45%|####5     | 10/22 [00:03<00:04,  2.58it/s, loss=4.54, v_num=1]
Epoch 19:  50%|#####     | 11/22 [00:03<00:03,  2.77it/s, loss=4.54, v_num=1]
Epoch 19:  50%|#####     | 11/22 [00:04<00:04,  2.73it/s, loss=4.53, v_num=1]
Epoch 19:  55%|#####4    | 12/22 [00:04<00:03,  2.90it/s, loss=4.53, v_num=1]
Epoch 19:  55%|#####4    | 12/22 [00:04<00:03,  2.85it/s, loss=4.53, v_num=1]
Epoch 19:  59%|#####9    | 13/22 [00:04<00:02,  3.02it/s, loss=4.53, v_num=1]
Epoch 19:  59%|#####9    | 13/22 [00:04<00:03,  2.97it/s, loss=4.53, v_num=1]
Epoch 19:  64%|######3   | 14/22 [00:04<00:02,  3.13it/s, loss=4.53, v_num=1]
Epoch 19:  64%|######3   | 14/22 [00:04<00:02,  3.08it/s, loss=4.53, v_num=1]
Epoch 19:  68%|######8   | 15/22 [00:04<00:02,  3.23it/s, loss=4.53, v_num=1]
Epoch 19:  68%|######8   | 15/22 [00:04<00:02,  3.19it/s, loss=4.53, v_num=1]
Epoch 19:  73%|#######2  | 16/22 [00:04<00:01,  3.32it/s, loss=4.53, v_num=1]
Epoch 19:  73%|#######2  | 16/22 [00:04<00:01,  3.28it/s, loss=4.53, v_num=1]
Epoch 19:  77%|#######7  | 17/22 [00:04<00:01,  3.42it/s, loss=4.53, v_num=1]
Epoch 19:  77%|#######7  | 17/22 [00:05<00:01,  3.38it/s, loss=4.53, v_num=1]
Epoch 19:  82%|########1 | 18/22 [00:05<00:01,  3.51it/s, loss=4.53, v_num=1]
Epoch 19:  82%|########1 | 18/22 [00:05<00:01,  3.46it/s, loss=4.53, v_num=1]
Epoch 19:  86%|########6 | 19/22 [00:05<00:00,  3.59it/s, loss=4.53, v_num=1]
Epoch 19:  86%|########6 | 19/22 [00:05<00:00,  3.55it/s, loss=4.54, v_num=1]
Epoch 19:  91%|######### | 20/22 [00:05<00:00,  3.67it/s, loss=4.54, v_num=1]
Epoch 19:  91%|######### | 20/22 [00:05<00:00,  3.63it/s, loss=4.54, v_num=1]
Epoch 19:  95%|#########5| 21/22 [00:05<00:00,  3.74it/s, loss=4.54, v_num=1]
Epoch 19:  95%|#########5| 21/22 [00:05<00:00,  3.70it/s, loss=4.54, v_num=1]
Epoch 19: 100%|##########| 22/22 [00:05<00:00,  3.81it/s, loss=4.54, v_num=1]
Epoch 19: 100%|##########| 22/22 [00:05<00:00,  3.77it/s, loss=4.53, v_num=1]
Epoch 19: 100%|##########| 22/22 [00:05<00:00,  3.77it/s, loss=4.53, v_num=1]
Epoch 19: 100%|##########| 22/22 [00:06<00:00,  3.64it/s, loss=4.53, v_num=1]

other example

plot_knn_examples(embeddings, filenames)
  • d=0.000, d=0.207, d=0.226
  • d=0.000, d=0.218, d=0.278
  • d=0.000, d=0.115, d=0.127
  • d=0.000, d=0.182, d=0.188
  • d=0.000, d=0.183, d=0.189
  • d=0.000, d=0.213, d=0.226

What’s next?

# You could use the pre-trained model and train a classifier on top.
pretrained_resnet_backbone = model.backbone

# you can also store the backbone and use it in another code
state_dict = {"resnet18_parameters": pretrained_resnet_backbone.state_dict()}
torch.save(state_dict, "model.pth")

THIS COULD BE IN A NEW FILE (e.g. inference.py)

Make sure you place the model.pth file in the same folder as this code

# load the model in a new file for inference
resnet18_new = torchvision.models.resnet18()

# note that we need to create exactly the same backbone in order to load the weights
backbone_new = nn.Sequential(*list(resnet18_new.children())[:-1])

ckpt = torch.load("model.pth")
backbone_new.load_state_dict(ckpt["resnet18_parameters"])
<All keys matched successfully>