Tutorial 3: Train SimCLR on Clothing

In this tutorial, we will train a SimCLR model using lightly. The model, augmentations and training procedure is from A Simple Framework for Contrastive Learning of Visual Representations.

The paper explores a rather simple training procedure for contrastive learning. Since we use the typical contrastive learning loss based on NCE the method greatly benefits from having larger batch sizes. In this example, we use a batch size of 256 and paired with the input resolution per image of 64x64 pixels and a resnet-18 model this example requires 16GB of GPU memory.

We use the clothing dataset from Alex Grigorev for this tutorial.

In this tutorial you will learn:

• How to create a SimCLR model

• How to generate image representations

• How different augmentations impact the learned representations

Imports

Import the Python frameworks we need for this tutorial.

import os

import matplotlib.pyplot as plt
import numpy as np
import pytorch_lightning as pl
import torch
import torch.nn as nn
import torchvision
from PIL import Image
from sklearn.neighbors import NearestNeighbors
from sklearn.preprocessing import normalize

from lightly.data import LightlyDataset
from lightly.transforms import SimCLRTransform, utils

Configuration

We set some configuration parameters for our experiment. Feel free to change them and analyze the effect.

The default configuration with a batch size of 256 and input resolution of 128 requires 6GB of GPU memory.

num_workers = 8
batch_size = 256
seed = 1
max_epochs = 20
input_size = 128
num_ftrs = 32

Let’s set the seed for our experiments

pl.seed_everything(seed)

1

Make sure path_to_data points to the downloaded clothing dataset. You can download it using git clone https://github.com/alexeygrigorev/clothing-dataset.git

path_to_data = "/datasets/clothing-dataset/images"

Setup data augmentations and loaders

The images from the dataset have been taken from above when the clothing was on a table, bed or floor. Therefore, we can make use of additional augmentations such as vertical flip or random rotation (90 degrees). By adding these augmentations we learn our model invariance regarding the orientation of the clothing piece. E.g. we don’t care if a shirt is upside down but more about the structure which make it a shirt.

You can learn more about the different augmentations and learned invariances here: Advanced Concepts in Self-Supervised Learning.

transform = SimCLRTransform(input_size=input_size, vf_prob=0.5, rr_prob=0.5)

# We create a torchvision transformation for embedding the dataset after
# training
test_transform = torchvision.transforms.Compose(
    [
        torchvision.transforms.Resize((input_size, input_size)),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize(
            mean=utils.IMAGENET_NORMALIZE["mean"],
            std=utils.IMAGENET_NORMALIZE["std"],
        ),
    ]
)

dataset_train_simclr = LightlyDataset(input_dir=path_to_data, transform=transform)

dataset_test = LightlyDataset(input_dir=path_to_data, transform=test_transform)

dataloader_train_simclr = torch.utils.data.DataLoader(
    dataset_train_simclr,
    batch_size=batch_size,
    shuffle=True,
    drop_last=True,
    num_workers=num_workers,
)

dataloader_test = torch.utils.data.DataLoader(
    dataset_test,
    batch_size=batch_size,
    shuffle=False,
    drop_last=False,
    num_workers=num_workers,
)

Create the SimCLR Model

Now we create the SimCLR model. We implement it as a PyTorch Lightning Module and use a ResNet-18 backbone from Torchvision. Lightly provides implementations of the SimCLR projection head and loss function in the SimCLRProjectionHead and NTXentLoss classes. We can simply import them and combine the building blocks in the module.

from lightly.loss import NTXentLoss
from lightly.models.modules.heads import SimCLRProjectionHead


class SimCLRModel(pl.LightningModule):
    def __init__(self):
        super().__init__()

        # create a ResNet backbone and remove the classification head
        resnet = torchvision.models.resnet18()
        self.backbone = nn.Sequential(*list(resnet.children())[:-1])

        hidden_dim = resnet.fc.in_features
        self.projection_head = SimCLRProjectionHead(hidden_dim, hidden_dim, 128)

        self.criterion = NTXentLoss()

    def forward(self, x):
        h = self.backbone(x).flatten(start_dim=1)
        z = self.projection_head(h)
        return z

    def training_step(self, batch, batch_idx):
        (x0, x1), _, _ = batch
        z0 = self.forward(x0)
        z1 = self.forward(x1)
        loss = self.criterion(z0, z1)
        self.log("train_loss_ssl", loss)
        return loss

    def configure_optimizers(self):
        optim = torch.optim.SGD(
            self.parameters(), lr=6e-2, momentum=0.9, weight_decay=5e-4
        )
        scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim, max_epochs)
        return [optim], [scheduler]

Train the module using the PyTorch Lightning Trainer on a single GPU.

model = SimCLRModel()
trainer = pl.Trainer(max_epochs=max_epochs, devices=1, accelerator="gpu")
trainer.fit(model, dataloader_train_simclr)

/datasets/actions-runner/core_gpu_runner_01/_work/lightly-core/lightly-core/.venv/lib/python3.12/site-packages/lightning_fabric/plugins/environments/slurm.py:204: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python /datasets/actions-runner/core_gpu_runner_01/_work/li ...
/datasets/actions-runner/core_gpu_runner_01/_work/lightly-core/lightly-core/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:283: UserWarning:
    Found GPU1 NVIDIA GeForce GTX 1080 Ti which is of cuda capability 6.1.
    Minimum and Maximum cuda capability supported by this version of PyTorch is
    (7.0) - (12.0)

  warnings.warn(
/datasets/actions-runner/core_gpu_runner_01/_work/lightly-core/lightly-core/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:304: UserWarning:
    Please install PyTorch with a following CUDA
    configurations:  12.6 following instructions at
    https://pytorch.org/get-started/locally/

  warnings.warn(matched_cuda_warn.format(matched_arches))
/datasets/actions-runner/core_gpu_runner_01/_work/lightly-core/lightly-core/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:326: UserWarning:
NVIDIA GeForce GTX 1080 Ti with CUDA capability sm_61 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_70 sm_75 sm_80 sm_86 sm_90 sm_100 sm_120.
If you want to use the NVIDIA GeForce GTX 1080 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(
/datasets/actions-runner/core_gpu_runner_01/_work/lightly-core/lightly-core/.venv/lib/python3.12/site-packages/pytorch_lightning/loops/fit_loop.py:310: The number of training batches (22) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.

Training: |          | 0/? [00:00<?, ?it/s]
Training:   0%|          | 0/22 [00:00<?, ?it/s]
Epoch 0:   0%|          | 0/22 [00:00<?, ?it/s]
Epoch 0:   5%|▍         | 1/22 [00:02<00:47,  0.45it/s]
Epoch 0:   5%|▍         | 1/22 [00:02<00:47,  0.45it/s, v_num=0]
Epoch 0:   9%|▉         | 2/22 [00:02<00:23,  0.84it/s, v_num=0]
Epoch 0:   9%|▉         | 2/22 [00:02<00:23,  0.84it/s, v_num=0]
Epoch 0:  14%|█▎        | 3/22 [00:02<00:15,  1.21it/s, v_num=0]
Epoch 0:  14%|█▎        | 3/22 [00:02<00:15,  1.21it/s, v_num=0]
Epoch 0:  18%|█▊        | 4/22 [00:02<00:11,  1.53it/s, v_num=0]
Epoch 0:  18%|█▊        | 4/22 [00:02<00:11,  1.53it/s, v_num=0]
Epoch 0:  23%|██▎       | 5/22 [00:02<00:09,  1.81it/s, v_num=0]
Epoch 0:  23%|██▎       | 5/22 [00:02<00:09,  1.81it/s, v_num=0]
Epoch 0:  27%|██▋       | 6/22 [00:02<00:07,  2.06it/s, v_num=0]
Epoch 0:  27%|██▋       | 6/22 [00:02<00:07,  2.06it/s, v_num=0]
Epoch 0:  32%|███▏      | 7/22 [00:03<00:06,  2.28it/s, v_num=0]
Epoch 0:  32%|███▏      | 7/22 [00:03<00:06,  2.28it/s, v_num=0]
Epoch 0:  36%|███▋      | 8/22 [00:03<00:05,  2.49it/s, v_num=0]
Epoch 0:  36%|███▋      | 8/22 [00:03<00:05,  2.49it/s, v_num=0]
Epoch 0:  41%|████      | 9/22 [00:03<00:04,  2.67it/s, v_num=0]
Epoch 0:  41%|████      | 9/22 [00:03<00:04,  2.67it/s, v_num=0]
Epoch 0:  45%|████▌     | 10/22 [00:03<00:04,  2.85it/s, v_num=0]
Epoch 0:  45%|████▌     | 10/22 [00:03<00:04,  2.85it/s, v_num=0]
Epoch 0:  50%|█████     | 11/22 [00:03<00:03,  3.01it/s, v_num=0]
Epoch 0:  50%|█████     | 11/22 [00:03<00:03,  3.01it/s, v_num=0]
Epoch 0:  55%|█████▍    | 12/22 [00:03<00:03,  3.16it/s, v_num=0]
Epoch 0:  55%|█████▍    | 12/22 [00:03<00:03,  3.16it/s, v_num=0]
Epoch 0:  59%|█████▉    | 13/22 [00:03<00:02,  3.29it/s, v_num=0]
Epoch 0:  59%|█████▉    | 13/22 [00:03<00:02,  3.29it/s, v_num=0]
Epoch 0:  64%|██████▎   | 14/22 [00:04<00:02,  3.42it/s, v_num=0]
Epoch 0:  64%|██████▎   | 14/22 [00:04<00:02,  3.42it/s, v_num=0]
Epoch 0:  68%|██████▊   | 15/22 [00:04<00:01,  3.54it/s, v_num=0]
Epoch 0:  68%|██████▊   | 15/22 [00:04<00:01,  3.54it/s, v_num=0]
Epoch 0:  73%|███████▎  | 16/22 [00:04<00:01,  3.65it/s, v_num=0]
Epoch 0:  73%|███████▎  | 16/22 [00:04<00:01,  3.65it/s, v_num=0]
Epoch 0:  77%|███████▋  | 17/22 [00:04<00:01,  3.71it/s, v_num=0]
Epoch 0:  77%|███████▋  | 17/22 [00:04<00:01,  3.71it/s, v_num=0]
Epoch 0:  82%|████████▏ | 18/22 [00:04<00:01,  3.80it/s, v_num=0]
Epoch 0:  82%|████████▏ | 18/22 [00:04<00:01,  3.80it/s, v_num=0]
Epoch 0:  86%|████████▋ | 19/22 [00:04<00:00,  3.90it/s, v_num=0]
Epoch 0:  86%|████████▋ | 19/22 [00:04<00:00,  3.90it/s, v_num=0]
Epoch 0:  91%|█████████ | 20/22 [00:05<00:00,  3.98it/s, v_num=0]
Epoch 0:  91%|█████████ | 20/22 [00:05<00:00,  3.98it/s, v_num=0]
Epoch 0:  95%|█████████▌| 21/22 [00:05<00:00,  4.06it/s, v_num=0]
Epoch 0:  95%|█████████▌| 21/22 [00:05<00:00,  4.06it/s, v_num=0]
Epoch 0: 100%|██████████| 22/22 [00:05<00:00,  4.14it/s, v_num=0]
Epoch 0: 100%|██████████| 22/22 [00:05<00:00,  4.14it/s, v_num=0]
Epoch 0: 100%|██████████| 22/22 [00:05<00:00,  4.14it/s, v_num=0]
Epoch 0:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 1:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 1:   5%|▍         | 1/22 [00:01<00:38,  0.55it/s, v_num=0]
Epoch 1:   5%|▍         | 1/22 [00:01<00:38,  0.55it/s, v_num=0]
Epoch 1:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 1:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 1:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, v_num=0]
Epoch 1:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, v_num=0]
Epoch 1:  18%|█▊        | 4/22 [00:02<00:10,  1.78it/s, v_num=0]
Epoch 1:  18%|█▊        | 4/22 [00:02<00:10,  1.78it/s, v_num=0]
Epoch 1:  23%|██▎       | 5/22 [00:02<00:08,  2.09it/s, v_num=0]
Epoch 1:  23%|██▎       | 5/22 [00:02<00:08,  2.09it/s, v_num=0]
Epoch 1:  27%|██▋       | 6/22 [00:02<00:06,  2.36it/s, v_num=0]
Epoch 1:  27%|██▋       | 6/22 [00:02<00:06,  2.36it/s, v_num=0]
Epoch 1:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, v_num=0]
Epoch 1:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, v_num=0]
Epoch 1:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, v_num=0]
Epoch 1:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, v_num=0]
Epoch 1:  41%|████      | 9/22 [00:03<00:04,  2.83it/s, v_num=0]
Epoch 1:  41%|████      | 9/22 [00:03<00:04,  2.83it/s, v_num=0]
Epoch 1:  45%|████▌     | 10/22 [00:03<00:03,  3.01it/s, v_num=0]
Epoch 1:  45%|████▌     | 10/22 [00:03<00:03,  3.01it/s, v_num=0]
Epoch 1:  50%|█████     | 11/22 [00:03<00:03,  3.17it/s, v_num=0]
Epoch 1:  50%|█████     | 11/22 [00:03<00:03,  3.17it/s, v_num=0]
Epoch 1:  55%|█████▍    | 12/22 [00:03<00:03,  3.32it/s, v_num=0]
Epoch 1:  55%|█████▍    | 12/22 [00:03<00:03,  3.32it/s, v_num=0]
Epoch 1:  59%|█████▉    | 13/22 [00:03<00:02,  3.46it/s, v_num=0]
Epoch 1:  59%|█████▉    | 13/22 [00:03<00:02,  3.46it/s, v_num=0]
Epoch 1:  64%|██████▎   | 14/22 [00:03<00:02,  3.59it/s, v_num=0]
Epoch 1:  64%|██████▎   | 14/22 [00:03<00:02,  3.59it/s, v_num=0]
Epoch 1:  68%|██████▊   | 15/22 [00:04<00:01,  3.71it/s, v_num=0]
Epoch 1:  68%|██████▊   | 15/22 [00:04<00:01,  3.71it/s, v_num=0]
Epoch 1:  73%|███████▎  | 16/22 [00:04<00:01,  3.82it/s, v_num=0]
Epoch 1:  73%|███████▎  | 16/22 [00:04<00:01,  3.82it/s, v_num=0]
Epoch 1:  77%|███████▋  | 17/22 [00:04<00:01,  3.76it/s, v_num=0]
Epoch 1:  77%|███████▋  | 17/22 [00:04<00:01,  3.76it/s, v_num=0]
Epoch 1:  82%|████████▏ | 18/22 [00:04<00:01,  3.86it/s, v_num=0]
Epoch 1:  82%|████████▏ | 18/22 [00:04<00:01,  3.86it/s, v_num=0]
Epoch 1:  86%|████████▋ | 19/22 [00:04<00:00,  3.95it/s, v_num=0]
Epoch 1:  86%|████████▋ | 19/22 [00:04<00:00,  3.95it/s, v_num=0]
Epoch 1:  91%|█████████ | 20/22 [00:04<00:00,  4.04it/s, v_num=0]
Epoch 1:  91%|█████████ | 20/22 [00:04<00:00,  4.04it/s, v_num=0]
Epoch 1:  95%|█████████▌| 21/22 [00:05<00:00,  4.12it/s, v_num=0]
Epoch 1:  95%|█████████▌| 21/22 [00:05<00:00,  4.12it/s, v_num=0]
Epoch 1: 100%|██████████| 22/22 [00:05<00:00,  4.20it/s, v_num=0]
Epoch 1: 100%|██████████| 22/22 [00:05<00:00,  4.20it/s, v_num=0]
Epoch 1: 100%|██████████| 22/22 [00:05<00:00,  4.20it/s, v_num=0]
Epoch 1:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 2:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 2:   5%|▍         | 1/22 [00:01<00:38,  0.55it/s, v_num=0]
Epoch 2:   5%|▍         | 1/22 [00:01<00:38,  0.55it/s, v_num=0]
Epoch 2:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 2:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 2:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, v_num=0]
Epoch 2:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, v_num=0]
Epoch 2:  18%|█▊        | 4/22 [00:02<00:10,  1.78it/s, v_num=0]
Epoch 2:  18%|█▊        | 4/22 [00:02<00:10,  1.78it/s, v_num=0]
Epoch 2:  23%|██▎       | 5/22 [00:02<00:08,  2.09it/s, v_num=0]
Epoch 2:  23%|██▎       | 5/22 [00:02<00:08,  2.09it/s, v_num=0]
Epoch 2:  27%|██▋       | 6/22 [00:02<00:06,  2.37it/s, v_num=0]
Epoch 2:  27%|██▋       | 6/22 [00:02<00:06,  2.37it/s, v_num=0]
Epoch 2:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, v_num=0]
Epoch 2:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, v_num=0]
Epoch 2:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, v_num=0]
Epoch 2:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, v_num=0]
Epoch 2:  41%|████      | 9/22 [00:03<00:04,  2.88it/s, v_num=0]
Epoch 2:  41%|████      | 9/22 [00:03<00:04,  2.88it/s, v_num=0]
Epoch 2:  45%|████▌     | 10/22 [00:03<00:03,  3.06it/s, v_num=0]
Epoch 2:  45%|████▌     | 10/22 [00:03<00:03,  3.06it/s, v_num=0]
Epoch 2:  50%|█████     | 11/22 [00:03<00:03,  3.22it/s, v_num=0]
Epoch 2:  50%|█████     | 11/22 [00:03<00:03,  3.22it/s, v_num=0]
Epoch 2:  55%|█████▍    | 12/22 [00:03<00:02,  3.37it/s, v_num=0]
Epoch 2:  55%|█████▍    | 12/22 [00:03<00:02,  3.37it/s, v_num=0]
Epoch 2:  59%|█████▉    | 13/22 [00:03<00:02,  3.51it/s, v_num=0]
Epoch 2:  59%|█████▉    | 13/22 [00:03<00:02,  3.51it/s, v_num=0]
Epoch 2:  64%|██████▎   | 14/22 [00:03<00:02,  3.64it/s, v_num=0]
Epoch 2:  64%|██████▎   | 14/22 [00:03<00:02,  3.64it/s, v_num=0]
Epoch 2:  68%|██████▊   | 15/22 [00:03<00:01,  3.75it/s, v_num=0]
Epoch 2:  68%|██████▊   | 15/22 [00:03<00:01,  3.75it/s, v_num=0]
Epoch 2:  73%|███████▎  | 16/22 [00:04<00:01,  3.86it/s, v_num=0]
Epoch 2:  73%|███████▎  | 16/22 [00:04<00:01,  3.86it/s, v_num=0]
Epoch 2:  77%|███████▋  | 17/22 [00:04<00:01,  3.85it/s, v_num=0]
Epoch 2:  77%|███████▋  | 17/22 [00:04<00:01,  3.85it/s, v_num=0]
Epoch 2:  82%|████████▏ | 18/22 [00:04<00:01,  3.95it/s, v_num=0]
Epoch 2:  82%|████████▏ | 18/22 [00:04<00:01,  3.95it/s, v_num=0]
Epoch 2:  86%|████████▋ | 19/22 [00:04<00:00,  4.04it/s, v_num=0]
Epoch 2:  86%|████████▋ | 19/22 [00:04<00:00,  4.04it/s, v_num=0]
Epoch 2:  91%|█████████ | 20/22 [00:04<00:00,  4.12it/s, v_num=0]
Epoch 2:  91%|█████████ | 20/22 [00:04<00:00,  4.12it/s, v_num=0]
Epoch 2:  95%|█████████▌| 21/22 [00:04<00:00,  4.20it/s, v_num=0]
Epoch 2:  95%|█████████▌| 21/22 [00:04<00:00,  4.20it/s, v_num=0]
Epoch 2: 100%|██████████| 22/22 [00:05<00:00,  4.28it/s, v_num=0]
Epoch 2: 100%|██████████| 22/22 [00:05<00:00,  4.28it/s, v_num=0]
Epoch 2: 100%|██████████| 22/22 [00:05<00:00,  4.28it/s, v_num=0]
Epoch 2:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 3:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 3:   5%|▍         | 1/22 [00:01<00:38,  0.55it/s, v_num=0]
Epoch 3:   5%|▍         | 1/22 [00:01<00:38,  0.55it/s, v_num=0]
Epoch 3:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 3:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 3:  14%|█▎        | 3/22 [00:02<00:13,  1.42it/s, v_num=0]
Epoch 3:  14%|█▎        | 3/22 [00:02<00:13,  1.42it/s, v_num=0]
Epoch 3:  18%|█▊        | 4/22 [00:02<00:10,  1.77it/s, v_num=0]
Epoch 3:  18%|█▊        | 4/22 [00:02<00:10,  1.77it/s, v_num=0]
Epoch 3:  23%|██▎       | 5/22 [00:02<00:08,  2.08it/s, v_num=0]
Epoch 3:  23%|██▎       | 5/22 [00:02<00:08,  2.08it/s, v_num=0]
Epoch 3:  27%|██▋       | 6/22 [00:02<00:06,  2.36it/s, v_num=0]
Epoch 3:  27%|██▋       | 6/22 [00:02<00:06,  2.36it/s, v_num=0]
Epoch 3:  32%|███▏      | 7/22 [00:02<00:05,  2.60it/s, v_num=0]
Epoch 3:  32%|███▏      | 7/22 [00:02<00:05,  2.60it/s, v_num=0]
Epoch 3:  36%|███▋      | 8/22 [00:02<00:04,  2.82it/s, v_num=0]
Epoch 3:  36%|███▋      | 8/22 [00:02<00:04,  2.82it/s, v_num=0]
Epoch 3:  41%|████      | 9/22 [00:03<00:04,  2.86it/s, v_num=0]
Epoch 3:  41%|████      | 9/22 [00:03<00:04,  2.86it/s, v_num=0]
Epoch 3:  45%|████▌     | 10/22 [00:03<00:03,  3.03it/s, v_num=0]
Epoch 3:  45%|████▌     | 10/22 [00:03<00:03,  3.03it/s, v_num=0]
Epoch 3:  50%|█████     | 11/22 [00:03<00:03,  3.20it/s, v_num=0]
Epoch 3:  50%|█████     | 11/22 [00:03<00:03,  3.20it/s, v_num=0]
Epoch 3:  55%|█████▍    | 12/22 [00:03<00:02,  3.34it/s, v_num=0]
Epoch 3:  55%|█████▍    | 12/22 [00:03<00:02,  3.34it/s, v_num=0]
Epoch 3:  59%|█████▉    | 13/22 [00:03<00:02,  3.48it/s, v_num=0]
Epoch 3:  59%|█████▉    | 13/22 [00:03<00:02,  3.48it/s, v_num=0]
Epoch 3:  64%|██████▎   | 14/22 [00:03<00:02,  3.61it/s, v_num=0]
Epoch 3:  64%|██████▎   | 14/22 [00:03<00:02,  3.61it/s, v_num=0]
Epoch 3:  68%|██████▊   | 15/22 [00:04<00:01,  3.73it/s, v_num=0]
Epoch 3:  68%|██████▊   | 15/22 [00:04<00:01,  3.73it/s, v_num=0]
Epoch 3:  73%|███████▎  | 16/22 [00:04<00:01,  3.84it/s, v_num=0]
Epoch 3:  73%|███████▎  | 16/22 [00:04<00:01,  3.84it/s, v_num=0]
Epoch 3:  77%|███████▋  | 17/22 [00:04<00:01,  3.79it/s, v_num=0]
Epoch 3:  77%|███████▋  | 17/22 [00:04<00:01,  3.79it/s, v_num=0]
Epoch 3:  82%|████████▏ | 18/22 [00:04<00:01,  3.89it/s, v_num=0]
Epoch 3:  82%|████████▏ | 18/22 [00:04<00:01,  3.89it/s, v_num=0]
Epoch 3:  86%|████████▋ | 19/22 [00:04<00:00,  3.98it/s, v_num=0]
Epoch 3:  86%|████████▋ | 19/22 [00:04<00:00,  3.98it/s, v_num=0]
Epoch 3:  91%|█████████ | 20/22 [00:04<00:00,  4.06it/s, v_num=0]
Epoch 3:  91%|█████████ | 20/22 [00:04<00:00,  4.06it/s, v_num=0]
Epoch 3:  95%|█████████▌| 21/22 [00:05<00:00,  4.15it/s, v_num=0]
Epoch 3:  95%|█████████▌| 21/22 [00:05<00:00,  4.15it/s, v_num=0]
Epoch 3: 100%|██████████| 22/22 [00:05<00:00,  4.22it/s, v_num=0]
Epoch 3: 100%|██████████| 22/22 [00:05<00:00,  4.22it/s, v_num=0]
Epoch 3: 100%|██████████| 22/22 [00:05<00:00,  4.22it/s, v_num=0]
Epoch 3:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 4:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 4:   5%|▍         | 1/22 [00:01<00:37,  0.55it/s, v_num=0]
Epoch 4:   5%|▍         | 1/22 [00:01<00:37,  0.55it/s, v_num=0]
Epoch 4:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 4:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 4:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, v_num=0]
Epoch 4:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, v_num=0]
Epoch 4:  18%|█▊        | 4/22 [00:02<00:10,  1.78it/s, v_num=0]
Epoch 4:  18%|█▊        | 4/22 [00:02<00:10,  1.78it/s, v_num=0]
Epoch 4:  23%|██▎       | 5/22 [00:02<00:08,  2.09it/s, v_num=0]
Epoch 4:  23%|██▎       | 5/22 [00:02<00:08,  2.09it/s, v_num=0]
Epoch 4:  27%|██▋       | 6/22 [00:02<00:06,  2.36it/s, v_num=0]
Epoch 4:  27%|██▋       | 6/22 [00:02<00:06,  2.36it/s, v_num=0]
Epoch 4:  32%|███▏      | 7/22 [00:02<00:05,  2.60it/s, v_num=0]
Epoch 4:  32%|███▏      | 7/22 [00:02<00:05,  2.60it/s, v_num=0]
Epoch 4:  36%|███▋      | 8/22 [00:02<00:04,  2.82it/s, v_num=0]
Epoch 4:  36%|███▋      | 8/22 [00:02<00:04,  2.82it/s, v_num=0]
Epoch 4:  41%|████      | 9/22 [00:03<00:04,  2.88it/s, v_num=0]
Epoch 4:  41%|████      | 9/22 [00:03<00:04,  2.88it/s, v_num=0]
Epoch 4:  45%|████▌     | 10/22 [00:03<00:03,  3.05it/s, v_num=0]
Epoch 4:  45%|████▌     | 10/22 [00:03<00:03,  3.05it/s, v_num=0]
Epoch 4:  50%|█████     | 11/22 [00:03<00:03,  3.22it/s, v_num=0]
Epoch 4:  50%|█████     | 11/22 [00:03<00:03,  3.22it/s, v_num=0]
Epoch 4:  55%|█████▍    | 12/22 [00:03<00:02,  3.37it/s, v_num=0]
Epoch 4:  55%|█████▍    | 12/22 [00:03<00:02,  3.37it/s, v_num=0]
Epoch 4:  59%|█████▉    | 13/22 [00:03<00:02,  3.50it/s, v_num=0]
Epoch 4:  59%|█████▉    | 13/22 [00:03<00:02,  3.50it/s, v_num=0]
Epoch 4:  64%|██████▎   | 14/22 [00:03<00:02,  3.63it/s, v_num=0]
Epoch 4:  64%|██████▎   | 14/22 [00:03<00:02,  3.63it/s, v_num=0]
Epoch 4:  68%|██████▊   | 15/22 [00:03<00:01,  3.75it/s, v_num=0]
Epoch 4:  68%|██████▊   | 15/22 [00:03<00:01,  3.75it/s, v_num=0]
Epoch 4:  73%|███████▎  | 16/22 [00:04<00:01,  3.86it/s, v_num=0]
Epoch 4:  73%|███████▎  | 16/22 [00:04<00:01,  3.86it/s, v_num=0]
Epoch 4:  77%|███████▋  | 17/22 [00:04<00:01,  3.87it/s, v_num=0]
Epoch 4:  77%|███████▋  | 17/22 [00:04<00:01,  3.87it/s, v_num=0]
Epoch 4:  82%|████████▏ | 18/22 [00:04<00:01,  3.97it/s, v_num=0]
Epoch 4:  82%|████████▏ | 18/22 [00:04<00:01,  3.97it/s, v_num=0]
Epoch 4:  86%|████████▋ | 19/22 [00:04<00:00,  4.06it/s, v_num=0]
Epoch 4:  86%|████████▋ | 19/22 [00:04<00:00,  4.06it/s, v_num=0]
Epoch 4:  91%|█████████ | 20/22 [00:04<00:00,  4.15it/s, v_num=0]
Epoch 4:  91%|█████████ | 20/22 [00:04<00:00,  4.15it/s, v_num=0]
Epoch 4:  95%|█████████▌| 21/22 [00:04<00:00,  4.23it/s, v_num=0]
Epoch 4:  95%|█████████▌| 21/22 [00:04<00:00,  4.23it/s, v_num=0]
Epoch 4: 100%|██████████| 22/22 [00:05<00:00,  4.30it/s, v_num=0]
Epoch 4: 100%|██████████| 22/22 [00:05<00:00,  4.30it/s, v_num=0]
Epoch 4: 100%|██████████| 22/22 [00:05<00:00,  4.30it/s, v_num=0]
Epoch 4:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 5:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 5:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 5:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 5:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 5:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 5:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, v_num=0]
Epoch 5:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, v_num=0]
Epoch 5:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, v_num=0]
Epoch 5:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, v_num=0]
Epoch 5:  23%|██▎       | 5/22 [00:02<00:08,  2.11it/s, v_num=0]
Epoch 5:  23%|██▎       | 5/22 [00:02<00:08,  2.10it/s, v_num=0]
Epoch 5:  27%|██▋       | 6/22 [00:02<00:06,  2.38it/s, v_num=0]
Epoch 5:  27%|██▋       | 6/22 [00:02<00:06,  2.38it/s, v_num=0]
Epoch 5:  32%|███▏      | 7/22 [00:02<00:05,  2.62it/s, v_num=0]
Epoch 5:  32%|███▏      | 7/22 [00:02<00:05,  2.62it/s, v_num=0]
Epoch 5:  36%|███▋      | 8/22 [00:02<00:04,  2.84it/s, v_num=0]
Epoch 5:  36%|███▋      | 8/22 [00:02<00:04,  2.84it/s, v_num=0]
Epoch 5:  41%|████      | 9/22 [00:03<00:04,  2.89it/s, v_num=0]
Epoch 5:  41%|████      | 9/22 [00:03<00:04,  2.89it/s, v_num=0]
Epoch 5:  45%|████▌     | 10/22 [00:03<00:03,  3.07it/s, v_num=0]
Epoch 5:  45%|████▌     | 10/22 [00:03<00:03,  3.07it/s, v_num=0]
Epoch 5:  50%|█████     | 11/22 [00:03<00:03,  3.23it/s, v_num=0]
Epoch 5:  50%|█████     | 11/22 [00:03<00:03,  3.23it/s, v_num=0]
Epoch 5:  55%|█████▍    | 12/22 [00:03<00:02,  3.38it/s, v_num=0]
Epoch 5:  55%|█████▍    | 12/22 [00:03<00:02,  3.38it/s, v_num=0]
Epoch 5:  59%|█████▉    | 13/22 [00:03<00:02,  3.52it/s, v_num=0]
Epoch 5:  59%|█████▉    | 13/22 [00:03<00:02,  3.52it/s, v_num=0]
Epoch 5:  64%|██████▎   | 14/22 [00:03<00:02,  3.64it/s, v_num=0]
Epoch 5:  64%|██████▎   | 14/22 [00:03<00:02,  3.64it/s, v_num=0]
Epoch 5:  68%|██████▊   | 15/22 [00:03<00:01,  3.76it/s, v_num=0]
Epoch 5:  68%|██████▊   | 15/22 [00:03<00:01,  3.76it/s, v_num=0]
Epoch 5:  73%|███████▎  | 16/22 [00:04<00:01,  3.87it/s, v_num=0]
Epoch 5:  73%|███████▎  | 16/22 [00:04<00:01,  3.87it/s, v_num=0]
Epoch 5:  77%|███████▋  | 17/22 [00:04<00:01,  3.87it/s, v_num=0]
Epoch 5:  77%|███████▋  | 17/22 [00:04<00:01,  3.87it/s, v_num=0]
Epoch 5:  82%|████████▏ | 18/22 [00:04<00:01,  3.97it/s, v_num=0]
Epoch 5:  82%|████████▏ | 18/22 [00:04<00:01,  3.97it/s, v_num=0]
Epoch 5:  86%|████████▋ | 19/22 [00:04<00:00,  4.06it/s, v_num=0]
Epoch 5:  86%|████████▋ | 19/22 [00:04<00:00,  4.06it/s, v_num=0]
Epoch 5:  91%|█████████ | 20/22 [00:04<00:00,  4.14it/s, v_num=0]
Epoch 5:  91%|█████████ | 20/22 [00:04<00:00,  4.14it/s, v_num=0]
Epoch 5:  95%|█████████▌| 21/22 [00:04<00:00,  4.22it/s, v_num=0]
Epoch 5:  95%|█████████▌| 21/22 [00:04<00:00,  4.22it/s, v_num=0]
Epoch 5: 100%|██████████| 22/22 [00:05<00:00,  4.29it/s, v_num=0]
Epoch 5: 100%|██████████| 22/22 [00:05<00:00,  4.29it/s, v_num=0]
Epoch 5: 100%|██████████| 22/22 [00:05<00:00,  4.29it/s, v_num=0]
Epoch 5:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 6:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 6:   5%|▍         | 1/22 [00:01<00:37,  0.57it/s, v_num=0]
Epoch 6:   5%|▍         | 1/22 [00:01<00:37,  0.57it/s, v_num=0]
Epoch 6:   9%|▉         | 2/22 [00:01<00:19,  1.05it/s, v_num=0]
Epoch 6:   9%|▉         | 2/22 [00:01<00:19,  1.05it/s, v_num=0]
Epoch 6:  14%|█▎        | 3/22 [00:02<00:13,  1.46it/s, v_num=0]
Epoch 6:  14%|█▎        | 3/22 [00:02<00:13,  1.46it/s, v_num=0]
Epoch 6:  18%|█▊        | 4/22 [00:02<00:09,  1.81it/s, v_num=0]
Epoch 6:  18%|█▊        | 4/22 [00:02<00:09,  1.81it/s, v_num=0]
Epoch 6:  23%|██▎       | 5/22 [00:02<00:07,  2.13it/s, v_num=0]
Epoch 6:  23%|██▎       | 5/22 [00:02<00:07,  2.13it/s, v_num=0]
Epoch 6:  27%|██▋       | 6/22 [00:02<00:06,  2.40it/s, v_num=0]
Epoch 6:  27%|██▋       | 6/22 [00:02<00:06,  2.40it/s, v_num=0]
Epoch 6:  32%|███▏      | 7/22 [00:02<00:05,  2.65it/s, v_num=0]
Epoch 6:  32%|███▏      | 7/22 [00:02<00:05,  2.65it/s, v_num=0]
Epoch 6:  36%|███▋      | 8/22 [00:02<00:04,  2.86it/s, v_num=0]
Epoch 6:  36%|███▋      | 8/22 [00:02<00:04,  2.86it/s, v_num=0]
Epoch 6:  41%|████      | 9/22 [00:03<00:04,  2.89it/s, v_num=0]
Epoch 6:  41%|████      | 9/22 [00:03<00:04,  2.89it/s, v_num=0]
Epoch 6:  45%|████▌     | 10/22 [00:03<00:03,  3.07it/s, v_num=0]
Epoch 6:  45%|████▌     | 10/22 [00:03<00:03,  3.07it/s, v_num=0]
Epoch 6:  50%|█████     | 11/22 [00:03<00:03,  3.23it/s, v_num=0]
Epoch 6:  50%|█████     | 11/22 [00:03<00:03,  3.23it/s, v_num=0]
Epoch 6:  55%|█████▍    | 12/22 [00:03<00:02,  3.37it/s, v_num=0]
Epoch 6:  55%|█████▍    | 12/22 [00:03<00:02,  3.37it/s, v_num=0]
Epoch 6:  59%|█████▉    | 13/22 [00:03<00:02,  3.50it/s, v_num=0]
Epoch 6:  59%|█████▉    | 13/22 [00:03<00:02,  3.50it/s, v_num=0]
Epoch 6:  64%|██████▎   | 14/22 [00:03<00:02,  3.63it/s, v_num=0]
Epoch 6:  64%|██████▎   | 14/22 [00:03<00:02,  3.63it/s, v_num=0]
Epoch 6:  68%|██████▊   | 15/22 [00:04<00:01,  3.75it/s, v_num=0]
Epoch 6:  68%|██████▊   | 15/22 [00:04<00:01,  3.75it/s, v_num=0]
Epoch 6:  73%|███████▎  | 16/22 [00:04<00:01,  3.86it/s, v_num=0]
Epoch 6:  73%|███████▎  | 16/22 [00:04<00:01,  3.86it/s, v_num=0]
Epoch 6:  77%|███████▋  | 17/22 [00:04<00:01,  3.84it/s, v_num=0]
Epoch 6:  77%|███████▋  | 17/22 [00:04<00:01,  3.84it/s, v_num=0]
Epoch 6:  82%|████████▏ | 18/22 [00:04<00:01,  3.93it/s, v_num=0]
Epoch 6:  82%|████████▏ | 18/22 [00:04<00:01,  3.93it/s, v_num=0]
Epoch 6:  86%|████████▋ | 19/22 [00:04<00:00,  4.02it/s, v_num=0]
Epoch 6:  86%|████████▋ | 19/22 [00:04<00:00,  4.02it/s, v_num=0]
Epoch 6:  91%|█████████ | 20/22 [00:04<00:00,  4.10it/s, v_num=0]
Epoch 6:  91%|█████████ | 20/22 [00:04<00:00,  4.10it/s, v_num=0]
Epoch 6:  95%|█████████▌| 21/22 [00:05<00:00,  4.18it/s, v_num=0]
Epoch 6:  95%|█████████▌| 21/22 [00:05<00:00,  4.18it/s, v_num=0]
Epoch 6: 100%|██████████| 22/22 [00:05<00:00,  4.25it/s, v_num=0]
Epoch 6: 100%|██████████| 22/22 [00:05<00:00,  4.25it/s, v_num=0]
Epoch 6: 100%|██████████| 22/22 [00:05<00:00,  4.25it/s, v_num=0]
Epoch 6:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 7:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 7:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 7:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 7:   9%|▉         | 2/22 [00:01<00:19,  1.04it/s, v_num=0]
Epoch 7:   9%|▉         | 2/22 [00:01<00:19,  1.04it/s, v_num=0]
Epoch 7:  14%|█▎        | 3/22 [00:02<00:13,  1.45it/s, v_num=0]
Epoch 7:  14%|█▎        | 3/22 [00:02<00:13,  1.45it/s, v_num=0]
Epoch 7:  18%|█▊        | 4/22 [00:02<00:09,  1.80it/s, v_num=0]
Epoch 7:  18%|█▊        | 4/22 [00:02<00:09,  1.80it/s, v_num=0]
Epoch 7:  23%|██▎       | 5/22 [00:02<00:08,  2.11it/s, v_num=0]
Epoch 7:  23%|██▎       | 5/22 [00:02<00:08,  2.11it/s, v_num=0]
Epoch 7:  27%|██▋       | 6/22 [00:02<00:06,  2.39it/s, v_num=0]
Epoch 7:  27%|██▋       | 6/22 [00:02<00:06,  2.39it/s, v_num=0]
Epoch 7:  32%|███▏      | 7/22 [00:02<00:05,  2.63it/s, v_num=0]
Epoch 7:  32%|███▏      | 7/22 [00:02<00:05,  2.63it/s, v_num=0]
Epoch 7:  36%|███▋      | 8/22 [00:02<00:04,  2.85it/s, v_num=0]
Epoch 7:  36%|███▋      | 8/22 [00:02<00:04,  2.85it/s, v_num=0]
Epoch 7:  41%|████      | 9/22 [00:03<00:04,  2.84it/s, v_num=0]
Epoch 7:  41%|████      | 9/22 [00:03<00:04,  2.84it/s, v_num=0]
Epoch 7:  45%|████▌     | 10/22 [00:03<00:03,  3.01it/s, v_num=0]
Epoch 7:  45%|████▌     | 10/22 [00:03<00:03,  3.01it/s, v_num=0]
Epoch 7:  50%|█████     | 11/22 [00:03<00:03,  3.18it/s, v_num=0]
Epoch 7:  50%|█████     | 11/22 [00:03<00:03,  3.17it/s, v_num=0]
Epoch 7:  55%|█████▍    | 12/22 [00:03<00:03,  3.33it/s, v_num=0]
Epoch 7:  55%|█████▍    | 12/22 [00:03<00:03,  3.32it/s, v_num=0]
Epoch 7:  59%|█████▉    | 13/22 [00:03<00:02,  3.46it/s, v_num=0]
Epoch 7:  59%|█████▉    | 13/22 [00:03<00:02,  3.46it/s, v_num=0]
Epoch 7:  64%|██████▎   | 14/22 [00:03<00:02,  3.59it/s, v_num=0]
Epoch 7:  64%|██████▎   | 14/22 [00:03<00:02,  3.59it/s, v_num=0]
Epoch 7:  68%|██████▊   | 15/22 [00:04<00:01,  3.71it/s, v_num=0]
Epoch 7:  68%|██████▊   | 15/22 [00:04<00:01,  3.71it/s, v_num=0]
Epoch 7:  73%|███████▎  | 16/22 [00:04<00:01,  3.82it/s, v_num=0]
Epoch 7:  73%|███████▎  | 16/22 [00:04<00:01,  3.82it/s, v_num=0]
Epoch 7:  77%|███████▋  | 17/22 [00:04<00:01,  3.77it/s, v_num=0]
Epoch 7:  77%|███████▋  | 17/22 [00:04<00:01,  3.77it/s, v_num=0]
Epoch 7:  82%|████████▏ | 18/22 [00:04<00:01,  3.86it/s, v_num=0]
Epoch 7:  82%|████████▏ | 18/22 [00:04<00:01,  3.86it/s, v_num=0]
Epoch 7:  86%|████████▋ | 19/22 [00:04<00:00,  3.95it/s, v_num=0]
Epoch 7:  86%|████████▋ | 19/22 [00:04<00:00,  3.95it/s, v_num=0]
Epoch 7:  91%|█████████ | 20/22 [00:04<00:00,  4.04it/s, v_num=0]
Epoch 7:  91%|█████████ | 20/22 [00:04<00:00,  4.04it/s, v_num=0]
Epoch 7:  95%|█████████▌| 21/22 [00:05<00:00,  4.12it/s, v_num=0]
Epoch 7:  95%|█████████▌| 21/22 [00:05<00:00,  4.12it/s, v_num=0]
Epoch 7: 100%|██████████| 22/22 [00:05<00:00,  4.20it/s, v_num=0]
Epoch 7: 100%|██████████| 22/22 [00:05<00:00,  4.20it/s, v_num=0]
Epoch 7: 100%|██████████| 22/22 [00:05<00:00,  4.20it/s, v_num=0]
Epoch 7:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 8:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 8:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 8:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 8:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 8:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 8:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, v_num=0]
Epoch 8:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, v_num=0]
Epoch 8:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, v_num=0]
Epoch 8:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, v_num=0]
Epoch 8:  23%|██▎       | 5/22 [00:02<00:08,  2.09it/s, v_num=0]
Epoch 8:  23%|██▎       | 5/22 [00:02<00:08,  2.09it/s, v_num=0]
Epoch 8:  27%|██▋       | 6/22 [00:02<00:06,  2.37it/s, v_num=0]
Epoch 8:  27%|██▋       | 6/22 [00:02<00:06,  2.37it/s, v_num=0]
Epoch 8:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, v_num=0]
Epoch 8:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, v_num=0]
Epoch 8:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, v_num=0]
Epoch 8:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, v_num=0]
Epoch 8:  41%|████      | 9/22 [00:03<00:04,  2.87it/s, v_num=0]
Epoch 8:  41%|████      | 9/22 [00:03<00:04,  2.87it/s, v_num=0]
Epoch 8:  45%|████▌     | 10/22 [00:03<00:03,  3.05it/s, v_num=0]
Epoch 8:  45%|████▌     | 10/22 [00:03<00:03,  3.05it/s, v_num=0]
Epoch 8:  50%|█████     | 11/22 [00:03<00:03,  3.21it/s, v_num=0]
Epoch 8:  50%|█████     | 11/22 [00:03<00:03,  3.21it/s, v_num=0]
Epoch 8:  55%|█████▍    | 12/22 [00:03<00:02,  3.36it/s, v_num=0]
Epoch 8:  55%|█████▍    | 12/22 [00:03<00:02,  3.36it/s, v_num=0]
Epoch 8:  59%|█████▉    | 13/22 [00:03<00:02,  3.49it/s, v_num=0]
Epoch 8:  59%|█████▉    | 13/22 [00:03<00:02,  3.49it/s, v_num=0]
Epoch 8:  64%|██████▎   | 14/22 [00:03<00:02,  3.62it/s, v_num=0]
Epoch 8:  64%|██████▎   | 14/22 [00:03<00:02,  3.62it/s, v_num=0]
Epoch 8:  68%|██████▊   | 15/22 [00:04<00:01,  3.74it/s, v_num=0]
Epoch 8:  68%|██████▊   | 15/22 [00:04<00:01,  3.74it/s, v_num=0]
Epoch 8:  73%|███████▎  | 16/22 [00:04<00:01,  3.85it/s, v_num=0]
Epoch 8:  73%|███████▎  | 16/22 [00:04<00:01,  3.85it/s, v_num=0]
Epoch 8:  77%|███████▋  | 17/22 [00:04<00:01,  3.82it/s, v_num=0]
Epoch 8:  77%|███████▋  | 17/22 [00:04<00:01,  3.82it/s, v_num=0]
Epoch 8:  82%|████████▏ | 18/22 [00:04<00:01,  3.92it/s, v_num=0]
Epoch 8:  82%|████████▏ | 18/22 [00:04<00:01,  3.92it/s, v_num=0]
Epoch 8:  86%|████████▋ | 19/22 [00:04<00:00,  4.01it/s, v_num=0]
Epoch 8:  86%|████████▋ | 19/22 [00:04<00:00,  4.01it/s, v_num=0]
Epoch 8:  91%|█████████ | 20/22 [00:04<00:00,  4.09it/s, v_num=0]
Epoch 8:  91%|█████████ | 20/22 [00:04<00:00,  4.09it/s, v_num=0]
Epoch 8:  95%|█████████▌| 21/22 [00:05<00:00,  4.17it/s, v_num=0]
Epoch 8:  95%|█████████▌| 21/22 [00:05<00:00,  4.17it/s, v_num=0]
Epoch 8: 100%|██████████| 22/22 [00:05<00:00,  4.24it/s, v_num=0]
Epoch 8: 100%|██████████| 22/22 [00:05<00:00,  4.24it/s, v_num=0]
Epoch 8: 100%|██████████| 22/22 [00:05<00:00,  4.24it/s, v_num=0]
Epoch 8:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 9:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 9:   5%|▍         | 1/22 [00:01<00:37,  0.55it/s, v_num=0]
Epoch 9:   5%|▍         | 1/22 [00:01<00:37,  0.55it/s, v_num=0]
Epoch 9:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 9:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 9:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, v_num=0]
Epoch 9:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, v_num=0]
Epoch 9:  18%|█▊        | 4/22 [00:02<00:10,  1.78it/s, v_num=0]
Epoch 9:  18%|█▊        | 4/22 [00:02<00:10,  1.78it/s, v_num=0]
Epoch 9:  23%|██▎       | 5/22 [00:02<00:08,  2.09it/s, v_num=0]
Epoch 9:  23%|██▎       | 5/22 [00:02<00:08,  2.09it/s, v_num=0]
Epoch 9:  27%|██▋       | 6/22 [00:02<00:06,  2.36it/s, v_num=0]
Epoch 9:  27%|██▋       | 6/22 [00:02<00:06,  2.36it/s, v_num=0]
Epoch 9:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, v_num=0]
Epoch 9:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, v_num=0]
Epoch 9:  36%|███▋      | 8/22 [00:02<00:04,  2.82it/s, v_num=0]
Epoch 9:  36%|███▋      | 8/22 [00:02<00:04,  2.82it/s, v_num=0]
Epoch 9:  41%|████      | 9/22 [00:03<00:04,  2.86it/s, v_num=0]
Epoch 9:  41%|████      | 9/22 [00:03<00:04,  2.86it/s, v_num=0]
Epoch 9:  45%|████▌     | 10/22 [00:03<00:03,  3.04it/s, v_num=0]
Epoch 9:  45%|████▌     | 10/22 [00:03<00:03,  3.04it/s, v_num=0]
Epoch 9:  50%|█████     | 11/22 [00:03<00:03,  3.20it/s, v_num=0]
Epoch 9:  50%|█████     | 11/22 [00:03<00:03,  3.20it/s, v_num=0]
Epoch 9:  55%|█████▍    | 12/22 [00:03<00:02,  3.35it/s, v_num=0]
Epoch 9:  55%|█████▍    | 12/22 [00:03<00:02,  3.35it/s, v_num=0]
Epoch 9:  59%|█████▉    | 13/22 [00:03<00:02,  3.48it/s, v_num=0]
Epoch 9:  59%|█████▉    | 13/22 [00:03<00:02,  3.48it/s, v_num=0]
Epoch 9:  64%|██████▎   | 14/22 [00:03<00:02,  3.61it/s, v_num=0]
Epoch 9:  64%|██████▎   | 14/22 [00:03<00:02,  3.61it/s, v_num=0]
Epoch 9:  68%|██████▊   | 15/22 [00:04<00:01,  3.72it/s, v_num=0]
Epoch 9:  68%|██████▊   | 15/22 [00:04<00:01,  3.72it/s, v_num=0]
Epoch 9:  73%|███████▎  | 16/22 [00:04<00:01,  3.83it/s, v_num=0]
Epoch 9:  73%|███████▎  | 16/22 [00:04<00:01,  3.83it/s, v_num=0]
Epoch 9:  77%|███████▋  | 17/22 [00:04<00:01,  3.78it/s, v_num=0]
Epoch 9:  77%|███████▋  | 17/22 [00:04<00:01,  3.78it/s, v_num=0]
Epoch 9:  82%|████████▏ | 18/22 [00:04<00:01,  3.87it/s, v_num=0]
Epoch 9:  82%|████████▏ | 18/22 [00:04<00:01,  3.87it/s, v_num=0]
Epoch 9:  86%|████████▋ | 19/22 [00:04<00:00,  3.96it/s, v_num=0]
Epoch 9:  86%|████████▋ | 19/22 [00:04<00:00,  3.96it/s, v_num=0]
Epoch 9:  91%|█████████ | 20/22 [00:04<00:00,  4.04it/s, v_num=0]
Epoch 9:  91%|█████████ | 20/22 [00:04<00:00,  4.04it/s, v_num=0]
Epoch 9:  95%|█████████▌| 21/22 [00:05<00:00,  4.13it/s, v_num=0]
Epoch 9:  95%|█████████▌| 21/22 [00:05<00:00,  4.12it/s, v_num=0]
Epoch 9: 100%|██████████| 22/22 [00:05<00:00,  4.20it/s, v_num=0]
Epoch 9: 100%|██████████| 22/22 [00:05<00:00,  4.20it/s, v_num=0]
Epoch 9: 100%|██████████| 22/22 [00:05<00:00,  4.20it/s, v_num=0]
Epoch 9:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 10:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 10:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 10:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 10:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 10:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 10:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, v_num=0]
Epoch 10:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, v_num=0]
Epoch 10:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, v_num=0]
Epoch 10:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, v_num=0]
Epoch 10:  23%|██▎       | 5/22 [00:02<00:08,  2.10it/s, v_num=0]
Epoch 10:  23%|██▎       | 5/22 [00:02<00:08,  2.10it/s, v_num=0]
Epoch 10:  27%|██▋       | 6/22 [00:02<00:06,  2.38it/s, v_num=0]
Epoch 10:  27%|██▋       | 6/22 [00:02<00:06,  2.38it/s, v_num=0]
Epoch 10:  32%|███▏      | 7/22 [00:02<00:05,  2.62it/s, v_num=0]
Epoch 10:  32%|███▏      | 7/22 [00:02<00:05,  2.62it/s, v_num=0]
Epoch 10:  36%|███▋      | 8/22 [00:02<00:04,  2.84it/s, v_num=0]
Epoch 10:  36%|███▋      | 8/22 [00:02<00:04,  2.84it/s, v_num=0]
Epoch 10:  41%|████      | 9/22 [00:03<00:04,  2.89it/s, v_num=0]
Epoch 10:  41%|████      | 9/22 [00:03<00:04,  2.89it/s, v_num=0]
Epoch 10:  45%|████▌     | 10/22 [00:03<00:03,  3.07it/s, v_num=0]
Epoch 10:  45%|████▌     | 10/22 [00:03<00:03,  3.07it/s, v_num=0]
Epoch 10:  50%|█████     | 11/22 [00:03<00:03,  3.23it/s, v_num=0]
Epoch 10:  50%|█████     | 11/22 [00:03<00:03,  3.23it/s, v_num=0]
Epoch 10:  55%|█████▍    | 12/22 [00:03<00:02,  3.38it/s, v_num=0]
Epoch 10:  55%|█████▍    | 12/22 [00:03<00:02,  3.38it/s, v_num=0]
Epoch 10:  59%|█████▉    | 13/22 [00:03<00:02,  3.52it/s, v_num=0]
Epoch 10:  59%|█████▉    | 13/22 [00:03<00:02,  3.52it/s, v_num=0]
Epoch 10:  64%|██████▎   | 14/22 [00:03<00:02,  3.64it/s, v_num=0]
Epoch 10:  64%|██████▎   | 14/22 [00:03<00:02,  3.64it/s, v_num=0]
Epoch 10:  68%|██████▊   | 15/22 [00:03<00:01,  3.76it/s, v_num=0]
Epoch 10:  68%|██████▊   | 15/22 [00:03<00:01,  3.76it/s, v_num=0]
Epoch 10:  73%|███████▎  | 16/22 [00:04<00:01,  3.87it/s, v_num=0]
Epoch 10:  73%|███████▎  | 16/22 [00:04<00:01,  3.87it/s, v_num=0]
Epoch 10:  77%|███████▋  | 17/22 [00:04<00:01,  3.80it/s, v_num=0]
Epoch 10:  77%|███████▋  | 17/22 [00:04<00:01,  3.79it/s, v_num=0]
Epoch 10:  82%|████████▏ | 18/22 [00:04<00:01,  3.89it/s, v_num=0]
Epoch 10:  82%|████████▏ | 18/22 [00:04<00:01,  3.89it/s, v_num=0]
Epoch 10:  86%|████████▋ | 19/22 [00:04<00:00,  3.98it/s, v_num=0]
Epoch 10:  86%|████████▋ | 19/22 [00:04<00:00,  3.98it/s, v_num=0]
Epoch 10:  91%|█████████ | 20/22 [00:04<00:00,  4.06it/s, v_num=0]
Epoch 10:  91%|█████████ | 20/22 [00:04<00:00,  4.06it/s, v_num=0]
Epoch 10:  95%|█████████▌| 21/22 [00:05<00:00,  4.14it/s, v_num=0]
Epoch 10:  95%|█████████▌| 21/22 [00:05<00:00,  4.14it/s, v_num=0]
Epoch 10: 100%|██████████| 22/22 [00:05<00:00,  4.22it/s, v_num=0]
Epoch 10: 100%|██████████| 22/22 [00:05<00:00,  4.22it/s, v_num=0]
Epoch 10: 100%|██████████| 22/22 [00:05<00:00,  4.22it/s, v_num=0]
Epoch 10:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 11:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 11:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 11:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 11:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 11:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 11:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, v_num=0]
Epoch 11:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, v_num=0]
Epoch 11:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, v_num=0]
Epoch 11:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, v_num=0]
Epoch 11:  23%|██▎       | 5/22 [00:02<00:08,  2.10it/s, v_num=0]
Epoch 11:  23%|██▎       | 5/22 [00:02<00:08,  2.10it/s, v_num=0]
Epoch 11:  27%|██▋       | 6/22 [00:02<00:06,  2.37it/s, v_num=0]
Epoch 11:  27%|██▋       | 6/22 [00:02<00:06,  2.37it/s, v_num=0]
Epoch 11:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, v_num=0]
Epoch 11:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, v_num=0]
Epoch 11:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, v_num=0]
Epoch 11:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, v_num=0]
Epoch 11:  41%|████      | 9/22 [00:03<00:04,  2.86it/s, v_num=0]
Epoch 11:  41%|████      | 9/22 [00:03<00:04,  2.86it/s, v_num=0]
Epoch 11:  45%|████▌     | 10/22 [00:03<00:03,  3.03it/s, v_num=0]
Epoch 11:  45%|████▌     | 10/22 [00:03<00:03,  3.03it/s, v_num=0]
Epoch 11:  50%|█████     | 11/22 [00:03<00:03,  3.20it/s, v_num=0]
Epoch 11:  50%|█████     | 11/22 [00:03<00:03,  3.19it/s, v_num=0]
Epoch 11:  55%|█████▍    | 12/22 [00:03<00:02,  3.34it/s, v_num=0]
Epoch 11:  55%|█████▍    | 12/22 [00:03<00:02,  3.34it/s, v_num=0]
Epoch 11:  59%|█████▉    | 13/22 [00:03<00:02,  3.48it/s, v_num=0]
Epoch 11:  59%|█████▉    | 13/22 [00:03<00:02,  3.48it/s, v_num=0]
Epoch 11:  64%|██████▎   | 14/22 [00:03<00:02,  3.60it/s, v_num=0]
Epoch 11:  64%|██████▎   | 14/22 [00:03<00:02,  3.60it/s, v_num=0]
Epoch 11:  68%|██████▊   | 15/22 [00:04<00:01,  3.72it/s, v_num=0]
Epoch 11:  68%|██████▊   | 15/22 [00:04<00:01,  3.72it/s, v_num=0]
Epoch 11:  73%|███████▎  | 16/22 [00:04<00:01,  3.83it/s, v_num=0]
Epoch 11:  73%|███████▎  | 16/22 [00:04<00:01,  3.83it/s, v_num=0]
Epoch 11:  77%|███████▋  | 17/22 [00:04<00:01,  3.82it/s, v_num=0]
Epoch 11:  77%|███████▋  | 17/22 [00:04<00:01,  3.82it/s, v_num=0]
Epoch 11:  82%|████████▏ | 18/22 [00:04<00:01,  3.91it/s, v_num=0]
Epoch 11:  82%|████████▏ | 18/22 [00:04<00:01,  3.91it/s, v_num=0]
Epoch 11:  86%|████████▋ | 19/22 [00:04<00:00,  4.00it/s, v_num=0]
Epoch 11:  86%|████████▋ | 19/22 [00:04<00:00,  4.00it/s, v_num=0]
Epoch 11:  91%|█████████ | 20/22 [00:04<00:00,  4.09it/s, v_num=0]
Epoch 11:  91%|█████████ | 20/22 [00:04<00:00,  4.09it/s, v_num=0]
Epoch 11:  95%|█████████▌| 21/22 [00:05<00:00,  4.17it/s, v_num=0]
Epoch 11:  95%|█████████▌| 21/22 [00:05<00:00,  4.17it/s, v_num=0]
Epoch 11: 100%|██████████| 22/22 [00:05<00:00,  4.24it/s, v_num=0]
Epoch 11: 100%|██████████| 22/22 [00:05<00:00,  4.24it/s, v_num=0]
Epoch 11: 100%|██████████| 22/22 [00:05<00:00,  4.24it/s, v_num=0]
Epoch 11:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 12:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 12:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 12:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 12:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 12:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 12:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, v_num=0]
Epoch 12:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, v_num=0]
Epoch 12:  18%|█▊        | 4/22 [00:02<00:10,  1.80it/s, v_num=0]
Epoch 12:  18%|█▊        | 4/22 [00:02<00:10,  1.80it/s, v_num=0]
Epoch 12:  23%|██▎       | 5/22 [00:02<00:08,  2.11it/s, v_num=0]
Epoch 12:  23%|██▎       | 5/22 [00:02<00:08,  2.11it/s, v_num=0]
Epoch 12:  27%|██▋       | 6/22 [00:02<00:06,  2.38it/s, v_num=0]
Epoch 12:  27%|██▋       | 6/22 [00:02<00:06,  2.38it/s, v_num=0]
Epoch 12:  32%|███▏      | 7/22 [00:02<00:05,  2.62it/s, v_num=0]
Epoch 12:  32%|███▏      | 7/22 [00:02<00:05,  2.62it/s, v_num=0]
Epoch 12:  36%|███▋      | 8/22 [00:02<00:04,  2.84it/s, v_num=0]
Epoch 12:  36%|███▋      | 8/22 [00:02<00:04,  2.84it/s, v_num=0]
Epoch 12:  41%|████      | 9/22 [00:03<00:04,  2.83it/s, v_num=0]
Epoch 12:  41%|████      | 9/22 [00:03<00:04,  2.83it/s, v_num=0]
Epoch 12:  45%|████▌     | 10/22 [00:03<00:03,  3.00it/s, v_num=0]
Epoch 12:  45%|████▌     | 10/22 [00:03<00:03,  3.00it/s, v_num=0]
Epoch 12:  50%|█████     | 11/22 [00:03<00:03,  3.17it/s, v_num=0]
Epoch 12:  50%|█████     | 11/22 [00:03<00:03,  3.17it/s, v_num=0]
Epoch 12:  55%|█████▍    | 12/22 [00:03<00:03,  3.32it/s, v_num=0]
Epoch 12:  55%|█████▍    | 12/22 [00:03<00:03,  3.32it/s, v_num=0]
Epoch 12:  59%|█████▉    | 13/22 [00:03<00:02,  3.45it/s, v_num=0]
Epoch 12:  59%|█████▉    | 13/22 [00:03<00:02,  3.45it/s, v_num=0]
Epoch 12:  64%|██████▎   | 14/22 [00:03<00:02,  3.58it/s, v_num=0]
Epoch 12:  64%|██████▎   | 14/22 [00:03<00:02,  3.58it/s, v_num=0]
Epoch 12:  68%|██████▊   | 15/22 [00:04<00:01,  3.70it/s, v_num=0]
Epoch 12:  68%|██████▊   | 15/22 [00:04<00:01,  3.70it/s, v_num=0]
Epoch 12:  73%|███████▎  | 16/22 [00:04<00:01,  3.80it/s, v_num=0]
Epoch 12:  73%|███████▎  | 16/22 [00:04<00:01,  3.80it/s, v_num=0]
Epoch 12:  77%|███████▋  | 17/22 [00:04<00:01,  3.77it/s, v_num=0]
Epoch 12:  77%|███████▋  | 17/22 [00:04<00:01,  3.77it/s, v_num=0]
Epoch 12:  82%|████████▏ | 18/22 [00:04<00:01,  3.87it/s, v_num=0]
Epoch 12:  82%|████████▏ | 18/22 [00:04<00:01,  3.87it/s, v_num=0]
Epoch 12:  86%|████████▋ | 19/22 [00:04<00:00,  3.96it/s, v_num=0]
Epoch 12:  86%|████████▋ | 19/22 [00:04<00:00,  3.96it/s, v_num=0]
Epoch 12:  91%|█████████ | 20/22 [00:04<00:00,  4.04it/s, v_num=0]
Epoch 12:  91%|█████████ | 20/22 [00:04<00:00,  4.04it/s, v_num=0]
Epoch 12:  95%|█████████▌| 21/22 [00:05<00:00,  4.12it/s, v_num=0]
Epoch 12:  95%|█████████▌| 21/22 [00:05<00:00,  4.12it/s, v_num=0]
Epoch 12: 100%|██████████| 22/22 [00:05<00:00,  4.20it/s, v_num=0]
Epoch 12: 100%|██████████| 22/22 [00:05<00:00,  4.20it/s, v_num=0]
Epoch 12: 100%|██████████| 22/22 [00:05<00:00,  4.20it/s, v_num=0]
Epoch 12:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 13:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 13:   5%|▍         | 1/22 [00:01<00:36,  0.57it/s, v_num=0]
Epoch 13:   5%|▍         | 1/22 [00:01<00:36,  0.57it/s, v_num=0]
Epoch 13:   9%|▉         | 2/22 [00:01<00:18,  1.05it/s, v_num=0]
Epoch 13:   9%|▉         | 2/22 [00:01<00:18,  1.05it/s, v_num=0]
Epoch 13:  14%|█▎        | 3/22 [00:02<00:12,  1.47it/s, v_num=0]
Epoch 13:  14%|█▎        | 3/22 [00:02<00:12,  1.47it/s, v_num=0]
Epoch 13:  18%|█▊        | 4/22 [00:02<00:09,  1.82it/s, v_num=0]
Epoch 13:  18%|█▊        | 4/22 [00:02<00:09,  1.82it/s, v_num=0]
Epoch 13:  23%|██▎       | 5/22 [00:02<00:07,  2.14it/s, v_num=0]
Epoch 13:  23%|██▎       | 5/22 [00:02<00:07,  2.14it/s, v_num=0]
Epoch 13:  27%|██▋       | 6/22 [00:02<00:06,  2.41it/s, v_num=0]
Epoch 13:  27%|██▋       | 6/22 [00:02<00:06,  2.41it/s, v_num=0]
Epoch 13:  32%|███▏      | 7/22 [00:02<00:05,  2.66it/s, v_num=0]
Epoch 13:  32%|███▏      | 7/22 [00:02<00:05,  2.66it/s, v_num=0]
Epoch 13:  36%|███▋      | 8/22 [00:02<00:04,  2.88it/s, v_num=0]
Epoch 13:  36%|███▋      | 8/22 [00:02<00:04,  2.88it/s, v_num=0]
Epoch 13:  41%|████      | 9/22 [00:03<00:04,  2.95it/s, v_num=0]
Epoch 13:  41%|████      | 9/22 [00:03<00:04,  2.95it/s, v_num=0]
Epoch 13:  45%|████▌     | 10/22 [00:03<00:03,  3.13it/s, v_num=0]
Epoch 13:  45%|████▌     | 10/22 [00:03<00:03,  3.13it/s, v_num=0]
Epoch 13:  50%|█████     | 11/22 [00:03<00:03,  3.29it/s, v_num=0]
Epoch 13:  50%|█████     | 11/22 [00:03<00:03,  3.29it/s, v_num=0]
Epoch 13:  55%|█████▍    | 12/22 [00:03<00:02,  3.44it/s, v_num=0]
Epoch 13:  55%|█████▍    | 12/22 [00:03<00:02,  3.44it/s, v_num=0]
Epoch 13:  59%|█████▉    | 13/22 [00:03<00:02,  3.57it/s, v_num=0]
Epoch 13:  59%|█████▉    | 13/22 [00:03<00:02,  3.57it/s, v_num=0]
Epoch 13:  64%|██████▎   | 14/22 [00:03<00:02,  3.70it/s, v_num=0]
Epoch 13:  64%|██████▎   | 14/22 [00:03<00:02,  3.70it/s, v_num=0]
Epoch 13:  68%|██████▊   | 15/22 [00:03<00:01,  3.81it/s, v_num=0]
Epoch 13:  68%|██████▊   | 15/22 [00:03<00:01,  3.81it/s, v_num=0]
Epoch 13:  73%|███████▎  | 16/22 [00:04<00:01,  3.92it/s, v_num=0]
Epoch 13:  73%|███████▎  | 16/22 [00:04<00:01,  3.92it/s, v_num=0]
Epoch 13:  77%|███████▋  | 17/22 [00:04<00:01,  3.91it/s, v_num=0]
Epoch 13:  77%|███████▋  | 17/22 [00:04<00:01,  3.91it/s, v_num=0]
Epoch 13:  82%|████████▏ | 18/22 [00:04<00:00,  4.01it/s, v_num=0]
Epoch 13:  82%|████████▏ | 18/22 [00:04<00:00,  4.01it/s, v_num=0]
Epoch 13:  86%|████████▋ | 19/22 [00:04<00:00,  4.10it/s, v_num=0]
Epoch 13:  86%|████████▋ | 19/22 [00:04<00:00,  4.10it/s, v_num=0]
Epoch 13:  91%|█████████ | 20/22 [00:04<00:00,  4.19it/s, v_num=0]
Epoch 13:  91%|█████████ | 20/22 [00:04<00:00,  4.19it/s, v_num=0]
Epoch 13:  95%|█████████▌| 21/22 [00:04<00:00,  4.27it/s, v_num=0]
Epoch 13:  95%|█████████▌| 21/22 [00:04<00:00,  4.27it/s, v_num=0]
Epoch 13: 100%|██████████| 22/22 [00:05<00:00,  4.34it/s, v_num=0]
Epoch 13: 100%|██████████| 22/22 [00:05<00:00,  4.34it/s, v_num=0]
Epoch 13: 100%|██████████| 22/22 [00:05<00:00,  4.34it/s, v_num=0]
Epoch 13:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 14:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 14:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 14:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 14:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 14:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 14:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, v_num=0]
Epoch 14:  14%|█▎        | 3/22 [00:02<00:13,  1.44it/s, v_num=0]
Epoch 14:  18%|█▊        | 4/22 [00:02<00:10,  1.80it/s, v_num=0]
Epoch 14:  18%|█▊        | 4/22 [00:02<00:10,  1.80it/s, v_num=0]
Epoch 14:  23%|██▎       | 5/22 [00:02<00:08,  2.11it/s, v_num=0]
Epoch 14:  23%|██▎       | 5/22 [00:02<00:08,  2.11it/s, v_num=0]
Epoch 14:  27%|██▋       | 6/22 [00:02<00:06,  2.38it/s, v_num=0]
Epoch 14:  27%|██▋       | 6/22 [00:02<00:06,  2.38it/s, v_num=0]
Epoch 14:  32%|███▏      | 7/22 [00:02<00:05,  2.62it/s, v_num=0]
Epoch 14:  32%|███▏      | 7/22 [00:02<00:05,  2.62it/s, v_num=0]
Epoch 14:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, v_num=0]
Epoch 14:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, v_num=0]
Epoch 14:  41%|████      | 9/22 [00:03<00:04,  2.84it/s, v_num=0]
Epoch 14:  41%|████      | 9/22 [00:03<00:04,  2.84it/s, v_num=0]
Epoch 14:  45%|████▌     | 10/22 [00:03<00:03,  3.02it/s, v_num=0]
Epoch 14:  45%|████▌     | 10/22 [00:03<00:03,  3.02it/s, v_num=0]
Epoch 14:  50%|█████     | 11/22 [00:03<00:03,  3.18it/s, v_num=0]
Epoch 14:  50%|█████     | 11/22 [00:03<00:03,  3.18it/s, v_num=0]
Epoch 14:  55%|█████▍    | 12/22 [00:03<00:03,  3.32it/s, v_num=0]
Epoch 14:  55%|█████▍    | 12/22 [00:03<00:03,  3.32it/s, v_num=0]
Epoch 14:  59%|█████▉    | 13/22 [00:03<00:02,  3.46it/s, v_num=0]
Epoch 14:  59%|█████▉    | 13/22 [00:03<00:02,  3.46it/s, v_num=0]
Epoch 14:  64%|██████▎   | 14/22 [00:03<00:02,  3.59it/s, v_num=0]
Epoch 14:  64%|██████▎   | 14/22 [00:03<00:02,  3.59it/s, v_num=0]
Epoch 14:  68%|██████▊   | 15/22 [00:04<00:01,  3.70it/s, v_num=0]
Epoch 14:  68%|██████▊   | 15/22 [00:04<00:01,  3.70it/s, v_num=0]
Epoch 14:  73%|███████▎  | 16/22 [00:04<00:01,  3.81it/s, v_num=0]
Epoch 14:  73%|███████▎  | 16/22 [00:04<00:01,  3.81it/s, v_num=0]
Epoch 14:  77%|███████▋  | 17/22 [00:04<00:01,  3.73it/s, v_num=0]
Epoch 14:  77%|███████▋  | 17/22 [00:04<00:01,  3.73it/s, v_num=0]
Epoch 14:  82%|████████▏ | 18/22 [00:04<00:01,  3.83it/s, v_num=0]
Epoch 14:  82%|████████▏ | 18/22 [00:04<00:01,  3.83it/s, v_num=0]
Epoch 14:  86%|████████▋ | 19/22 [00:04<00:00,  3.92it/s, v_num=0]
Epoch 14:  86%|████████▋ | 19/22 [00:04<00:00,  3.92it/s, v_num=0]
Epoch 14:  91%|█████████ | 20/22 [00:04<00:00,  4.00it/s, v_num=0]
Epoch 14:  91%|█████████ | 20/22 [00:04<00:00,  4.00it/s, v_num=0]
Epoch 14:  95%|█████████▌| 21/22 [00:05<00:00,  4.09it/s, v_num=0]
Epoch 14:  95%|█████████▌| 21/22 [00:05<00:00,  4.09it/s, v_num=0]
Epoch 14: 100%|██████████| 22/22 [00:05<00:00,  4.16it/s, v_num=0]
Epoch 14: 100%|██████████| 22/22 [00:05<00:00,  4.16it/s, v_num=0]
Epoch 14: 100%|██████████| 22/22 [00:05<00:00,  4.16it/s, v_num=0]
Epoch 14:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 15:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 15:   5%|▍         | 1/22 [00:01<00:37,  0.55it/s, v_num=0]
Epoch 15:   5%|▍         | 1/22 [00:01<00:37,  0.55it/s, v_num=0]
Epoch 15:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 15:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 15:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, v_num=0]
Epoch 15:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, v_num=0]
Epoch 15:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, v_num=0]
Epoch 15:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, v_num=0]
Epoch 15:  23%|██▎       | 5/22 [00:02<00:08,  2.10it/s, v_num=0]
Epoch 15:  23%|██▎       | 5/22 [00:02<00:08,  2.10it/s, v_num=0]
Epoch 15:  27%|██▋       | 6/22 [00:02<00:06,  2.37it/s, v_num=0]
Epoch 15:  27%|██▋       | 6/22 [00:02<00:06,  2.37it/s, v_num=0]
Epoch 15:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, v_num=0]
Epoch 15:  32%|███▏      | 7/22 [00:02<00:05,  2.61it/s, v_num=0]
Epoch 15:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, v_num=0]
Epoch 15:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, v_num=0]
Epoch 15:  41%|████      | 9/22 [00:03<00:04,  2.90it/s, v_num=0]
Epoch 15:  41%|████      | 9/22 [00:03<00:04,  2.90it/s, v_num=0]
Epoch 15:  45%|████▌     | 10/22 [00:03<00:03,  3.07it/s, v_num=0]
Epoch 15:  45%|████▌     | 10/22 [00:03<00:03,  3.07it/s, v_num=0]
Epoch 15:  50%|█████     | 11/22 [00:03<00:03,  3.23it/s, v_num=0]
Epoch 15:  50%|█████     | 11/22 [00:03<00:03,  3.23it/s, v_num=0]
Epoch 15:  55%|█████▍    | 12/22 [00:03<00:02,  3.38it/s, v_num=0]
Epoch 15:  55%|█████▍    | 12/22 [00:03<00:02,  3.38it/s, v_num=0]
Epoch 15:  59%|█████▉    | 13/22 [00:03<00:02,  3.52it/s, v_num=0]
Epoch 15:  59%|█████▉    | 13/22 [00:03<00:02,  3.52it/s, v_num=0]
Epoch 15:  64%|██████▎   | 14/22 [00:03<00:02,  3.64it/s, v_num=0]
Epoch 15:  64%|██████▎   | 14/22 [00:03<00:02,  3.64it/s, v_num=0]
Epoch 15:  68%|██████▊   | 15/22 [00:03<00:01,  3.76it/s, v_num=0]
Epoch 15:  68%|██████▊   | 15/22 [00:03<00:01,  3.76it/s, v_num=0]
Epoch 15:  73%|███████▎  | 16/22 [00:04<00:01,  3.86it/s, v_num=0]
Epoch 15:  73%|███████▎  | 16/22 [00:04<00:01,  3.86it/s, v_num=0]
Epoch 15:  77%|███████▋  | 17/22 [00:04<00:01,  3.79it/s, v_num=0]
Epoch 15:  77%|███████▋  | 17/22 [00:04<00:01,  3.79it/s, v_num=0]
Epoch 15:  82%|████████▏ | 18/22 [00:04<00:01,  3.89it/s, v_num=0]
Epoch 15:  82%|████████▏ | 18/22 [00:04<00:01,  3.89it/s, v_num=0]
Epoch 15:  86%|████████▋ | 19/22 [00:04<00:00,  3.98it/s, v_num=0]
Epoch 15:  86%|████████▋ | 19/22 [00:04<00:00,  3.98it/s, v_num=0]
Epoch 15:  91%|█████████ | 20/22 [00:04<00:00,  4.06it/s, v_num=0]
Epoch 15:  91%|█████████ | 20/22 [00:04<00:00,  4.06it/s, v_num=0]
Epoch 15:  95%|█████████▌| 21/22 [00:05<00:00,  4.14it/s, v_num=0]
Epoch 15:  95%|█████████▌| 21/22 [00:05<00:00,  4.14it/s, v_num=0]
Epoch 15: 100%|██████████| 22/22 [00:05<00:00,  4.21it/s, v_num=0]
Epoch 15: 100%|██████████| 22/22 [00:05<00:00,  4.21it/s, v_num=0]
Epoch 15: 100%|██████████| 22/22 [00:05<00:00,  4.21it/s, v_num=0]
Epoch 15:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 16:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 16:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 16:   5%|▍         | 1/22 [00:01<00:37,  0.56it/s, v_num=0]
Epoch 16:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 16:   9%|▉         | 2/22 [00:01<00:19,  1.03it/s, v_num=0]
Epoch 16:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, v_num=0]
Epoch 16:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, v_num=0]
Epoch 16:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, v_num=0]
Epoch 16:  18%|█▊        | 4/22 [00:02<00:10,  1.79it/s, v_num=0]
Epoch 16:  23%|██▎       | 5/22 [00:02<00:08,  2.10it/s, v_num=0]
Epoch 16:  23%|██▎       | 5/22 [00:02<00:08,  2.10it/s, v_num=0]
Epoch 16:  27%|██▋       | 6/22 [00:02<00:06,  2.37it/s, v_num=0]
Epoch 16:  27%|██▋       | 6/22 [00:02<00:06,  2.37it/s, v_num=0]
Epoch 16:  32%|███▏      | 7/22 [00:02<00:05,  2.62it/s, v_num=0]
Epoch 16:  32%|███▏      | 7/22 [00:02<00:05,  2.62it/s, v_num=0]
Epoch 16:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, v_num=0]
Epoch 16:  36%|███▋      | 8/22 [00:02<00:04,  2.83it/s, v_num=0]
Epoch 16:  41%|████      | 9/22 [00:03<00:04,  2.89it/s, v_num=0]
Epoch 16:  41%|████      | 9/22 [00:03<00:04,  2.89it/s, v_num=0]
Epoch 16:  45%|████▌     | 10/22 [00:03<00:03,  3.07it/s, v_num=0]
Epoch 16:  45%|████▌     | 10/22 [00:03<00:03,  3.07it/s, v_num=0]
Epoch 16:  50%|█████     | 11/22 [00:03<00:03,  3.23it/s, v_num=0]
Epoch 16:  50%|█████     | 11/22 [00:03<00:03,  3.23it/s, v_num=0]
Epoch 16:  55%|█████▍    | 12/22 [00:03<00:02,  3.37it/s, v_num=0]
Epoch 16:  55%|█████▍    | 12/22 [00:03<00:02,  3.37it/s, v_num=0]
Epoch 16:  59%|█████▉    | 13/22 [00:03<00:02,  3.52it/s, v_num=0]
Epoch 16:  59%|█████▉    | 13/22 [00:03<00:02,  3.52it/s, v_num=0]
Epoch 16:  64%|██████▎   | 14/22 [00:03<00:02,  3.64it/s, v_num=0]
Epoch 16:  64%|██████▎   | 14/22 [00:03<00:02,  3.64it/s, v_num=0]
Epoch 16:  68%|██████▊   | 15/22 [00:03<00:01,  3.76it/s, v_num=0]
Epoch 16:  68%|██████▊   | 15/22 [00:03<00:01,  3.76it/s, v_num=0]
Epoch 16:  73%|███████▎  | 16/22 [00:04<00:01,  3.87it/s, v_num=0]
Epoch 16:  73%|███████▎  | 16/22 [00:04<00:01,  3.87it/s, v_num=0]
Epoch 16:  77%|███████▋  | 17/22 [00:04<00:01,  3.85it/s, v_num=0]
Epoch 16:  77%|███████▋  | 17/22 [00:04<00:01,  3.85it/s, v_num=0]
Epoch 16:  82%|████████▏ | 18/22 [00:04<00:01,  3.94it/s, v_num=0]
Epoch 16:  82%|████████▏ | 18/22 [00:04<00:01,  3.94it/s, v_num=0]
Epoch 16:  86%|████████▋ | 19/22 [00:04<00:00,  4.03it/s, v_num=0]
Epoch 16:  86%|████████▋ | 19/22 [00:04<00:00,  4.03it/s, v_num=0]
Epoch 16:  91%|█████████ | 20/22 [00:04<00:00,  4.11it/s, v_num=0]
Epoch 16:  91%|█████████ | 20/22 [00:04<00:00,  4.11it/s, v_num=0]
Epoch 16:  95%|█████████▌| 21/22 [00:05<00:00,  4.19it/s, v_num=0]
Epoch 16:  95%|█████████▌| 21/22 [00:05<00:00,  4.19it/s, v_num=0]
Epoch 16: 100%|██████████| 22/22 [00:05<00:00,  4.27it/s, v_num=0]
Epoch 16: 100%|██████████| 22/22 [00:05<00:00,  4.27it/s, v_num=0]
Epoch 16: 100%|██████████| 22/22 [00:05<00:00,  4.27it/s, v_num=0]
Epoch 16:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 17:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 17:   5%|▍         | 1/22 [00:01<00:38,  0.55it/s, v_num=0]
Epoch 17:   5%|▍         | 1/22 [00:01<00:38,  0.55it/s, v_num=0]
Epoch 17:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 17:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 17:  14%|█▎        | 3/22 [00:02<00:13,  1.42it/s, v_num=0]
Epoch 17:  14%|█▎        | 3/22 [00:02<00:13,  1.42it/s, v_num=0]
Epoch 17:  18%|█▊        | 4/22 [00:02<00:10,  1.77it/s, v_num=0]
Epoch 17:  18%|█▊        | 4/22 [00:02<00:10,  1.77it/s, v_num=0]
Epoch 17:  23%|██▎       | 5/22 [00:02<00:08,  2.08it/s, v_num=0]
Epoch 17:  23%|██▎       | 5/22 [00:02<00:08,  2.08it/s, v_num=0]
Epoch 17:  27%|██▋       | 6/22 [00:02<00:06,  2.35it/s, v_num=0]
Epoch 17:  27%|██▋       | 6/22 [00:02<00:06,  2.35it/s, v_num=0]
Epoch 17:  32%|███▏      | 7/22 [00:02<00:05,  2.59it/s, v_num=0]
Epoch 17:  32%|███▏      | 7/22 [00:02<00:05,  2.59it/s, v_num=0]
Epoch 17:  36%|███▋      | 8/22 [00:02<00:04,  2.81it/s, v_num=0]
Epoch 17:  36%|███▋      | 8/22 [00:02<00:04,  2.81it/s, v_num=0]
Epoch 17:  41%|████      | 9/22 [00:03<00:04,  2.82it/s, v_num=0]
Epoch 17:  41%|████      | 9/22 [00:03<00:04,  2.82it/s, v_num=0]
Epoch 17:  45%|████▌     | 10/22 [00:03<00:03,  3.00it/s, v_num=0]
Epoch 17:  45%|████▌     | 10/22 [00:03<00:03,  3.00it/s, v_num=0]
Epoch 17:  50%|█████     | 11/22 [00:03<00:03,  3.16it/s, v_num=0]
Epoch 17:  50%|█████     | 11/22 [00:03<00:03,  3.16it/s, v_num=0]
Epoch 17:  55%|█████▍    | 12/22 [00:03<00:03,  3.31it/s, v_num=0]
Epoch 17:  55%|█████▍    | 12/22 [00:03<00:03,  3.31it/s, v_num=0]
Epoch 17:  59%|█████▉    | 13/22 [00:03<00:02,  3.44it/s, v_num=0]
Epoch 17:  59%|█████▉    | 13/22 [00:03<00:02,  3.44it/s, v_num=0]
Epoch 17:  64%|██████▎   | 14/22 [00:03<00:02,  3.57it/s, v_num=0]
Epoch 17:  64%|██████▎   | 14/22 [00:03<00:02,  3.57it/s, v_num=0]
Epoch 17:  68%|██████▊   | 15/22 [00:04<00:01,  3.69it/s, v_num=0]
Epoch 17:  68%|██████▊   | 15/22 [00:04<00:01,  3.69it/s, v_num=0]
Epoch 17:  73%|███████▎  | 16/22 [00:04<00:01,  3.79it/s, v_num=0]
Epoch 17:  73%|███████▎  | 16/22 [00:04<00:01,  3.79it/s, v_num=0]
Epoch 17:  77%|███████▋  | 17/22 [00:04<00:01,  3.75it/s, v_num=0]
Epoch 17:  77%|███████▋  | 17/22 [00:04<00:01,  3.75it/s, v_num=0]
Epoch 17:  82%|████████▏ | 18/22 [00:04<00:01,  3.85it/s, v_num=0]
Epoch 17:  82%|████████▏ | 18/22 [00:04<00:01,  3.85it/s, v_num=0]
Epoch 17:  86%|████████▋ | 19/22 [00:04<00:00,  3.93it/s, v_num=0]
Epoch 17:  86%|████████▋ | 19/22 [00:04<00:00,  3.93it/s, v_num=0]
Epoch 17:  91%|█████████ | 20/22 [00:04<00:00,  4.02it/s, v_num=0]
Epoch 17:  91%|█████████ | 20/22 [00:04<00:00,  4.02it/s, v_num=0]
Epoch 17:  95%|█████████▌| 21/22 [00:05<00:00,  4.10it/s, v_num=0]
Epoch 17:  95%|█████████▌| 21/22 [00:05<00:00,  4.09it/s, v_num=0]
Epoch 17: 100%|██████████| 22/22 [00:05<00:00,  4.17it/s, v_num=0]
Epoch 17: 100%|██████████| 22/22 [00:05<00:00,  4.17it/s, v_num=0]
Epoch 17: 100%|██████████| 22/22 [00:05<00:00,  4.17it/s, v_num=0]
Epoch 17:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 18:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 18:   5%|▍         | 1/22 [00:01<00:38,  0.55it/s, v_num=0]
Epoch 18:   5%|▍         | 1/22 [00:01<00:38,  0.55it/s, v_num=0]
Epoch 18:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 18:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 18:  14%|█▎        | 3/22 [00:02<00:13,  1.42it/s, v_num=0]
Epoch 18:  14%|█▎        | 3/22 [00:02<00:13,  1.42it/s, v_num=0]
Epoch 18:  18%|█▊        | 4/22 [00:02<00:10,  1.77it/s, v_num=0]
Epoch 18:  18%|█▊        | 4/22 [00:02<00:10,  1.77it/s, v_num=0]
Epoch 18:  23%|██▎       | 5/22 [00:02<00:08,  2.07it/s, v_num=0]
Epoch 18:  23%|██▎       | 5/22 [00:02<00:08,  2.07it/s, v_num=0]
Epoch 18:  27%|██▋       | 6/22 [00:02<00:06,  2.35it/s, v_num=0]
Epoch 18:  27%|██▋       | 6/22 [00:02<00:06,  2.35it/s, v_num=0]
Epoch 18:  32%|███▏      | 7/22 [00:02<00:05,  2.59it/s, v_num=0]
Epoch 18:  32%|███▏      | 7/22 [00:02<00:05,  2.59it/s, v_num=0]
Epoch 18:  36%|███▋      | 8/22 [00:02<00:04,  2.81it/s, v_num=0]
Epoch 18:  36%|███▋      | 8/22 [00:02<00:04,  2.81it/s, v_num=0]
Epoch 18:  41%|████      | 9/22 [00:03<00:04,  2.83it/s, v_num=0]
Epoch 18:  41%|████      | 9/22 [00:03<00:04,  2.83it/s, v_num=0]
Epoch 18:  45%|████▌     | 10/22 [00:03<00:03,  3.01it/s, v_num=0]
Epoch 18:  45%|████▌     | 10/22 [00:03<00:03,  3.01it/s, v_num=0]
Epoch 18:  50%|█████     | 11/22 [00:03<00:03,  3.17it/s, v_num=0]
Epoch 18:  50%|█████     | 11/22 [00:03<00:03,  3.17it/s, v_num=0]
Epoch 18:  55%|█████▍    | 12/22 [00:03<00:03,  3.32it/s, v_num=0]
Epoch 18:  55%|█████▍    | 12/22 [00:03<00:03,  3.32it/s, v_num=0]
Epoch 18:  59%|█████▉    | 13/22 [00:03<00:02,  3.46it/s, v_num=0]
Epoch 18:  59%|█████▉    | 13/22 [00:03<00:02,  3.46it/s, v_num=0]
Epoch 18:  64%|██████▎   | 14/22 [00:03<00:02,  3.58it/s, v_num=0]
Epoch 18:  64%|██████▎   | 14/22 [00:03<00:02,  3.58it/s, v_num=0]
Epoch 18:  68%|██████▊   | 15/22 [00:04<00:01,  3.69it/s, v_num=0]
Epoch 18:  68%|██████▊   | 15/22 [00:04<00:01,  3.69it/s, v_num=0]
Epoch 18:  73%|███████▎  | 16/22 [00:04<00:01,  3.80it/s, v_num=0]
Epoch 18:  73%|███████▎  | 16/22 [00:04<00:01,  3.80it/s, v_num=0]
Epoch 18:  77%|███████▋  | 17/22 [00:04<00:01,  3.79it/s, v_num=0]
Epoch 18:  77%|███████▋  | 17/22 [00:04<00:01,  3.79it/s, v_num=0]
Epoch 18:  82%|████████▏ | 18/22 [00:04<00:01,  3.89it/s, v_num=0]
Epoch 18:  82%|████████▏ | 18/22 [00:04<00:01,  3.89it/s, v_num=0]
Epoch 18:  86%|████████▋ | 19/22 [00:04<00:00,  3.98it/s, v_num=0]
Epoch 18:  86%|████████▋ | 19/22 [00:04<00:00,  3.98it/s, v_num=0]
Epoch 18:  91%|█████████ | 20/22 [00:04<00:00,  4.07it/s, v_num=0]
Epoch 18:  91%|█████████ | 20/22 [00:04<00:00,  4.06it/s, v_num=0]
Epoch 18:  95%|█████████▌| 21/22 [00:05<00:00,  4.14it/s, v_num=0]
Epoch 18:  95%|█████████▌| 21/22 [00:05<00:00,  4.14it/s, v_num=0]
Epoch 18: 100%|██████████| 22/22 [00:05<00:00,  4.22it/s, v_num=0]
Epoch 18: 100%|██████████| 22/22 [00:05<00:00,  4.22it/s, v_num=0]
Epoch 18: 100%|██████████| 22/22 [00:05<00:00,  4.22it/s, v_num=0]
Epoch 18:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 19:   0%|          | 0/22 [00:00<?, ?it/s, v_num=0]
Epoch 19:   5%|▍         | 1/22 [00:01<00:37,  0.55it/s, v_num=0]
Epoch 19:   5%|▍         | 1/22 [00:01<00:37,  0.55it/s, v_num=0]
Epoch 19:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 19:   9%|▉         | 2/22 [00:01<00:19,  1.02it/s, v_num=0]
Epoch 19:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, v_num=0]
Epoch 19:  14%|█▎        | 3/22 [00:02<00:13,  1.43it/s, v_num=0]
Epoch 19:  18%|█▊        | 4/22 [00:02<00:10,  1.78it/s, v_num=0]
Epoch 19:  18%|█▊        | 4/22 [00:02<00:10,  1.78it/s, v_num=0]
Epoch 19:  23%|██▎       | 5/22 [00:02<00:08,  2.08it/s, v_num=0]
Epoch 19:  23%|██▎       | 5/22 [00:02<00:08,  2.08it/s, v_num=0]
Epoch 19:  27%|██▋       | 6/22 [00:02<00:06,  2.36it/s, v_num=0]
Epoch 19:  27%|██▋       | 6/22 [00:02<00:06,  2.36it/s, v_num=0]
Epoch 19:  32%|███▏      | 7/22 [00:02<00:05,  2.60it/s, v_num=0]
Epoch 19:  32%|███▏      | 7/22 [00:02<00:05,  2.60it/s, v_num=0]
Epoch 19:  36%|███▋      | 8/22 [00:02<00:04,  2.82it/s, v_num=0]
Epoch 19:  36%|███▋      | 8/22 [00:02<00:04,  2.82it/s, v_num=0]
Epoch 19:  41%|████      | 9/22 [00:03<00:04,  2.85it/s, v_num=0]
Epoch 19:  41%|████      | 9/22 [00:03<00:04,  2.85it/s, v_num=0]
Epoch 19:  45%|████▌     | 10/22 [00:03<00:03,  3.02it/s, v_num=0]
Epoch 19:  45%|████▌     | 10/22 [00:03<00:03,  3.02it/s, v_num=0]
Epoch 19:  50%|█████     | 11/22 [00:03<00:03,  3.18it/s, v_num=0]
Epoch 19:  50%|█████     | 11/22 [00:03<00:03,  3.18it/s, v_num=0]
Epoch 19:  55%|█████▍    | 12/22 [00:03<00:02,  3.33it/s, v_num=0]
Epoch 19:  55%|█████▍    | 12/22 [00:03<00:02,  3.33it/s, v_num=0]
Epoch 19:  59%|█████▉    | 13/22 [00:03<00:02,  3.47it/s, v_num=0]
Epoch 19:  59%|█████▉    | 13/22 [00:03<00:02,  3.47it/s, v_num=0]
Epoch 19:  64%|██████▎   | 14/22 [00:03<00:02,  3.60it/s, v_num=0]
Epoch 19:  64%|██████▎   | 14/22 [00:03<00:02,  3.60it/s, v_num=0]
Epoch 19:  68%|██████▊   | 15/22 [00:04<00:01,  3.71it/s, v_num=0]
Epoch 19:  68%|██████▊   | 15/22 [00:04<00:01,  3.71it/s, v_num=0]
Epoch 19:  73%|███████▎  | 16/22 [00:04<00:01,  3.82it/s, v_num=0]
Epoch 19:  73%|███████▎  | 16/22 [00:04<00:01,  3.82it/s, v_num=0]
Epoch 19:  77%|███████▋  | 17/22 [00:04<00:01,  3.77it/s, v_num=0]
Epoch 19:  77%|███████▋  | 17/22 [00:04<00:01,  3.77it/s, v_num=0]
Epoch 19:  82%|████████▏ | 18/22 [00:04<00:01,  3.86it/s, v_num=0]
Epoch 19:  82%|████████▏ | 18/22 [00:04<00:01,  3.86it/s, v_num=0]
Epoch 19:  86%|████████▋ | 19/22 [00:04<00:00,  3.95it/s, v_num=0]
Epoch 19:  86%|████████▋ | 19/22 [00:04<00:00,  3.95it/s, v_num=0]
Epoch 19:  91%|█████████ | 20/22 [00:04<00:00,  4.04it/s, v_num=0]
Epoch 19:  91%|█████████ | 20/22 [00:04<00:00,  4.04it/s, v_num=0]
Epoch 19:  95%|█████████▌| 21/22 [00:05<00:00,  4.12it/s, v_num=0]
Epoch 19:  95%|█████████▌| 21/22 [00:05<00:00,  4.12it/s, v_num=0]
Epoch 19: 100%|██████████| 22/22 [00:05<00:00,  4.19it/s, v_num=0]
Epoch 19: 100%|██████████| 22/22 [00:05<00:00,  4.19it/s, v_num=0]
Epoch 19: 100%|██████████| 22/22 [00:05<00:00,  4.19it/s, v_num=0]
Epoch 19: 100%|██████████| 22/22 [00:05<00:00,  3.83it/s, v_num=0]

Next we create a helper function to generate embeddings from our test images using the model we just trained. Note that only the backbone is needed to generate embeddings, the projection head is only required for the training. Make sure to put the model into eval mode for this part!

def generate_embeddings(model, dataloader):
    """Generates representations for all images in the dataloader with
    the given model
    """
    embeddings = []
    filenames = []
    with torch.no_grad():
        for img, _, fnames in dataloader:
            img = img.to(model.device)
            emb = model.backbone(img).flatten(start_dim=1)
            embeddings.append(emb)
            filenames.extend(fnames)

    embeddings = torch.cat(embeddings, 0)
    embeddings = normalize(embeddings)
    return embeddings, filenames


model.eval()
embeddings, filenames = generate_embeddings(model, dataloader_test)

Visualize Nearest Neighbors

Let’s look at the trained embedding and visualize the nearest neighbors for a few random samples.

We create some helper functions to simplify the work

def get_image_as_np_array(filename: str):
    """Returns an image as an numpy array"""
    img = Image.open(filename)
    return np.asarray(img)


def plot_knn_examples(embeddings, filenames, n_neighbors=3, num_examples=6):
    """Plots multiple rows of random images with their nearest neighbors"""
    # lets look at the nearest neighbors for some samples
    # we use the sklearn library
    nbrs = NearestNeighbors(n_neighbors=n_neighbors).fit(embeddings)
    distances, indices = nbrs.kneighbors(embeddings)

    # get 5 random samples
    samples_idx = np.random.choice(len(indices), size=num_examples, replace=False)

    # loop through our randomly picked samples
    for idx in samples_idx:
        fig = plt.figure()
        # loop through their nearest neighbors
        for plot_x_offset, neighbor_idx in enumerate(indices[idx]):
            # add the subplot
            ax = fig.add_subplot(1, len(indices[idx]), plot_x_offset + 1)
            # get the corresponding filename for the current index
            fname = os.path.join(path_to_data, filenames[neighbor_idx])
            # plot the image
            plt.imshow(get_image_as_np_array(fname))
            # set the title to the distance of the neighbor
            ax.set_title(f"d={distances[idx][plot_x_offset]:.3f}")
            # let's disable the axis
            plt.axis("off")

Let’s do the plot of the images. The leftmost image is the query image whereas the ones next to it on the same row are the nearest neighbors. In the title we see the distance of the neighbor.

plot_knn_examples(embeddings, filenames)

• 
• 
• 
• 
• 
• 

Color Invariance

Let’s train again without color augmentation. This will force our model to respect the colors in the images.

# Set color jitter and gray scale probability to 0
new_transform = SimCLRTransform(
    input_size=input_size, vf_prob=0.5, rr_prob=0.5, cj_prob=0.0, random_gray_scale=0.0
)

# let's update the transform on the training dataset
dataset_train_simclr.transform = new_transform

# then train a new model
model = SimCLRModel()
trainer = pl.Trainer(max_epochs=max_epochs, devices=1, accelerator="gpu")
trainer.fit(model, dataloader_train_simclr)

# and generate again embeddings from the test set
model.eval()
embeddings, filenames = generate_embeddings(model, dataloader_test)

/datasets/actions-runner/core_gpu_runner_01/_work/lightly-core/lightly-core/.venv/lib/python3.12/site-packages/lightning_fabric/plugins/environments/slurm.py:204: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python /datasets/actions-runner/core_gpu_runner_01/_work/li ...
/datasets/actions-runner/core_gpu_runner_01/_work/lightly-core/lightly-core/.venv/lib/python3.12/site-packages/pytorch_lightning/loops/fit_loop.py:310: The number of training batches (22) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.

Training: |          | 0/? [00:00<?, ?it/s]
Training:   0%|          | 0/22 [00:00<?, ?it/s]
Epoch 0:   0%|          | 0/22 [00:00<?, ?it/s]
Epoch 0:   5%|▍         | 1/22 [00:01<00:23,  0.90it/s]
Epoch 0:   5%|▍         | 1/22 [00:01<00:23,  0.90it/s, v_num=1]
Epoch 0:   9%|▉         | 2/22 [00:01<00:12,  1.59it/s, v_num=1]
Epoch 0:   9%|▉         | 2/22 [00:01<00:12,  1.59it/s, v_num=1]
Epoch 0:  14%|█▎        | 3/22 [00:01<00:08,  2.14it/s, v_num=1]
Epoch 0:  14%|█▎        | 3/22 [00:01<00:08,  2.14it/s, v_num=1]
Epoch 0:  18%|█▊        | 4/22 [00:01<00:06,  2.57it/s, v_num=1]
Epoch 0:  18%|█▊        | 4/22 [00:01<00:06,  2.57it/s, v_num=1]
Epoch 0:  23%|██▎       | 5/22 [00:01<00:05,  2.94it/s, v_num=1]
Epoch 0:  23%|██▎       | 5/22 [00:01<00:05,  2.94it/s, v_num=1]
Epoch 0:  27%|██▋       | 6/22 [00:01<00:04,  3.24it/s, v_num=1]
Epoch 0:  27%|██▋       | 6/22 [00:01<00:04,  3.24it/s, v_num=1]
Epoch 0:  32%|███▏      | 7/22 [00:01<00:04,  3.50it/s, v_num=1]
Epoch 0:  32%|███▏      | 7/22 [00:01<00:04,  3.50it/s, v_num=1]
Epoch 0:  36%|███▋      | 8/22 [00:02<00:03,  3.74it/s, v_num=1]
Epoch 0:  36%|███▋      | 8/22 [00:02<00:03,  3.74it/s, v_num=1]
Epoch 0:  41%|████      | 9/22 [00:02<00:03,  3.94it/s, v_num=1]
Epoch 0:  41%|████      | 9/22 [00:02<00:03,  3.94it/s, v_num=1]
Epoch 0:  45%|████▌     | 10/22 [00:02<00:02,  4.11it/s, v_num=1]
Epoch 0:  45%|████▌     | 10/22 [00:02<00:02,  4.11it/s, v_num=1]
Epoch 0:  50%|█████     | 11/22 [00:02<00:02,  4.26it/s, v_num=1]
Epoch 0:  50%|█████     | 11/22 [00:02<00:02,  4.26it/s, v_num=1]
Epoch 0:  55%|█████▍    | 12/22 [00:02<00:02,  4.40it/s, v_num=1]
Epoch 0:  55%|█████▍    | 12/22 [00:02<00:02,  4.40it/s, v_num=1]
Epoch 0:  59%|█████▉    | 13/22 [00:02<00:01,  4.51it/s, v_num=1]
Epoch 0:  59%|█████▉    | 13/22 [00:02<00:01,  4.51it/s, v_num=1]
Epoch 0:  64%|██████▎   | 14/22 [00:03<00:01,  4.63it/s, v_num=1]
Epoch 0:  64%|██████▎   | 14/22 [00:03<00:01,  4.63it/s, v_num=1]
Epoch 0:  68%|██████▊   | 15/22 [00:03<00:01,  4.73it/s, v_num=1]
Epoch 0:  68%|██████▊   | 15/22 [00:03<00:01,  4.73it/s, v_num=1]
Epoch 0:  73%|███████▎  | 16/22 [00:03<00:01,  4.83it/s, v_num=1]
Epoch 0:  73%|███████▎  | 16/22 [00:03<00:01,  4.82it/s, v_num=1]
Epoch 0:  77%|███████▋  | 17/22 [00:03<00:01,  4.91it/s, v_num=1]
Epoch 0:  77%|███████▋  | 17/22 [00:03<00:01,  4.91it/s, v_num=1]
Epoch 0:  82%|████████▏ | 18/22 [00:03<00:00,  4.99it/s, v_num=1]
Epoch 0:  82%|████████▏ | 18/22 [00:03<00:00,  4.99it/s, v_num=1]
Epoch 0:  86%|████████▋ | 19/22 [00:03<00:00,  5.06it/s, v_num=1]
Epoch 0:  86%|████████▋ | 19/22 [00:03<00:00,  5.06it/s, v_num=1]
Epoch 0:  91%|█████████ | 20/22 [00:03<00:00,  5.12it/s, v_num=1]
Epoch 0:  91%|█████████ | 20/22 [00:03<00:00,  5.12it/s, v_num=1]
Epoch 0:  95%|█████████▌| 21/22 [00:04<00:00,  5.18it/s, v_num=1]
Epoch 0:  95%|█████████▌| 21/22 [00:04<00:00,  5.18it/s, v_num=1]
Epoch 0: 100%|██████████| 22/22 [00:04<00:00,  5.24it/s, v_num=1]
Epoch 0: 100%|██████████| 22/22 [00:04<00:00,  5.24it/s, v_num=1]
Epoch 0: 100%|██████████| 22/22 [00:04<00:00,  5.24it/s, v_num=1]
Epoch 0:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 1:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 1:   5%|▍         | 1/22 [00:01<00:31,  0.68it/s, v_num=1]
Epoch 1:   5%|▍         | 1/22 [00:01<00:31,  0.68it/s, v_num=1]
Epoch 1:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, v_num=1]
Epoch 1:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, v_num=1]
Epoch 1:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 1:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 1:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 1:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 1:  23%|██▎       | 5/22 [00:02<00:07,  2.42it/s, v_num=1]
Epoch 1:  23%|██▎       | 5/22 [00:02<00:07,  2.42it/s, v_num=1]
Epoch 1:  27%|██▋       | 6/22 [00:02<00:05,  2.71it/s, v_num=1]
Epoch 1:  27%|██▋       | 6/22 [00:02<00:05,  2.71it/s, v_num=1]
Epoch 1:  32%|███▏      | 7/22 [00:02<00:05,  2.96it/s, v_num=1]
Epoch 1:  32%|███▏      | 7/22 [00:02<00:05,  2.96it/s, v_num=1]
Epoch 1:  36%|███▋      | 8/22 [00:02<00:04,  3.18it/s, v_num=1]
Epoch 1:  36%|███▋      | 8/22 [00:02<00:04,  3.18it/s, v_num=1]
Epoch 1:  41%|████      | 9/22 [00:02<00:03,  3.39it/s, v_num=1]
Epoch 1:  41%|████      | 9/22 [00:02<00:03,  3.39it/s, v_num=1]
Epoch 1:  45%|████▌     | 10/22 [00:02<00:03,  3.56it/s, v_num=1]
Epoch 1:  45%|████▌     | 10/22 [00:02<00:03,  3.56it/s, v_num=1]
Epoch 1:  50%|█████     | 11/22 [00:02<00:02,  3.72it/s, v_num=1]
Epoch 1:  50%|█████     | 11/22 [00:02<00:02,  3.72it/s, v_num=1]
Epoch 1:  55%|█████▍    | 12/22 [00:03<00:02,  3.87it/s, v_num=1]
Epoch 1:  55%|█████▍    | 12/22 [00:03<00:02,  3.87it/s, v_num=1]
Epoch 1:  59%|█████▉    | 13/22 [00:03<00:02,  4.00it/s, v_num=1]
Epoch 1:  59%|█████▉    | 13/22 [00:03<00:02,  4.00it/s, v_num=1]
Epoch 1:  64%|██████▎   | 14/22 [00:03<00:01,  4.12it/s, v_num=1]
Epoch 1:  64%|██████▎   | 14/22 [00:03<00:01,  4.12it/s, v_num=1]
Epoch 1:  68%|██████▊   | 15/22 [00:03<00:01,  4.23it/s, v_num=1]
Epoch 1:  68%|██████▊   | 15/22 [00:03<00:01,  4.23it/s, v_num=1]
Epoch 1:  73%|███████▎  | 16/22 [00:03<00:01,  4.34it/s, v_num=1]
Epoch 1:  73%|███████▎  | 16/22 [00:03<00:01,  4.34it/s, v_num=1]
Epoch 1:  77%|███████▋  | 17/22 [00:03<00:01,  4.43it/s, v_num=1]
Epoch 1:  77%|███████▋  | 17/22 [00:03<00:01,  4.43it/s, v_num=1]
Epoch 1:  82%|████████▏ | 18/22 [00:03<00:00,  4.52it/s, v_num=1]
Epoch 1:  82%|████████▏ | 18/22 [00:03<00:00,  4.52it/s, v_num=1]
Epoch 1:  86%|████████▋ | 19/22 [00:04<00:00,  4.60it/s, v_num=1]
Epoch 1:  86%|████████▋ | 19/22 [00:04<00:00,  4.60it/s, v_num=1]
Epoch 1:  91%|█████████ | 20/22 [00:04<00:00,  4.69it/s, v_num=1]
Epoch 1:  91%|█████████ | 20/22 [00:04<00:00,  4.68it/s, v_num=1]
Epoch 1:  95%|█████████▌| 21/22 [00:04<00:00,  4.75it/s, v_num=1]
Epoch 1:  95%|█████████▌| 21/22 [00:04<00:00,  4.75it/s, v_num=1]
Epoch 1: 100%|██████████| 22/22 [00:04<00:00,  4.82it/s, v_num=1]
Epoch 1: 100%|██████████| 22/22 [00:04<00:00,  4.82it/s, v_num=1]
Epoch 1: 100%|██████████| 22/22 [00:04<00:00,  4.82it/s, v_num=1]
Epoch 1:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 2:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 2:   5%|▍         | 1/22 [00:01<00:30,  0.68it/s, v_num=1]
Epoch 2:   5%|▍         | 1/22 [00:01<00:30,  0.68it/s, v_num=1]
Epoch 2:   9%|▉         | 2/22 [00:01<00:16,  1.24it/s, v_num=1]
Epoch 2:   9%|▉         | 2/22 [00:01<00:16,  1.24it/s, v_num=1]
Epoch 2:  14%|█▎        | 3/22 [00:01<00:11,  1.71it/s, v_num=1]
Epoch 2:  14%|█▎        | 3/22 [00:01<00:11,  1.71it/s, v_num=1]
Epoch 2:  18%|█▊        | 4/22 [00:01<00:08,  2.10it/s, v_num=1]
Epoch 2:  18%|█▊        | 4/22 [00:01<00:08,  2.10it/s, v_num=1]
Epoch 2:  23%|██▎       | 5/22 [00:02<00:06,  2.44it/s, v_num=1]
Epoch 2:  23%|██▎       | 5/22 [00:02<00:06,  2.43it/s, v_num=1]
Epoch 2:  27%|██▋       | 6/22 [00:02<00:05,  2.72it/s, v_num=1]
Epoch 2:  27%|██▋       | 6/22 [00:02<00:05,  2.72it/s, v_num=1]
Epoch 2:  32%|███▏      | 7/22 [00:02<00:05,  2.97it/s, v_num=1]
Epoch 2:  32%|███▏      | 7/22 [00:02<00:05,  2.97it/s, v_num=1]
Epoch 2:  36%|███▋      | 8/22 [00:02<00:04,  3.20it/s, v_num=1]
Epoch 2:  36%|███▋      | 8/22 [00:02<00:04,  3.20it/s, v_num=1]
Epoch 2:  41%|████      | 9/22 [00:02<00:03,  3.40it/s, v_num=1]
Epoch 2:  41%|████      | 9/22 [00:02<00:03,  3.40it/s, v_num=1]
Epoch 2:  45%|████▌     | 10/22 [00:02<00:03,  3.58it/s, v_num=1]
Epoch 2:  45%|████▌     | 10/22 [00:02<00:03,  3.58it/s, v_num=1]
Epoch 2:  50%|█████     | 11/22 [00:02<00:02,  3.74it/s, v_num=1]
Epoch 2:  50%|█████     | 11/22 [00:02<00:02,  3.74it/s, v_num=1]
Epoch 2:  55%|█████▍    | 12/22 [00:03<00:02,  3.89it/s, v_num=1]
Epoch 2:  55%|█████▍    | 12/22 [00:03<00:02,  3.89it/s, v_num=1]
Epoch 2:  59%|█████▉    | 13/22 [00:03<00:02,  4.02it/s, v_num=1]
Epoch 2:  59%|█████▉    | 13/22 [00:03<00:02,  4.02it/s, v_num=1]
Epoch 2:  64%|██████▎   | 14/22 [00:03<00:01,  4.15it/s, v_num=1]
Epoch 2:  64%|██████▎   | 14/22 [00:03<00:01,  4.15it/s, v_num=1]
Epoch 2:  68%|██████▊   | 15/22 [00:03<00:01,  4.26it/s, v_num=1]
Epoch 2:  68%|██████▊   | 15/22 [00:03<00:01,  4.26it/s, v_num=1]
Epoch 2:  73%|███████▎  | 16/22 [00:03<00:01,  4.37it/s, v_num=1]
Epoch 2:  73%|███████▎  | 16/22 [00:03<00:01,  4.37it/s, v_num=1]
Epoch 2:  77%|███████▋  | 17/22 [00:03<00:01,  4.47it/s, v_num=1]
Epoch 2:  77%|███████▋  | 17/22 [00:03<00:01,  4.47it/s, v_num=1]
Epoch 2:  82%|████████▏ | 18/22 [00:03<00:00,  4.55it/s, v_num=1]
Epoch 2:  82%|████████▏ | 18/22 [00:03<00:00,  4.55it/s, v_num=1]
Epoch 2:  86%|████████▋ | 19/22 [00:04<00:00,  4.64it/s, v_num=1]
Epoch 2:  86%|████████▋ | 19/22 [00:04<00:00,  4.64it/s, v_num=1]
Epoch 2:  91%|█████████ | 20/22 [00:04<00:00,  4.72it/s, v_num=1]
Epoch 2:  91%|█████████ | 20/22 [00:04<00:00,  4.72it/s, v_num=1]
Epoch 2:  95%|█████████▌| 21/22 [00:04<00:00,  4.79it/s, v_num=1]
Epoch 2:  95%|█████████▌| 21/22 [00:04<00:00,  4.79it/s, v_num=1]
Epoch 2: 100%|██████████| 22/22 [00:04<00:00,  4.86it/s, v_num=1]
Epoch 2: 100%|██████████| 22/22 [00:04<00:00,  4.86it/s, v_num=1]
Epoch 2: 100%|██████████| 22/22 [00:04<00:00,  4.86it/s, v_num=1]
Epoch 2:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 3:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 3:   5%|▍         | 1/22 [00:01<00:31,  0.66it/s, v_num=1]
Epoch 3:   5%|▍         | 1/22 [00:01<00:31,  0.66it/s, v_num=1]
Epoch 3:   9%|▉         | 2/22 [00:01<00:16,  1.20it/s, v_num=1]
Epoch 3:   9%|▉         | 2/22 [00:01<00:16,  1.20it/s, v_num=1]
Epoch 3:  14%|█▎        | 3/22 [00:01<00:11,  1.65it/s, v_num=1]
Epoch 3:  14%|█▎        | 3/22 [00:01<00:11,  1.65it/s, v_num=1]
Epoch 3:  18%|█▊        | 4/22 [00:01<00:08,  2.04it/s, v_num=1]
Epoch 3:  18%|█▊        | 4/22 [00:01<00:08,  2.04it/s, v_num=1]
Epoch 3:  23%|██▎       | 5/22 [00:02<00:07,  2.37it/s, v_num=1]
Epoch 3:  23%|██▎       | 5/22 [00:02<00:07,  2.37it/s, v_num=1]
Epoch 3:  27%|██▋       | 6/22 [00:02<00:06,  2.66it/s, v_num=1]
Epoch 3:  27%|██▋       | 6/22 [00:02<00:06,  2.66it/s, v_num=1]
Epoch 3:  32%|███▏      | 7/22 [00:02<00:05,  2.91it/s, v_num=1]
Epoch 3:  32%|███▏      | 7/22 [00:02<00:05,  2.91it/s, v_num=1]
Epoch 3:  36%|███▋      | 8/22 [00:02<00:04,  3.14it/s, v_num=1]
Epoch 3:  36%|███▋      | 8/22 [00:02<00:04,  3.14it/s, v_num=1]
Epoch 3:  41%|████      | 9/22 [00:02<00:03,  3.34it/s, v_num=1]
Epoch 3:  41%|████      | 9/22 [00:02<00:03,  3.34it/s, v_num=1]
Epoch 3:  45%|████▌     | 10/22 [00:02<00:03,  3.52it/s, v_num=1]
Epoch 3:  45%|████▌     | 10/22 [00:02<00:03,  3.52it/s, v_num=1]
Epoch 3:  50%|█████     | 11/22 [00:02<00:02,  3.68it/s, v_num=1]
Epoch 3:  50%|█████     | 11/22 [00:02<00:02,  3.68it/s, v_num=1]
Epoch 3:  55%|█████▍    | 12/22 [00:03<00:02,  3.83it/s, v_num=1]
Epoch 3:  55%|█████▍    | 12/22 [00:03<00:02,  3.83it/s, v_num=1]
Epoch 3:  59%|█████▉    | 13/22 [00:03<00:02,  3.96it/s, v_num=1]
Epoch 3:  59%|█████▉    | 13/22 [00:03<00:02,  3.96it/s, v_num=1]
Epoch 3:  64%|██████▎   | 14/22 [00:03<00:01,  4.08it/s, v_num=1]
Epoch 3:  64%|██████▎   | 14/22 [00:03<00:01,  4.08it/s, v_num=1]
Epoch 3:  68%|██████▊   | 15/22 [00:03<00:01,  4.20it/s, v_num=1]
Epoch 3:  68%|██████▊   | 15/22 [00:03<00:01,  4.20it/s, v_num=1]
Epoch 3:  73%|███████▎  | 16/22 [00:03<00:01,  4.30it/s, v_num=1]
Epoch 3:  73%|███████▎  | 16/22 [00:03<00:01,  4.30it/s, v_num=1]
Epoch 3:  77%|███████▋  | 17/22 [00:03<00:01,  4.39it/s, v_num=1]
Epoch 3:  77%|███████▋  | 17/22 [00:03<00:01,  4.39it/s, v_num=1]
Epoch 3:  82%|████████▏ | 18/22 [00:04<00:00,  4.48it/s, v_num=1]
Epoch 3:  82%|████████▏ | 18/22 [00:04<00:00,  4.48it/s, v_num=1]
Epoch 3:  86%|████████▋ | 19/22 [00:04<00:00,  4.57it/s, v_num=1]
Epoch 3:  86%|████████▋ | 19/22 [00:04<00:00,  4.57it/s, v_num=1]
Epoch 3:  91%|█████████ | 20/22 [00:04<00:00,  4.65it/s, v_num=1]
Epoch 3:  91%|█████████ | 20/22 [00:04<00:00,  4.64it/s, v_num=1]
Epoch 3:  95%|█████████▌| 21/22 [00:04<00:00,  4.72it/s, v_num=1]
Epoch 3:  95%|█████████▌| 21/22 [00:04<00:00,  4.72it/s, v_num=1]
Epoch 3: 100%|██████████| 22/22 [00:04<00:00,  4.79it/s, v_num=1]
Epoch 3: 100%|██████████| 22/22 [00:04<00:00,  4.79it/s, v_num=1]
Epoch 3: 100%|██████████| 22/22 [00:04<00:00,  4.79it/s, v_num=1]
Epoch 3:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 4:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 4:   5%|▍         | 1/22 [00:01<00:31,  0.67it/s, v_num=1]
Epoch 4:   5%|▍         | 1/22 [00:01<00:31,  0.67it/s, v_num=1]
Epoch 4:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, v_num=1]
Epoch 4:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, v_num=1]
Epoch 4:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 4:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 4:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 4:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 4:  23%|██▎       | 5/22 [00:02<00:07,  2.41it/s, v_num=1]
Epoch 4:  23%|██▎       | 5/22 [00:02<00:07,  2.41it/s, v_num=1]
Epoch 4:  27%|██▋       | 6/22 [00:02<00:05,  2.70it/s, v_num=1]
Epoch 4:  27%|██▋       | 6/22 [00:02<00:05,  2.70it/s, v_num=1]
Epoch 4:  32%|███▏      | 7/22 [00:02<00:05,  2.95it/s, v_num=1]
Epoch 4:  32%|███▏      | 7/22 [00:02<00:05,  2.95it/s, v_num=1]
Epoch 4:  36%|███▋      | 8/22 [00:02<00:04,  3.18it/s, v_num=1]
Epoch 4:  36%|███▋      | 8/22 [00:02<00:04,  3.18it/s, v_num=1]
Epoch 4:  41%|████      | 9/22 [00:02<00:03,  3.38it/s, v_num=1]
Epoch 4:  41%|████      | 9/22 [00:02<00:03,  3.38it/s, v_num=1]
Epoch 4:  45%|████▌     | 10/22 [00:02<00:03,  3.57it/s, v_num=1]
Epoch 4:  45%|████▌     | 10/22 [00:02<00:03,  3.56it/s, v_num=1]
Epoch 4:  50%|█████     | 11/22 [00:02<00:02,  3.73it/s, v_num=1]
Epoch 4:  50%|█████     | 11/22 [00:02<00:02,  3.73it/s, v_num=1]
Epoch 4:  55%|█████▍    | 12/22 [00:03<00:02,  3.87it/s, v_num=1]
Epoch 4:  55%|█████▍    | 12/22 [00:03<00:02,  3.87it/s, v_num=1]
Epoch 4:  59%|█████▉    | 13/22 [00:03<00:02,  4.00it/s, v_num=1]
Epoch 4:  59%|█████▉    | 13/22 [00:03<00:02,  4.00it/s, v_num=1]
Epoch 4:  64%|██████▎   | 14/22 [00:03<00:01,  4.13it/s, v_num=1]
Epoch 4:  64%|██████▎   | 14/22 [00:03<00:01,  4.13it/s, v_num=1]
Epoch 4:  68%|██████▊   | 15/22 [00:03<00:01,  4.24it/s, v_num=1]
Epoch 4:  68%|██████▊   | 15/22 [00:03<00:01,  4.24it/s, v_num=1]
Epoch 4:  73%|███████▎  | 16/22 [00:03<00:01,  4.34it/s, v_num=1]
Epoch 4:  73%|███████▎  | 16/22 [00:03<00:01,  4.34it/s, v_num=1]
Epoch 4:  77%|███████▋  | 17/22 [00:03<00:01,  4.44it/s, v_num=1]
Epoch 4:  77%|███████▋  | 17/22 [00:03<00:01,  4.44it/s, v_num=1]
Epoch 4:  82%|████████▏ | 18/22 [00:03<00:00,  4.53it/s, v_num=1]
Epoch 4:  82%|████████▏ | 18/22 [00:03<00:00,  4.53it/s, v_num=1]
Epoch 4:  86%|████████▋ | 19/22 [00:04<00:00,  4.61it/s, v_num=1]
Epoch 4:  86%|████████▋ | 19/22 [00:04<00:00,  4.61it/s, v_num=1]
Epoch 4:  91%|█████████ | 20/22 [00:04<00:00,  4.69it/s, v_num=1]
Epoch 4:  91%|█████████ | 20/22 [00:04<00:00,  4.69it/s, v_num=1]
Epoch 4:  95%|█████████▌| 21/22 [00:04<00:00,  4.76it/s, v_num=1]
Epoch 4:  95%|█████████▌| 21/22 [00:04<00:00,  4.76it/s, v_num=1]
Epoch 4: 100%|██████████| 22/22 [00:04<00:00,  4.83it/s, v_num=1]
Epoch 4: 100%|██████████| 22/22 [00:04<00:00,  4.83it/s, v_num=1]
Epoch 4: 100%|██████████| 22/22 [00:04<00:00,  4.83it/s, v_num=1]
Epoch 4:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 5:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 5:   5%|▍         | 1/22 [00:01<00:32,  0.65it/s, v_num=1]
Epoch 5:   5%|▍         | 1/22 [00:01<00:32,  0.65it/s, v_num=1]
Epoch 5:   9%|▉         | 2/22 [00:01<00:16,  1.19it/s, v_num=1]
Epoch 5:   9%|▉         | 2/22 [00:01<00:16,  1.19it/s, v_num=1]
Epoch 5:  14%|█▎        | 3/22 [00:01<00:11,  1.64it/s, v_num=1]
Epoch 5:  14%|█▎        | 3/22 [00:01<00:11,  1.64it/s, v_num=1]
Epoch 5:  18%|█▊        | 4/22 [00:01<00:08,  2.03it/s, v_num=1]
Epoch 5:  18%|█▊        | 4/22 [00:01<00:08,  2.03it/s, v_num=1]
Epoch 5:  23%|██▎       | 5/22 [00:02<00:07,  2.36it/s, v_num=1]
Epoch 5:  23%|██▎       | 5/22 [00:02<00:07,  2.36it/s, v_num=1]
Epoch 5:  27%|██▋       | 6/22 [00:02<00:06,  2.64it/s, v_num=1]
Epoch 5:  27%|██▋       | 6/22 [00:02<00:06,  2.64it/s, v_num=1]
Epoch 5:  32%|███▏      | 7/22 [00:02<00:05,  2.89it/s, v_num=1]
Epoch 5:  32%|███▏      | 7/22 [00:02<00:05,  2.89it/s, v_num=1]
Epoch 5:  36%|███▋      | 8/22 [00:02<00:04,  3.12it/s, v_num=1]
Epoch 5:  36%|███▋      | 8/22 [00:02<00:04,  3.12it/s, v_num=1]
Epoch 5:  41%|████      | 9/22 [00:02<00:03,  3.32it/s, v_num=1]
Epoch 5:  41%|████      | 9/22 [00:02<00:03,  3.32it/s, v_num=1]
Epoch 5:  45%|████▌     | 10/22 [00:02<00:03,  3.50it/s, v_num=1]
Epoch 5:  45%|████▌     | 10/22 [00:02<00:03,  3.50it/s, v_num=1]
Epoch 5:  50%|█████     | 11/22 [00:02<00:02,  3.67it/s, v_num=1]
Epoch 5:  50%|█████     | 11/22 [00:02<00:02,  3.67it/s, v_num=1]
Epoch 5:  55%|█████▍    | 12/22 [00:03<00:02,  3.82it/s, v_num=1]
Epoch 5:  55%|█████▍    | 12/22 [00:03<00:02,  3.81it/s, v_num=1]
Epoch 5:  59%|█████▉    | 13/22 [00:03<00:02,  3.95it/s, v_num=1]
Epoch 5:  59%|█████▉    | 13/22 [00:03<00:02,  3.95it/s, v_num=1]
Epoch 5:  64%|██████▎   | 14/22 [00:03<00:01,  4.07it/s, v_num=1]
Epoch 5:  64%|██████▎   | 14/22 [00:03<00:01,  4.07it/s, v_num=1]
Epoch 5:  68%|██████▊   | 15/22 [00:03<00:01,  4.18it/s, v_num=1]
Epoch 5:  68%|██████▊   | 15/22 [00:03<00:01,  4.18it/s, v_num=1]
Epoch 5:  73%|███████▎  | 16/22 [00:03<00:01,  4.29it/s, v_num=1]
Epoch 5:  73%|███████▎  | 16/22 [00:03<00:01,  4.29it/s, v_num=1]
Epoch 5:  77%|███████▋  | 17/22 [00:03<00:01,  4.39it/s, v_num=1]
Epoch 5:  77%|███████▋  | 17/22 [00:03<00:01,  4.39it/s, v_num=1]
Epoch 5:  82%|████████▏ | 18/22 [00:04<00:00,  4.47it/s, v_num=1]
Epoch 5:  82%|████████▏ | 18/22 [00:04<00:00,  4.47it/s, v_num=1]
Epoch 5:  86%|████████▋ | 19/22 [00:04<00:00,  4.56it/s, v_num=1]
Epoch 5:  86%|████████▋ | 19/22 [00:04<00:00,  4.56it/s, v_num=1]
Epoch 5:  91%|█████████ | 20/22 [00:04<00:00,  4.64it/s, v_num=1]
Epoch 5:  91%|█████████ | 20/22 [00:04<00:00,  4.64it/s, v_num=1]
Epoch 5:  95%|█████████▌| 21/22 [00:04<00:00,  4.71it/s, v_num=1]
Epoch 5:  95%|█████████▌| 21/22 [00:04<00:00,  4.71it/s, v_num=1]
Epoch 5: 100%|██████████| 22/22 [00:04<00:00,  4.78it/s, v_num=1]
Epoch 5: 100%|██████████| 22/22 [00:04<00:00,  4.78it/s, v_num=1]
Epoch 5: 100%|██████████| 22/22 [00:04<00:00,  4.78it/s, v_num=1]
Epoch 5:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 6:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 6:   5%|▍         | 1/22 [00:01<00:31,  0.68it/s, v_num=1]
Epoch 6:   5%|▍         | 1/22 [00:01<00:31,  0.68it/s, v_num=1]
Epoch 6:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, v_num=1]
Epoch 6:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, v_num=1]
Epoch 6:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 6:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 6:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 6:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 6:  23%|██▎       | 5/22 [00:02<00:07,  2.41it/s, v_num=1]
Epoch 6:  23%|██▎       | 5/22 [00:02<00:07,  2.41it/s, v_num=1]
Epoch 6:  27%|██▋       | 6/22 [00:02<00:05,  2.70it/s, v_num=1]
Epoch 6:  27%|██▋       | 6/22 [00:02<00:05,  2.70it/s, v_num=1]
Epoch 6:  32%|███▏      | 7/22 [00:02<00:05,  2.96it/s, v_num=1]
Epoch 6:  32%|███▏      | 7/22 [00:02<00:05,  2.96it/s, v_num=1]
Epoch 6:  36%|███▋      | 8/22 [00:02<00:04,  3.18it/s, v_num=1]
Epoch 6:  36%|███▋      | 8/22 [00:02<00:04,  3.18it/s, v_num=1]
Epoch 6:  41%|████      | 9/22 [00:02<00:03,  3.38it/s, v_num=1]
Epoch 6:  41%|████      | 9/22 [00:02<00:03,  3.38it/s, v_num=1]
Epoch 6:  45%|████▌     | 10/22 [00:02<00:03,  3.56it/s, v_num=1]
Epoch 6:  45%|████▌     | 10/22 [00:02<00:03,  3.56it/s, v_num=1]
Epoch 6:  50%|█████     | 11/22 [00:02<00:02,  3.72it/s, v_num=1]
Epoch 6:  50%|█████     | 11/22 [00:02<00:02,  3.72it/s, v_num=1]
Epoch 6:  55%|█████▍    | 12/22 [00:03<00:02,  3.87it/s, v_num=1]
Epoch 6:  55%|█████▍    | 12/22 [00:03<00:02,  3.87it/s, v_num=1]
Epoch 6:  59%|█████▉    | 13/22 [00:03<00:02,  4.00it/s, v_num=1]
Epoch 6:  59%|█████▉    | 13/22 [00:03<00:02,  4.00it/s, v_num=1]
Epoch 6:  64%|██████▎   | 14/22 [00:03<00:01,  4.12it/s, v_num=1]
Epoch 6:  64%|██████▎   | 14/22 [00:03<00:01,  4.12it/s, v_num=1]
Epoch 6:  68%|██████▊   | 15/22 [00:03<00:01,  4.23it/s, v_num=1]
Epoch 6:  68%|██████▊   | 15/22 [00:03<00:01,  4.23it/s, v_num=1]
Epoch 6:  73%|███████▎  | 16/22 [00:03<00:01,  4.33it/s, v_num=1]
Epoch 6:  73%|███████▎  | 16/22 [00:03<00:01,  4.33it/s, v_num=1]
Epoch 6:  77%|███████▋  | 17/22 [00:03<00:01,  4.42it/s, v_num=1]
Epoch 6:  77%|███████▋  | 17/22 [00:03<00:01,  4.42it/s, v_num=1]
Epoch 6:  82%|████████▏ | 18/22 [00:03<00:00,  4.51it/s, v_num=1]
Epoch 6:  82%|████████▏ | 18/22 [00:03<00:00,  4.51it/s, v_num=1]
Epoch 6:  86%|████████▋ | 19/22 [00:04<00:00,  4.59it/s, v_num=1]
Epoch 6:  86%|████████▋ | 19/22 [00:04<00:00,  4.59it/s, v_num=1]
Epoch 6:  91%|█████████ | 20/22 [00:04<00:00,  4.67it/s, v_num=1]
Epoch 6:  91%|█████████ | 20/22 [00:04<00:00,  4.67it/s, v_num=1]
Epoch 6:  95%|█████████▌| 21/22 [00:04<00:00,  4.74it/s, v_num=1]
Epoch 6:  95%|█████████▌| 21/22 [00:04<00:00,  4.74it/s, v_num=1]
Epoch 6: 100%|██████████| 22/22 [00:04<00:00,  4.80it/s, v_num=1]
Epoch 6: 100%|██████████| 22/22 [00:04<00:00,  4.80it/s, v_num=1]
Epoch 6: 100%|██████████| 22/22 [00:04<00:00,  4.80it/s, v_num=1]
Epoch 6:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 7:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 7:   5%|▍         | 1/22 [00:01<00:31,  0.66it/s, v_num=1]
Epoch 7:   5%|▍         | 1/22 [00:01<00:32,  0.66it/s, v_num=1]
Epoch 7:   9%|▉         | 2/22 [00:01<00:16,  1.20it/s, v_num=1]
Epoch 7:   9%|▉         | 2/22 [00:01<00:16,  1.20it/s, v_num=1]
Epoch 7:  14%|█▎        | 3/22 [00:01<00:11,  1.65it/s, v_num=1]
Epoch 7:  14%|█▎        | 3/22 [00:01<00:11,  1.65it/s, v_num=1]
Epoch 7:  18%|█▊        | 4/22 [00:01<00:08,  2.03it/s, v_num=1]
Epoch 7:  18%|█▊        | 4/22 [00:01<00:08,  2.03it/s, v_num=1]
Epoch 7:  23%|██▎       | 5/22 [00:02<00:07,  2.37it/s, v_num=1]
Epoch 7:  23%|██▎       | 5/22 [00:02<00:07,  2.37it/s, v_num=1]
Epoch 7:  27%|██▋       | 6/22 [00:02<00:06,  2.65it/s, v_num=1]
Epoch 7:  27%|██▋       | 6/22 [00:02<00:06,  2.65it/s, v_num=1]
Epoch 7:  32%|███▏      | 7/22 [00:02<00:05,  2.90it/s, v_num=1]
Epoch 7:  32%|███▏      | 7/22 [00:02<00:05,  2.90it/s, v_num=1]
Epoch 7:  36%|███▋      | 8/22 [00:02<00:04,  3.13it/s, v_num=1]
Epoch 7:  36%|███▋      | 8/22 [00:02<00:04,  3.13it/s, v_num=1]
Epoch 7:  41%|████      | 9/22 [00:02<00:03,  3.33it/s, v_num=1]
Epoch 7:  41%|████      | 9/22 [00:02<00:03,  3.33it/s, v_num=1]
Epoch 7:  45%|████▌     | 10/22 [00:02<00:03,  3.51it/s, v_num=1]
Epoch 7:  45%|████▌     | 10/22 [00:02<00:03,  3.51it/s, v_num=1]
Epoch 7:  50%|█████     | 11/22 [00:02<00:02,  3.67it/s, v_num=1]
Epoch 7:  50%|█████     | 11/22 [00:02<00:02,  3.67it/s, v_num=1]
Epoch 7:  55%|█████▍    | 12/22 [00:03<00:02,  3.81it/s, v_num=1]
Epoch 7:  55%|█████▍    | 12/22 [00:03<00:02,  3.81it/s, v_num=1]
Epoch 7:  59%|█████▉    | 13/22 [00:03<00:02,  3.95it/s, v_num=1]
Epoch 7:  59%|█████▉    | 13/22 [00:03<00:02,  3.95it/s, v_num=1]
Epoch 7:  64%|██████▎   | 14/22 [00:03<00:01,  4.07it/s, v_num=1]
Epoch 7:  64%|██████▎   | 14/22 [00:03<00:01,  4.07it/s, v_num=1]
Epoch 7:  68%|██████▊   | 15/22 [00:03<00:01,  4.19it/s, v_num=1]
Epoch 7:  68%|██████▊   | 15/22 [00:03<00:01,  4.19it/s, v_num=1]
Epoch 7:  73%|███████▎  | 16/22 [00:03<00:01,  4.29it/s, v_num=1]
Epoch 7:  73%|███████▎  | 16/22 [00:03<00:01,  4.29it/s, v_num=1]
Epoch 7:  77%|███████▋  | 17/22 [00:03<00:01,  4.39it/s, v_num=1]
Epoch 7:  77%|███████▋  | 17/22 [00:03<00:01,  4.39it/s, v_num=1]
Epoch 7:  82%|████████▏ | 18/22 [00:04<00:00,  4.48it/s, v_num=1]
Epoch 7:  82%|████████▏ | 18/22 [00:04<00:00,  4.48it/s, v_num=1]
Epoch 7:  86%|████████▋ | 19/22 [00:04<00:00,  4.56it/s, v_num=1]
Epoch 7:  86%|████████▋ | 19/22 [00:04<00:00,  4.56it/s, v_num=1]
Epoch 7:  91%|█████████ | 20/22 [00:04<00:00,  4.64it/s, v_num=1]
Epoch 7:  91%|█████████ | 20/22 [00:04<00:00,  4.64it/s, v_num=1]
Epoch 7:  95%|█████████▌| 21/22 [00:04<00:00,  4.71it/s, v_num=1]
Epoch 7:  95%|█████████▌| 21/22 [00:04<00:00,  4.71it/s, v_num=1]
Epoch 7: 100%|██████████| 22/22 [00:04<00:00,  4.78it/s, v_num=1]
Epoch 7: 100%|██████████| 22/22 [00:04<00:00,  4.78it/s, v_num=1]
Epoch 7: 100%|██████████| 22/22 [00:04<00:00,  4.78it/s, v_num=1]
Epoch 7:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 8:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 8:   5%|▍         | 1/22 [00:01<00:30,  0.68it/s, v_num=1]
Epoch 8:   5%|▍         | 1/22 [00:01<00:30,  0.68it/s, v_num=1]
Epoch 8:   9%|▉         | 2/22 [00:01<00:16,  1.24it/s, v_num=1]
Epoch 8:   9%|▉         | 2/22 [00:01<00:16,  1.24it/s, v_num=1]
Epoch 8:  14%|█▎        | 3/22 [00:01<00:11,  1.70it/s, v_num=1]
Epoch 8:  14%|█▎        | 3/22 [00:01<00:11,  1.70it/s, v_num=1]
Epoch 8:  18%|█▊        | 4/22 [00:01<00:08,  2.09it/s, v_num=1]
Epoch 8:  18%|█▊        | 4/22 [00:01<00:08,  2.09it/s, v_num=1]
Epoch 8:  23%|██▎       | 5/22 [00:02<00:06,  2.43it/s, v_num=1]
Epoch 8:  23%|██▎       | 5/22 [00:02<00:06,  2.43it/s, v_num=1]
Epoch 8:  27%|██▋       | 6/22 [00:02<00:05,  2.72it/s, v_num=1]
Epoch 8:  27%|██▋       | 6/22 [00:02<00:05,  2.72it/s, v_num=1]
Epoch 8:  32%|███▏      | 7/22 [00:02<00:05,  2.97it/s, v_num=1]
Epoch 8:  32%|███▏      | 7/22 [00:02<00:05,  2.97it/s, v_num=1]
Epoch 8:  36%|███▋      | 8/22 [00:02<00:04,  3.20it/s, v_num=1]
Epoch 8:  36%|███▋      | 8/22 [00:02<00:04,  3.20it/s, v_num=1]
Epoch 8:  41%|████      | 9/22 [00:02<00:03,  3.40it/s, v_num=1]
Epoch 8:  41%|████      | 9/22 [00:02<00:03,  3.40it/s, v_num=1]
Epoch 8:  45%|████▌     | 10/22 [00:02<00:03,  3.58it/s, v_num=1]
Epoch 8:  45%|████▌     | 10/22 [00:02<00:03,  3.58it/s, v_num=1]
Epoch 8:  50%|█████     | 11/22 [00:02<00:02,  3.74it/s, v_num=1]
Epoch 8:  50%|█████     | 11/22 [00:02<00:02,  3.74it/s, v_num=1]
Epoch 8:  55%|█████▍    | 12/22 [00:03<00:02,  3.89it/s, v_num=1]
Epoch 8:  55%|█████▍    | 12/22 [00:03<00:02,  3.89it/s, v_num=1]
Epoch 8:  59%|█████▉    | 13/22 [00:03<00:02,  4.02it/s, v_num=1]
Epoch 8:  59%|█████▉    | 13/22 [00:03<00:02,  4.02it/s, v_num=1]
Epoch 8:  64%|██████▎   | 14/22 [00:03<00:01,  4.14it/s, v_num=1]
Epoch 8:  64%|██████▎   | 14/22 [00:03<00:01,  4.14it/s, v_num=1]
Epoch 8:  68%|██████▊   | 15/22 [00:03<00:01,  4.25it/s, v_num=1]
Epoch 8:  68%|██████▊   | 15/22 [00:03<00:01,  4.25it/s, v_num=1]
Epoch 8:  73%|███████▎  | 16/22 [00:03<00:01,  4.36it/s, v_num=1]
Epoch 8:  73%|███████▎  | 16/22 [00:03<00:01,  4.36it/s, v_num=1]
Epoch 8:  77%|███████▋  | 17/22 [00:03<00:01,  4.45it/s, v_num=1]
Epoch 8:  77%|███████▋  | 17/22 [00:03<00:01,  4.45it/s, v_num=1]
Epoch 8:  82%|████████▏ | 18/22 [00:03<00:00,  4.54it/s, v_num=1]
Epoch 8:  82%|████████▏ | 18/22 [00:03<00:00,  4.54it/s, v_num=1]
Epoch 8:  86%|████████▋ | 19/22 [00:04<00:00,  4.63it/s, v_num=1]
Epoch 8:  86%|████████▋ | 19/22 [00:04<00:00,  4.63it/s, v_num=1]
Epoch 8:  91%|█████████ | 20/22 [00:04<00:00,  4.70it/s, v_num=1]
Epoch 8:  91%|█████████ | 20/22 [00:04<00:00,  4.70it/s, v_num=1]
Epoch 8:  95%|█████████▌| 21/22 [00:04<00:00,  4.77it/s, v_num=1]
Epoch 8:  95%|█████████▌| 21/22 [00:04<00:00,  4.77it/s, v_num=1]
Epoch 8: 100%|██████████| 22/22 [00:04<00:00,  4.84it/s, v_num=1]
Epoch 8: 100%|██████████| 22/22 [00:04<00:00,  4.84it/s, v_num=1]
Epoch 8: 100%|██████████| 22/22 [00:04<00:00,  4.83it/s, v_num=1]
Epoch 8:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 9:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 9:   5%|▍         | 1/22 [00:01<00:30,  0.68it/s, v_num=1]
Epoch 9:   5%|▍         | 1/22 [00:01<00:30,  0.68it/s, v_num=1]
Epoch 9:   9%|▉         | 2/22 [00:01<00:16,  1.24it/s, v_num=1]
Epoch 9:   9%|▉         | 2/22 [00:01<00:16,  1.24it/s, v_num=1]
Epoch 9:  14%|█▎        | 3/22 [00:01<00:11,  1.70it/s, v_num=1]
Epoch 9:  14%|█▎        | 3/22 [00:01<00:11,  1.70it/s, v_num=1]
Epoch 9:  18%|█▊        | 4/22 [00:01<00:08,  2.09it/s, v_num=1]
Epoch 9:  18%|█▊        | 4/22 [00:01<00:08,  2.09it/s, v_num=1]
Epoch 9:  23%|██▎       | 5/22 [00:02<00:06,  2.43it/s, v_num=1]
Epoch 9:  23%|██▎       | 5/22 [00:02<00:06,  2.43it/s, v_num=1]
Epoch 9:  27%|██▋       | 6/22 [00:02<00:05,  2.72it/s, v_num=1]
Epoch 9:  27%|██▋       | 6/22 [00:02<00:05,  2.72it/s, v_num=1]
Epoch 9:  32%|███▏      | 7/22 [00:02<00:05,  2.97it/s, v_num=1]
Epoch 9:  32%|███▏      | 7/22 [00:02<00:05,  2.97it/s, v_num=1]
Epoch 9:  36%|███▋      | 8/22 [00:02<00:04,  3.20it/s, v_num=1]
Epoch 9:  36%|███▋      | 8/22 [00:02<00:04,  3.20it/s, v_num=1]
Epoch 9:  41%|████      | 9/22 [00:02<00:03,  3.40it/s, v_num=1]
Epoch 9:  41%|████      | 9/22 [00:02<00:03,  3.40it/s, v_num=1]
Epoch 9:  45%|████▌     | 10/22 [00:02<00:03,  3.57it/s, v_num=1]
Epoch 9:  45%|████▌     | 10/22 [00:02<00:03,  3.57it/s, v_num=1]
Epoch 9:  50%|█████     | 11/22 [00:02<00:02,  3.73it/s, v_num=1]
Epoch 9:  50%|█████     | 11/22 [00:02<00:02,  3.73it/s, v_num=1]
Epoch 9:  55%|█████▍    | 12/22 [00:03<00:02,  3.88it/s, v_num=1]
Epoch 9:  55%|█████▍    | 12/22 [00:03<00:02,  3.88it/s, v_num=1]
Epoch 9:  59%|█████▉    | 13/22 [00:03<00:02,  4.01it/s, v_num=1]
Epoch 9:  59%|█████▉    | 13/22 [00:03<00:02,  4.01it/s, v_num=1]
Epoch 9:  64%|██████▎   | 14/22 [00:03<00:01,  4.14it/s, v_num=1]
Epoch 9:  64%|██████▎   | 14/22 [00:03<00:01,  4.14it/s, v_num=1]
Epoch 9:  68%|██████▊   | 15/22 [00:03<00:01,  4.25it/s, v_num=1]
Epoch 9:  68%|██████▊   | 15/22 [00:03<00:01,  4.25it/s, v_num=1]
Epoch 9:  73%|███████▎  | 16/22 [00:03<00:01,  4.35it/s, v_num=1]
Epoch 9:  73%|███████▎  | 16/22 [00:03<00:01,  4.35it/s, v_num=1]
Epoch 9:  77%|███████▋  | 17/22 [00:03<00:01,  4.45it/s, v_num=1]
Epoch 9:  77%|███████▋  | 17/22 [00:03<00:01,  4.45it/s, v_num=1]
Epoch 9:  82%|████████▏ | 18/22 [00:03<00:00,  4.53it/s, v_num=1]
Epoch 9:  82%|████████▏ | 18/22 [00:03<00:00,  4.53it/s, v_num=1]
Epoch 9:  86%|████████▋ | 19/22 [00:04<00:00,  4.61it/s, v_num=1]
Epoch 9:  86%|████████▋ | 19/22 [00:04<00:00,  4.61it/s, v_num=1]
Epoch 9:  91%|█████████ | 20/22 [00:04<00:00,  4.69it/s, v_num=1]
Epoch 9:  91%|█████████ | 20/22 [00:04<00:00,  4.69it/s, v_num=1]
Epoch 9:  95%|█████████▌| 21/22 [00:04<00:00,  4.76it/s, v_num=1]
Epoch 9:  95%|█████████▌| 21/22 [00:04<00:00,  4.76it/s, v_num=1]
Epoch 9: 100%|██████████| 22/22 [00:04<00:00,  4.83it/s, v_num=1]
Epoch 9: 100%|██████████| 22/22 [00:04<00:00,  4.83it/s, v_num=1]
Epoch 9: 100%|██████████| 22/22 [00:04<00:00,  4.83it/s, v_num=1]
Epoch 9:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 10:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 10:   5%|▍         | 1/22 [00:01<00:31,  0.68it/s, v_num=1]
Epoch 10:   5%|▍         | 1/22 [00:01<00:31,  0.67it/s, v_num=1]
Epoch 10:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, v_num=1]
Epoch 10:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, v_num=1]
Epoch 10:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 10:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 10:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 10:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 10:  23%|██▎       | 5/22 [00:02<00:07,  2.42it/s, v_num=1]
Epoch 10:  23%|██▎       | 5/22 [00:02<00:07,  2.42it/s, v_num=1]
Epoch 10:  27%|██▋       | 6/22 [00:02<00:05,  2.70it/s, v_num=1]
Epoch 10:  27%|██▋       | 6/22 [00:02<00:05,  2.70it/s, v_num=1]
Epoch 10:  32%|███▏      | 7/22 [00:02<00:05,  2.96it/s, v_num=1]
Epoch 10:  32%|███▏      | 7/22 [00:02<00:05,  2.96it/s, v_num=1]
Epoch 10:  36%|███▋      | 8/22 [00:02<00:04,  3.18it/s, v_num=1]
Epoch 10:  36%|███▋      | 8/22 [00:02<00:04,  3.18it/s, v_num=1]
Epoch 10:  41%|████      | 9/22 [00:02<00:03,  3.38it/s, v_num=1]
Epoch 10:  41%|████      | 9/22 [00:02<00:03,  3.38it/s, v_num=1]
Epoch 10:  45%|████▌     | 10/22 [00:02<00:03,  3.56it/s, v_num=1]
Epoch 10:  45%|████▌     | 10/22 [00:02<00:03,  3.56it/s, v_num=1]
Epoch 10:  50%|█████     | 11/22 [00:02<00:02,  3.72it/s, v_num=1]
Epoch 10:  50%|█████     | 11/22 [00:02<00:02,  3.72it/s, v_num=1]
Epoch 10:  55%|█████▍    | 12/22 [00:03<00:02,  3.87it/s, v_num=1]
Epoch 10:  55%|█████▍    | 12/22 [00:03<00:02,  3.87it/s, v_num=1]
Epoch 10:  59%|█████▉    | 13/22 [00:03<00:02,  4.00it/s, v_num=1]
Epoch 10:  59%|█████▉    | 13/22 [00:03<00:02,  4.00it/s, v_num=1]
Epoch 10:  64%|██████▎   | 14/22 [00:03<00:01,  4.12it/s, v_num=1]
Epoch 10:  64%|██████▎   | 14/22 [00:03<00:01,  4.12it/s, v_num=1]
Epoch 10:  68%|██████▊   | 15/22 [00:03<00:01,  4.23it/s, v_num=1]
Epoch 10:  68%|██████▊   | 15/22 [00:03<00:01,  4.23it/s, v_num=1]
Epoch 10:  73%|███████▎  | 16/22 [00:03<00:01,  4.34it/s, v_num=1]
Epoch 10:  73%|███████▎  | 16/22 [00:03<00:01,  4.34it/s, v_num=1]
Epoch 10:  77%|███████▋  | 17/22 [00:03<00:01,  4.43it/s, v_num=1]
Epoch 10:  77%|███████▋  | 17/22 [00:03<00:01,  4.43it/s, v_num=1]
Epoch 10:  82%|████████▏ | 18/22 [00:03<00:00,  4.52it/s, v_num=1]
Epoch 10:  82%|████████▏ | 18/22 [00:03<00:00,  4.52it/s, v_num=1]
Epoch 10:  86%|████████▋ | 19/22 [00:04<00:00,  4.61it/s, v_num=1]
Epoch 10:  86%|████████▋ | 19/22 [00:04<00:00,  4.61it/s, v_num=1]
Epoch 10:  91%|█████████ | 20/22 [00:04<00:00,  4.68it/s, v_num=1]
Epoch 10:  91%|█████████ | 20/22 [00:04<00:00,  4.68it/s, v_num=1]
Epoch 10:  95%|█████████▌| 21/22 [00:04<00:00,  4.75it/s, v_num=1]
Epoch 10:  95%|█████████▌| 21/22 [00:04<00:00,  4.75it/s, v_num=1]
Epoch 10: 100%|██████████| 22/22 [00:04<00:00,  4.82it/s, v_num=1]
Epoch 10: 100%|██████████| 22/22 [00:04<00:00,  4.82it/s, v_num=1]
Epoch 10: 100%|██████████| 22/22 [00:04<00:00,  4.82it/s, v_num=1]
Epoch 10:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 11:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 11:   5%|▍         | 1/22 [00:01<00:31,  0.67it/s, v_num=1]
Epoch 11:   5%|▍         | 1/22 [00:01<00:31,  0.67it/s, v_num=1]
Epoch 11:   9%|▉         | 2/22 [00:01<00:16,  1.22it/s, v_num=1]
Epoch 11:   9%|▉         | 2/22 [00:01<00:16,  1.22it/s, v_num=1]
Epoch 11:  14%|█▎        | 3/22 [00:01<00:11,  1.68it/s, v_num=1]
Epoch 11:  14%|█▎        | 3/22 [00:01<00:11,  1.68it/s, v_num=1]
Epoch 11:  18%|█▊        | 4/22 [00:01<00:08,  2.07it/s, v_num=1]
Epoch 11:  18%|█▊        | 4/22 [00:01<00:08,  2.07it/s, v_num=1]
Epoch 11:  23%|██▎       | 5/22 [00:02<00:07,  2.41it/s, v_num=1]
Epoch 11:  23%|██▎       | 5/22 [00:02<00:07,  2.41it/s, v_num=1]
Epoch 11:  27%|██▋       | 6/22 [00:02<00:05,  2.70it/s, v_num=1]
Epoch 11:  27%|██▋       | 6/22 [00:02<00:05,  2.70it/s, v_num=1]
Epoch 11:  32%|███▏      | 7/22 [00:02<00:05,  2.94it/s, v_num=1]
Epoch 11:  32%|███▏      | 7/22 [00:02<00:05,  2.94it/s, v_num=1]
Epoch 11:  36%|███▋      | 8/22 [00:02<00:04,  3.17it/s, v_num=1]
Epoch 11:  36%|███▋      | 8/22 [00:02<00:04,  3.17it/s, v_num=1]
Epoch 11:  41%|████      | 9/22 [00:02<00:03,  3.36it/s, v_num=1]
Epoch 11:  41%|████      | 9/22 [00:02<00:03,  3.36it/s, v_num=1]
Epoch 11:  45%|████▌     | 10/22 [00:02<00:03,  3.54it/s, v_num=1]
Epoch 11:  45%|████▌     | 10/22 [00:02<00:03,  3.54it/s, v_num=1]
Epoch 11:  50%|█████     | 11/22 [00:02<00:02,  3.70it/s, v_num=1]
Epoch 11:  50%|█████     | 11/22 [00:02<00:02,  3.70it/s, v_num=1]
Epoch 11:  55%|█████▍    | 12/22 [00:03<00:02,  3.85it/s, v_num=1]
Epoch 11:  55%|█████▍    | 12/22 [00:03<00:02,  3.85it/s, v_num=1]
Epoch 11:  59%|█████▉    | 13/22 [00:03<00:02,  3.98it/s, v_num=1]
Epoch 11:  59%|█████▉    | 13/22 [00:03<00:02,  3.97it/s, v_num=1]
Epoch 11:  64%|██████▎   | 14/22 [00:03<00:01,  4.10it/s, v_num=1]
Epoch 11:  64%|██████▎   | 14/22 [00:03<00:01,  4.10it/s, v_num=1]
Epoch 11:  68%|██████▊   | 15/22 [00:03<00:01,  4.21it/s, v_num=1]
Epoch 11:  68%|██████▊   | 15/22 [00:03<00:01,  4.21it/s, v_num=1]
Epoch 11:  73%|███████▎  | 16/22 [00:03<00:01,  4.31it/s, v_num=1]
Epoch 11:  73%|███████▎  | 16/22 [00:03<00:01,  4.31it/s, v_num=1]
Epoch 11:  77%|███████▋  | 17/22 [00:03<00:01,  4.41it/s, v_num=1]
Epoch 11:  77%|███████▋  | 17/22 [00:03<00:01,  4.41it/s, v_num=1]
Epoch 11:  82%|████████▏ | 18/22 [00:04<00:00,  4.50it/s, v_num=1]
Epoch 11:  82%|████████▏ | 18/22 [00:04<00:00,  4.50it/s, v_num=1]
Epoch 11:  86%|████████▋ | 19/22 [00:04<00:00,  4.58it/s, v_num=1]
Epoch 11:  86%|████████▋ | 19/22 [00:04<00:00,  4.58it/s, v_num=1]
Epoch 11:  91%|█████████ | 20/22 [00:04<00:00,  4.66it/s, v_num=1]
Epoch 11:  91%|█████████ | 20/22 [00:04<00:00,  4.66it/s, v_num=1]
Epoch 11:  95%|█████████▌| 21/22 [00:04<00:00,  4.73it/s, v_num=1]
Epoch 11:  95%|█████████▌| 21/22 [00:04<00:00,  4.73it/s, v_num=1]
Epoch 11: 100%|██████████| 22/22 [00:04<00:00,  4.80it/s, v_num=1]
Epoch 11: 100%|██████████| 22/22 [00:04<00:00,  4.80it/s, v_num=1]
Epoch 11: 100%|██████████| 22/22 [00:04<00:00,  4.80it/s, v_num=1]
Epoch 11:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 12:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 12:   5%|▍         | 1/22 [00:01<00:32,  0.65it/s, v_num=1]
Epoch 12:   5%|▍         | 1/22 [00:01<00:32,  0.65it/s, v_num=1]
Epoch 12:   9%|▉         | 2/22 [00:01<00:16,  1.19it/s, v_num=1]
Epoch 12:   9%|▉         | 2/22 [00:01<00:16,  1.19it/s, v_num=1]
Epoch 12:  14%|█▎        | 3/22 [00:01<00:11,  1.64it/s, v_num=1]
Epoch 12:  14%|█▎        | 3/22 [00:01<00:11,  1.64it/s, v_num=1]
Epoch 12:  18%|█▊        | 4/22 [00:01<00:08,  2.02it/s, v_num=1]
Epoch 12:  18%|█▊        | 4/22 [00:01<00:08,  2.02it/s, v_num=1]
Epoch 12:  23%|██▎       | 5/22 [00:02<00:07,  2.35it/s, v_num=1]
Epoch 12:  23%|██▎       | 5/22 [00:02<00:07,  2.35it/s, v_num=1]
Epoch 12:  27%|██▋       | 6/22 [00:02<00:06,  2.64it/s, v_num=1]
Epoch 12:  27%|██▋       | 6/22 [00:02<00:06,  2.64it/s, v_num=1]
Epoch 12:  32%|███▏      | 7/22 [00:02<00:05,  2.89it/s, v_num=1]
Epoch 12:  32%|███▏      | 7/22 [00:02<00:05,  2.89it/s, v_num=1]
Epoch 12:  36%|███▋      | 8/22 [00:02<00:04,  3.11it/s, v_num=1]
Epoch 12:  36%|███▋      | 8/22 [00:02<00:04,  3.11it/s, v_num=1]
Epoch 12:  41%|████      | 9/22 [00:02<00:03,  3.31it/s, v_num=1]
Epoch 12:  41%|████      | 9/22 [00:02<00:03,  3.31it/s, v_num=1]
Epoch 12:  45%|████▌     | 10/22 [00:02<00:03,  3.49it/s, v_num=1]
Epoch 12:  45%|████▌     | 10/22 [00:02<00:03,  3.49it/s, v_num=1]
Epoch 12:  50%|█████     | 11/22 [00:03<00:03,  3.65it/s, v_num=1]
Epoch 12:  50%|█████     | 11/22 [00:03<00:03,  3.65it/s, v_num=1]
Epoch 12:  55%|█████▍    | 12/22 [00:03<00:02,  3.79it/s, v_num=1]
Epoch 12:  55%|█████▍    | 12/22 [00:03<00:02,  3.79it/s, v_num=1]
Epoch 12:  59%|█████▉    | 13/22 [00:03<00:02,  3.93it/s, v_num=1]
Epoch 12:  59%|█████▉    | 13/22 [00:03<00:02,  3.92it/s, v_num=1]
Epoch 12:  64%|██████▎   | 14/22 [00:03<00:01,  4.05it/s, v_num=1]
Epoch 12:  64%|██████▎   | 14/22 [00:03<00:01,  4.05it/s, v_num=1]
Epoch 12:  68%|██████▊   | 15/22 [00:03<00:01,  4.16it/s, v_num=1]
Epoch 12:  68%|██████▊   | 15/22 [00:03<00:01,  4.16it/s, v_num=1]
Epoch 12:  73%|███████▎  | 16/22 [00:03<00:01,  4.27it/s, v_num=1]
Epoch 12:  73%|███████▎  | 16/22 [00:03<00:01,  4.27it/s, v_num=1]
Epoch 12:  77%|███████▋  | 17/22 [00:03<00:01,  4.36it/s, v_num=1]
Epoch 12:  77%|███████▋  | 17/22 [00:03<00:01,  4.36it/s, v_num=1]
Epoch 12:  82%|████████▏ | 18/22 [00:04<00:00,  4.45it/s, v_num=1]
Epoch 12:  82%|████████▏ | 18/22 [00:04<00:00,  4.45it/s, v_num=1]
Epoch 12:  86%|████████▋ | 19/22 [00:04<00:00,  4.53it/s, v_num=1]
Epoch 12:  86%|████████▋ | 19/22 [00:04<00:00,  4.53it/s, v_num=1]
Epoch 12:  91%|█████████ | 20/22 [00:04<00:00,  4.61it/s, v_num=1]
Epoch 12:  91%|█████████ | 20/22 [00:04<00:00,  4.61it/s, v_num=1]
Epoch 12:  95%|█████████▌| 21/22 [00:04<00:00,  4.68it/s, v_num=1]
Epoch 12:  95%|█████████▌| 21/22 [00:04<00:00,  4.68it/s, v_num=1]
Epoch 12: 100%|██████████| 22/22 [00:04<00:00,  4.75it/s, v_num=1]
Epoch 12: 100%|██████████| 22/22 [00:04<00:00,  4.75it/s, v_num=1]
Epoch 12: 100%|██████████| 22/22 [00:04<00:00,  4.75it/s, v_num=1]
Epoch 12:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 13:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 13:   5%|▍         | 1/22 [00:01<00:30,  0.68it/s, v_num=1]
Epoch 13:   5%|▍         | 1/22 [00:01<00:31,  0.68it/s, v_num=1]
Epoch 13:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, v_num=1]
Epoch 13:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, v_num=1]
Epoch 13:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 13:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 13:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 13:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 13:  23%|██▎       | 5/22 [00:02<00:07,  2.42it/s, v_num=1]
Epoch 13:  23%|██▎       | 5/22 [00:02<00:07,  2.42it/s, v_num=1]
Epoch 13:  27%|██▋       | 6/22 [00:02<00:05,  2.70it/s, v_num=1]
Epoch 13:  27%|██▋       | 6/22 [00:02<00:05,  2.70it/s, v_num=1]
Epoch 13:  32%|███▏      | 7/22 [00:02<00:05,  2.95it/s, v_num=1]
Epoch 13:  32%|███▏      | 7/22 [00:02<00:05,  2.95it/s, v_num=1]
Epoch 13:  36%|███▋      | 8/22 [00:02<00:04,  3.17it/s, v_num=1]
Epoch 13:  36%|███▋      | 8/22 [00:02<00:04,  3.17it/s, v_num=1]
Epoch 13:  41%|████      | 9/22 [00:02<00:03,  3.37it/s, v_num=1]
Epoch 13:  41%|████      | 9/22 [00:02<00:03,  3.37it/s, v_num=1]
Epoch 13:  45%|████▌     | 10/22 [00:02<00:03,  3.55it/s, v_num=1]
Epoch 13:  45%|████▌     | 10/22 [00:02<00:03,  3.55it/s, v_num=1]
Epoch 13:  50%|█████     | 11/22 [00:02<00:02,  3.71it/s, v_num=1]
Epoch 13:  50%|█████     | 11/22 [00:02<00:02,  3.71it/s, v_num=1]
Epoch 13:  55%|█████▍    | 12/22 [00:03<00:02,  3.86it/s, v_num=1]
Epoch 13:  55%|█████▍    | 12/22 [00:03<00:02,  3.85it/s, v_num=1]
Epoch 13:  59%|█████▉    | 13/22 [00:03<00:02,  3.98it/s, v_num=1]
Epoch 13:  59%|█████▉    | 13/22 [00:03<00:02,  3.98it/s, v_num=1]
Epoch 13:  64%|██████▎   | 14/22 [00:03<00:01,  4.11it/s, v_num=1]
Epoch 13:  64%|██████▎   | 14/22 [00:03<00:01,  4.11it/s, v_num=1]
Epoch 13:  68%|██████▊   | 15/22 [00:03<00:01,  4.22it/s, v_num=1]
Epoch 13:  68%|██████▊   | 15/22 [00:03<00:01,  4.22it/s, v_num=1]
Epoch 13:  73%|███████▎  | 16/22 [00:03<00:01,  4.32it/s, v_num=1]
Epoch 13:  73%|███████▎  | 16/22 [00:03<00:01,  4.32it/s, v_num=1]
Epoch 13:  77%|███████▋  | 17/22 [00:03<00:01,  4.42it/s, v_num=1]
Epoch 13:  77%|███████▋  | 17/22 [00:03<00:01,  4.42it/s, v_num=1]
Epoch 13:  82%|████████▏ | 18/22 [00:03<00:00,  4.50it/s, v_num=1]
Epoch 13:  82%|████████▏ | 18/22 [00:03<00:00,  4.50it/s, v_num=1]
Epoch 13:  86%|████████▋ | 19/22 [00:04<00:00,  4.58it/s, v_num=1]
Epoch 13:  86%|████████▋ | 19/22 [00:04<00:00,  4.58it/s, v_num=1]
Epoch 13:  91%|█████████ | 20/22 [00:04<00:00,  4.66it/s, v_num=1]
Epoch 13:  91%|█████████ | 20/22 [00:04<00:00,  4.66it/s, v_num=1]
Epoch 13:  95%|█████████▌| 21/22 [00:04<00:00,  4.73it/s, v_num=1]
Epoch 13:  95%|█████████▌| 21/22 [00:04<00:00,  4.73it/s, v_num=1]
Epoch 13: 100%|██████████| 22/22 [00:04<00:00,  4.80it/s, v_num=1]
Epoch 13: 100%|██████████| 22/22 [00:04<00:00,  4.80it/s, v_num=1]
Epoch 13: 100%|██████████| 22/22 [00:04<00:00,  4.80it/s, v_num=1]
Epoch 13:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 14:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 14:   5%|▍         | 1/22 [00:01<00:30,  0.68it/s, v_num=1]
Epoch 14:   5%|▍         | 1/22 [00:01<00:30,  0.68it/s, v_num=1]
Epoch 14:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, v_num=1]
Epoch 14:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, v_num=1]
Epoch 14:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 14:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 14:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 14:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 14:  23%|██▎       | 5/22 [00:02<00:07,  2.42it/s, v_num=1]
Epoch 14:  23%|██▎       | 5/22 [00:02<00:07,  2.42it/s, v_num=1]
Epoch 14:  27%|██▋       | 6/22 [00:02<00:05,  2.70it/s, v_num=1]
Epoch 14:  27%|██▋       | 6/22 [00:02<00:05,  2.70it/s, v_num=1]
Epoch 14:  32%|███▏      | 7/22 [00:02<00:05,  2.96it/s, v_num=1]
Epoch 14:  32%|███▏      | 7/22 [00:02<00:05,  2.96it/s, v_num=1]
Epoch 14:  36%|███▋      | 8/22 [00:02<00:04,  3.18it/s, v_num=1]
Epoch 14:  36%|███▋      | 8/22 [00:02<00:04,  3.18it/s, v_num=1]
Epoch 14:  41%|████      | 9/22 [00:02<00:03,  3.38it/s, v_num=1]
Epoch 14:  41%|████      | 9/22 [00:02<00:03,  3.38it/s, v_num=1]
Epoch 14:  45%|████▌     | 10/22 [00:02<00:03,  3.56it/s, v_num=1]
Epoch 14:  45%|████▌     | 10/22 [00:02<00:03,  3.56it/s, v_num=1]
Epoch 14:  50%|█████     | 11/22 [00:02<00:02,  3.72it/s, v_num=1]
Epoch 14:  50%|█████     | 11/22 [00:02<00:02,  3.72it/s, v_num=1]
Epoch 14:  55%|█████▍    | 12/22 [00:03<00:02,  3.86it/s, v_num=1]
Epoch 14:  55%|█████▍    | 12/22 [00:03<00:02,  3.86it/s, v_num=1]
Epoch 14:  59%|█████▉    | 13/22 [00:03<00:02,  3.99it/s, v_num=1]
Epoch 14:  59%|█████▉    | 13/22 [00:03<00:02,  3.99it/s, v_num=1]
Epoch 14:  64%|██████▎   | 14/22 [00:03<00:01,  4.11it/s, v_num=1]
Epoch 14:  64%|██████▎   | 14/22 [00:03<00:01,  4.11it/s, v_num=1]
Epoch 14:  68%|██████▊   | 15/22 [00:03<00:01,  4.22it/s, v_num=1]
Epoch 14:  68%|██████▊   | 15/22 [00:03<00:01,  4.22it/s, v_num=1]
Epoch 14:  73%|███████▎  | 16/22 [00:03<00:01,  4.32it/s, v_num=1]
Epoch 14:  73%|███████▎  | 16/22 [00:03<00:01,  4.32it/s, v_num=1]
Epoch 14:  77%|███████▋  | 17/22 [00:03<00:01,  4.42it/s, v_num=1]
Epoch 14:  77%|███████▋  | 17/22 [00:03<00:01,  4.42it/s, v_num=1]
Epoch 14:  82%|████████▏ | 18/22 [00:03<00:00,  4.51it/s, v_num=1]
Epoch 14:  82%|████████▏ | 18/22 [00:03<00:00,  4.51it/s, v_num=1]
Epoch 14:  86%|████████▋ | 19/22 [00:04<00:00,  4.59it/s, v_num=1]
Epoch 14:  86%|████████▋ | 19/22 [00:04<00:00,  4.59it/s, v_num=1]
Epoch 14:  91%|█████████ | 20/22 [00:04<00:00,  4.66it/s, v_num=1]
Epoch 14:  91%|█████████ | 20/22 [00:04<00:00,  4.66it/s, v_num=1]
Epoch 14:  95%|█████████▌| 21/22 [00:04<00:00,  4.74it/s, v_num=1]
Epoch 14:  95%|█████████▌| 21/22 [00:04<00:00,  4.74it/s, v_num=1]
Epoch 14: 100%|██████████| 22/22 [00:04<00:00,  4.80it/s, v_num=1]
Epoch 14: 100%|██████████| 22/22 [00:04<00:00,  4.80it/s, v_num=1]
Epoch 14: 100%|██████████| 22/22 [00:04<00:00,  4.80it/s, v_num=1]
Epoch 14:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 15:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 15:   5%|▍         | 1/22 [00:01<00:31,  0.67it/s, v_num=1]
Epoch 15:   5%|▍         | 1/22 [00:01<00:31,  0.67it/s, v_num=1]
Epoch 15:   9%|▉         | 2/22 [00:01<00:16,  1.22it/s, v_num=1]
Epoch 15:   9%|▉         | 2/22 [00:01<00:16,  1.22it/s, v_num=1]
Epoch 15:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 15:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 15:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 15:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 15:  23%|██▎       | 5/22 [00:02<00:07,  2.40it/s, v_num=1]
Epoch 15:  23%|██▎       | 5/22 [00:02<00:07,  2.40it/s, v_num=1]
Epoch 15:  27%|██▋       | 6/22 [00:02<00:05,  2.69it/s, v_num=1]
Epoch 15:  27%|██▋       | 6/22 [00:02<00:05,  2.69it/s, v_num=1]
Epoch 15:  32%|███▏      | 7/22 [00:02<00:05,  2.95it/s, v_num=1]
Epoch 15:  32%|███▏      | 7/22 [00:02<00:05,  2.95it/s, v_num=1]
Epoch 15:  36%|███▋      | 8/22 [00:02<00:04,  3.17it/s, v_num=1]
Epoch 15:  36%|███▋      | 8/22 [00:02<00:04,  3.17it/s, v_num=1]
Epoch 15:  41%|████      | 9/22 [00:02<00:03,  3.37it/s, v_num=1]
Epoch 15:  41%|████      | 9/22 [00:02<00:03,  3.36it/s, v_num=1]
Epoch 15:  45%|████▌     | 10/22 [00:02<00:03,  3.54it/s, v_num=1]
Epoch 15:  45%|████▌     | 10/22 [00:02<00:03,  3.54it/s, v_num=1]
Epoch 15:  50%|█████     | 11/22 [00:02<00:02,  3.71it/s, v_num=1]
Epoch 15:  50%|█████     | 11/22 [00:02<00:02,  3.71it/s, v_num=1]
Epoch 15:  55%|█████▍    | 12/22 [00:03<00:02,  3.85it/s, v_num=1]
Epoch 15:  55%|█████▍    | 12/22 [00:03<00:02,  3.85it/s, v_num=1]
Epoch 15:  59%|█████▉    | 13/22 [00:03<00:02,  3.98it/s, v_num=1]
Epoch 15:  59%|█████▉    | 13/22 [00:03<00:02,  3.98it/s, v_num=1]
Epoch 15:  64%|██████▎   | 14/22 [00:03<00:01,  4.10it/s, v_num=1]
Epoch 15:  64%|██████▎   | 14/22 [00:03<00:01,  4.10it/s, v_num=1]
Epoch 15:  68%|██████▊   | 15/22 [00:03<00:01,  4.21it/s, v_num=1]
Epoch 15:  68%|██████▊   | 15/22 [00:03<00:01,  4.21it/s, v_num=1]
Epoch 15:  73%|███████▎  | 16/22 [00:03<00:01,  4.31it/s, v_num=1]
Epoch 15:  73%|███████▎  | 16/22 [00:03<00:01,  4.31it/s, v_num=1]
Epoch 15:  77%|███████▋  | 17/22 [00:03<00:01,  4.41it/s, v_num=1]
Epoch 15:  77%|███████▋  | 17/22 [00:03<00:01,  4.40it/s, v_num=1]
Epoch 15:  82%|████████▏ | 18/22 [00:04<00:00,  4.49it/s, v_num=1]
Epoch 15:  82%|████████▏ | 18/22 [00:04<00:00,  4.49it/s, v_num=1]
Epoch 15:  86%|████████▋ | 19/22 [00:04<00:00,  4.57it/s, v_num=1]
Epoch 15:  86%|████████▋ | 19/22 [00:04<00:00,  4.57it/s, v_num=1]
Epoch 15:  91%|█████████ | 20/22 [00:04<00:00,  4.65it/s, v_num=1]
Epoch 15:  91%|█████████ | 20/22 [00:04<00:00,  4.65it/s, v_num=1]
Epoch 15:  95%|█████████▌| 21/22 [00:04<00:00,  4.72it/s, v_num=1]
Epoch 15:  95%|█████████▌| 21/22 [00:04<00:00,  4.72it/s, v_num=1]
Epoch 15: 100%|██████████| 22/22 [00:04<00:00,  4.79it/s, v_num=1]
Epoch 15: 100%|██████████| 22/22 [00:04<00:00,  4.79it/s, v_num=1]
Epoch 15: 100%|██████████| 22/22 [00:04<00:00,  4.79it/s, v_num=1]
Epoch 15:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 16:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 16:   5%|▍         | 1/22 [00:01<00:31,  0.66it/s, v_num=1]
Epoch 16:   5%|▍         | 1/22 [00:01<00:31,  0.66it/s, v_num=1]
Epoch 16:   9%|▉         | 2/22 [00:01<00:16,  1.20it/s, v_num=1]
Epoch 16:   9%|▉         | 2/22 [00:01<00:16,  1.20it/s, v_num=1]
Epoch 16:  14%|█▎        | 3/22 [00:01<00:11,  1.66it/s, v_num=1]
Epoch 16:  14%|█▎        | 3/22 [00:01<00:11,  1.66it/s, v_num=1]
Epoch 16:  18%|█▊        | 4/22 [00:01<00:08,  2.04it/s, v_num=1]
Epoch 16:  18%|█▊        | 4/22 [00:01<00:08,  2.04it/s, v_num=1]
Epoch 16:  23%|██▎       | 5/22 [00:02<00:07,  2.37it/s, v_num=1]
Epoch 16:  23%|██▎       | 5/22 [00:02<00:07,  2.37it/s, v_num=1]
Epoch 16:  27%|██▋       | 6/22 [00:02<00:06,  2.66it/s, v_num=1]
Epoch 16:  27%|██▋       | 6/22 [00:02<00:06,  2.66it/s, v_num=1]
Epoch 16:  32%|███▏      | 7/22 [00:02<00:05,  2.91it/s, v_num=1]
Epoch 16:  32%|███▏      | 7/22 [00:02<00:05,  2.91it/s, v_num=1]
Epoch 16:  36%|███▋      | 8/22 [00:02<00:04,  3.13it/s, v_num=1]
Epoch 16:  36%|███▋      | 8/22 [00:02<00:04,  3.13it/s, v_num=1]
Epoch 16:  41%|████      | 9/22 [00:02<00:03,  3.33it/s, v_num=1]
Epoch 16:  41%|████      | 9/22 [00:02<00:03,  3.33it/s, v_num=1]
Epoch 16:  45%|████▌     | 10/22 [00:02<00:03,  3.51it/s, v_num=1]
Epoch 16:  45%|████▌     | 10/22 [00:02<00:03,  3.51it/s, v_num=1]
Epoch 16:  50%|█████     | 11/22 [00:02<00:02,  3.67it/s, v_num=1]
Epoch 16:  50%|█████     | 11/22 [00:02<00:02,  3.67it/s, v_num=1]
Epoch 16:  55%|█████▍    | 12/22 [00:03<00:02,  3.82it/s, v_num=1]
Epoch 16:  55%|█████▍    | 12/22 [00:03<00:02,  3.81it/s, v_num=1]
Epoch 16:  59%|█████▉    | 13/22 [00:03<00:02,  3.95it/s, v_num=1]
Epoch 16:  59%|█████▉    | 13/22 [00:03<00:02,  3.95it/s, v_num=1]
Epoch 16:  64%|██████▎   | 14/22 [00:03<00:01,  4.07it/s, v_num=1]
Epoch 16:  64%|██████▎   | 14/22 [00:03<00:01,  4.07it/s, v_num=1]
Epoch 16:  68%|██████▊   | 15/22 [00:03<00:01,  4.18it/s, v_num=1]
Epoch 16:  68%|██████▊   | 15/22 [00:03<00:01,  4.18it/s, v_num=1]
Epoch 16:  73%|███████▎  | 16/22 [00:03<00:01,  4.29it/s, v_num=1]
Epoch 16:  73%|███████▎  | 16/22 [00:03<00:01,  4.29it/s, v_num=1]
Epoch 16:  77%|███████▋  | 17/22 [00:03<00:01,  4.38it/s, v_num=1]
Epoch 16:  77%|███████▋  | 17/22 [00:03<00:01,  4.38it/s, v_num=1]
Epoch 16:  82%|████████▏ | 18/22 [00:04<00:00,  4.47it/s, v_num=1]
Epoch 16:  82%|████████▏ | 18/22 [00:04<00:00,  4.47it/s, v_num=1]
Epoch 16:  86%|████████▋ | 19/22 [00:04<00:00,  4.55it/s, v_num=1]
Epoch 16:  86%|████████▋ | 19/22 [00:04<00:00,  4.55it/s, v_num=1]
Epoch 16:  91%|█████████ | 20/22 [00:04<00:00,  4.62it/s, v_num=1]
Epoch 16:  91%|█████████ | 20/22 [00:04<00:00,  4.62it/s, v_num=1]
Epoch 16:  95%|█████████▌| 21/22 [00:04<00:00,  4.70it/s, v_num=1]
Epoch 16:  95%|█████████▌| 21/22 [00:04<00:00,  4.70it/s, v_num=1]
Epoch 16: 100%|██████████| 22/22 [00:04<00:00,  4.76it/s, v_num=1]
Epoch 16: 100%|██████████| 22/22 [00:04<00:00,  4.76it/s, v_num=1]
Epoch 16: 100%|██████████| 22/22 [00:04<00:00,  4.76it/s, v_num=1]
Epoch 16:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 17:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 17:   5%|▍         | 1/22 [00:01<00:31,  0.68it/s, v_num=1]
Epoch 17:   5%|▍         | 1/22 [00:01<00:31,  0.68it/s, v_num=1]
Epoch 17:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, v_num=1]
Epoch 17:   9%|▉         | 2/22 [00:01<00:16,  1.23it/s, v_num=1]
Epoch 17:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 17:  14%|█▎        | 3/22 [00:01<00:11,  1.69it/s, v_num=1]
Epoch 17:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 17:  18%|█▊        | 4/22 [00:01<00:08,  2.08it/s, v_num=1]
Epoch 17:  23%|██▎       | 5/22 [00:02<00:07,  2.42it/s, v_num=1]
Epoch 17:  23%|██▎       | 5/22 [00:02<00:07,  2.42it/s, v_num=1]
Epoch 17:  27%|██▋       | 6/22 [00:02<00:05,  2.70it/s, v_num=1]
Epoch 17:  27%|██▋       | 6/22 [00:02<00:05,  2.70it/s, v_num=1]
Epoch 17:  32%|███▏      | 7/22 [00:02<00:05,  2.95it/s, v_num=1]
Epoch 17:  32%|███▏      | 7/22 [00:02<00:05,  2.95it/s, v_num=1]
Epoch 17:  36%|███▋      | 8/22 [00:02<00:04,  3.18it/s, v_num=1]
Epoch 17:  36%|███▋      | 8/22 [00:02<00:04,  3.18it/s, v_num=1]
Epoch 17:  41%|████      | 9/22 [00:02<00:03,  3.38it/s, v_num=1]
Epoch 17:  41%|████      | 9/22 [00:02<00:03,  3.38it/s, v_num=1]
Epoch 17:  45%|████▌     | 10/22 [00:02<00:03,  3.56it/s, v_num=1]
Epoch 17:  45%|████▌     | 10/22 [00:02<00:03,  3.56it/s, v_num=1]
Epoch 17:  50%|█████     | 11/22 [00:02<00:02,  3.72it/s, v_num=1]
Epoch 17:  50%|█████     | 11/22 [00:02<00:02,  3.72it/s, v_num=1]
Epoch 17:  55%|█████▍    | 12/22 [00:03<00:02,  3.87it/s, v_num=1]
Epoch 17:  55%|█████▍    | 12/22 [00:03<00:02,  3.87it/s, v_num=1]
Epoch 17:  59%|█████▉    | 13/22 [00:03<00:02,  4.00it/s, v_num=1]
Epoch 17:  59%|█████▉    | 13/22 [00:03<00:02,  4.00it/s, v_num=1]
Epoch 17:  64%|██████▎   | 14/22 [00:03<00:01,  4.12it/s, v_num=1]
Epoch 17:  64%|██████▎   | 14/22 [00:03<00:01,  4.12it/s, v_num=1]
Epoch 17:  68%|██████▊   | 15/22 [00:03<00:01,  4.23it/s, v_num=1]
Epoch 17:  68%|██████▊   | 15/22 [00:03<00:01,  4.23it/s, v_num=1]
Epoch 17:  73%|███████▎  | 16/22 [00:03<00:01,  4.33it/s, v_num=1]
Epoch 17:  73%|███████▎  | 16/22 [00:03<00:01,  4.33it/s, v_num=1]
Epoch 17:  77%|███████▋  | 17/22 [00:03<00:01,  4.43it/s, v_num=1]
Epoch 17:  77%|███████▋  | 17/22 [00:03<00:01,  4.43it/s, v_num=1]
Epoch 17:  82%|████████▏ | 18/22 [00:03<00:00,  4.52it/s, v_num=1]
Epoch 17:  82%|████████▏ | 18/22 [00:03<00:00,  4.52it/s, v_num=1]
Epoch 17:  86%|████████▋ | 19/22 [00:04<00:00,  4.60it/s, v_num=1]
Epoch 17:  86%|████████▋ | 19/22 [00:04<00:00,  4.60it/s, v_num=1]
Epoch 17:  91%|█████████ | 20/22 [00:04<00:00,  4.68it/s, v_num=1]
Epoch 17:  91%|█████████ | 20/22 [00:04<00:00,  4.68it/s, v_num=1]
Epoch 17:  95%|█████████▌| 21/22 [00:04<00:00,  4.75it/s, v_num=1]
Epoch 17:  95%|█████████▌| 21/22 [00:04<00:00,  4.75it/s, v_num=1]
Epoch 17: 100%|██████████| 22/22 [00:04<00:00,  4.81it/s, v_num=1]
Epoch 17: 100%|██████████| 22/22 [00:04<00:00,  4.81it/s, v_num=1]
Epoch 17: 100%|██████████| 22/22 [00:04<00:00,  4.81it/s, v_num=1]
Epoch 17:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 18:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 18:   5%|▍         | 1/22 [00:01<00:32,  0.65it/s, v_num=1]
Epoch 18:   5%|▍         | 1/22 [00:01<00:32,  0.65it/s, v_num=1]
Epoch 18:   9%|▉         | 2/22 [00:01<00:16,  1.19it/s, v_num=1]
Epoch 18:   9%|▉         | 2/22 [00:01<00:16,  1.19it/s, v_num=1]
Epoch 18:  14%|█▎        | 3/22 [00:01<00:11,  1.64it/s, v_num=1]
Epoch 18:  14%|█▎        | 3/22 [00:01<00:11,  1.64it/s, v_num=1]
Epoch 18:  18%|█▊        | 4/22 [00:01<00:08,  2.03it/s, v_num=1]
Epoch 18:  18%|█▊        | 4/22 [00:01<00:08,  2.03it/s, v_num=1]
Epoch 18:  23%|██▎       | 5/22 [00:02<00:07,  2.35it/s, v_num=1]
Epoch 18:  23%|██▎       | 5/22 [00:02<00:07,  2.35it/s, v_num=1]
Epoch 18:  27%|██▋       | 6/22 [00:02<00:06,  2.64it/s, v_num=1]
Epoch 18:  27%|██▋       | 6/22 [00:02<00:06,  2.64it/s, v_num=1]
Epoch 18:  32%|███▏      | 7/22 [00:02<00:05,  2.89it/s, v_num=1]
Epoch 18:  32%|███▏      | 7/22 [00:02<00:05,  2.89it/s, v_num=1]
Epoch 18:  36%|███▋      | 8/22 [00:02<00:04,  3.12it/s, v_num=1]
Epoch 18:  36%|███▋      | 8/22 [00:02<00:04,  3.12it/s, v_num=1]
Epoch 18:  41%|████      | 9/22 [00:02<00:03,  3.31it/s, v_num=1]
Epoch 18:  41%|████      | 9/22 [00:02<00:03,  3.31it/s, v_num=1]
Epoch 18:  45%|████▌     | 10/22 [00:02<00:03,  3.49it/s, v_num=1]
Epoch 18:  45%|████▌     | 10/22 [00:02<00:03,  3.49it/s, v_num=1]
Epoch 18:  50%|█████     | 11/22 [00:03<00:03,  3.65it/s, v_num=1]
Epoch 18:  50%|█████     | 11/22 [00:03<00:03,  3.65it/s, v_num=1]
Epoch 18:  55%|█████▍    | 12/22 [00:03<00:02,  3.80it/s, v_num=1]
Epoch 18:  55%|█████▍    | 12/22 [00:03<00:02,  3.79it/s, v_num=1]
Epoch 18:  59%|█████▉    | 13/22 [00:03<00:02,  3.93it/s, v_num=1]
Epoch 18:  59%|█████▉    | 13/22 [00:03<00:02,  3.92it/s, v_num=1]
Epoch 18:  64%|██████▎   | 14/22 [00:03<00:01,  4.05it/s, v_num=1]
Epoch 18:  64%|██████▎   | 14/22 [00:03<00:01,  4.05it/s, v_num=1]
Epoch 18:  68%|██████▊   | 15/22 [00:03<00:01,  4.16it/s, v_num=1]
Epoch 18:  68%|██████▊   | 15/22 [00:03<00:01,  4.16it/s, v_num=1]
Epoch 18:  73%|███████▎  | 16/22 [00:03<00:01,  4.26it/s, v_num=1]
Epoch 18:  73%|███████▎  | 16/22 [00:03<00:01,  4.26it/s, v_num=1]
Epoch 18:  77%|███████▋  | 17/22 [00:03<00:01,  4.36it/s, v_num=1]
Epoch 18:  77%|███████▋  | 17/22 [00:03<00:01,  4.36it/s, v_num=1]
Epoch 18:  82%|████████▏ | 18/22 [00:04<00:00,  4.45it/s, v_num=1]
Epoch 18:  82%|████████▏ | 18/22 [00:04<00:00,  4.45it/s, v_num=1]
Epoch 18:  86%|████████▋ | 19/22 [00:04<00:00,  4.53it/s, v_num=1]
Epoch 18:  86%|████████▋ | 19/22 [00:04<00:00,  4.53it/s, v_num=1]
Epoch 18:  91%|█████████ | 20/22 [00:04<00:00,  4.60it/s, v_num=1]
Epoch 18:  91%|█████████ | 20/22 [00:04<00:00,  4.60it/s, v_num=1]
Epoch 18:  95%|█████████▌| 21/22 [00:04<00:00,  4.67it/s, v_num=1]
Epoch 18:  95%|█████████▌| 21/22 [00:04<00:00,  4.67it/s, v_num=1]
Epoch 18: 100%|██████████| 22/22 [00:04<00:00,  4.74it/s, v_num=1]
Epoch 18: 100%|██████████| 22/22 [00:04<00:00,  4.74it/s, v_num=1]
Epoch 18: 100%|██████████| 22/22 [00:04<00:00,  4.74it/s, v_num=1]
Epoch 18:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 19:   0%|          | 0/22 [00:00<?, ?it/s, v_num=1]
Epoch 19:   5%|▍         | 1/22 [00:01<00:31,  0.67it/s, v_num=1]
Epoch 19:   5%|▍         | 1/22 [00:01<00:31,  0.67it/s, v_num=1]
Epoch 19:   9%|▉         | 2/22 [00:01<00:16,  1.22it/s, v_num=1]
Epoch 19:   9%|▉         | 2/22 [00:01<00:16,  1.22it/s, v_num=1]
Epoch 19:  14%|█▎        | 3/22 [00:01<00:11,  1.67it/s, v_num=1]
Epoch 19:  14%|█▎        | 3/22 [00:01<00:11,  1.67it/s, v_num=1]
Epoch 19:  18%|█▊        | 4/22 [00:01<00:08,  2.06it/s, v_num=1]
Epoch 19:  18%|█▊        | 4/22 [00:01<00:08,  2.06it/s, v_num=1]
Epoch 19:  23%|██▎       | 5/22 [00:02<00:07,  2.39it/s, v_num=1]
Epoch 19:  23%|██▎       | 5/22 [00:02<00:07,  2.39it/s, v_num=1]
Epoch 19:  27%|██▋       | 6/22 [00:02<00:05,  2.67it/s, v_num=1]
Epoch 19:  27%|██▋       | 6/22 [00:02<00:05,  2.67it/s, v_num=1]
Epoch 19:  32%|███▏      | 7/22 [00:02<00:05,  2.93it/s, v_num=1]
Epoch 19:  32%|███▏      | 7/22 [00:02<00:05,  2.93it/s, v_num=1]
Epoch 19:  36%|███▋      | 8/22 [00:02<00:04,  3.15it/s, v_num=1]
Epoch 19:  36%|███▋      | 8/22 [00:02<00:04,  3.15it/s, v_num=1]
Epoch 19:  41%|████      | 9/22 [00:02<00:03,  3.35it/s, v_num=1]
Epoch 19:  41%|████      | 9/22 [00:02<00:03,  3.35it/s, v_num=1]
Epoch 19:  45%|████▌     | 10/22 [00:02<00:03,  3.53it/s, v_num=1]
Epoch 19:  45%|████▌     | 10/22 [00:02<00:03,  3.53it/s, v_num=1]
Epoch 19:  50%|█████     | 11/22 [00:02<00:02,  3.70it/s, v_num=1]
Epoch 19:  50%|█████     | 11/22 [00:02<00:02,  3.70it/s, v_num=1]
Epoch 19:  55%|█████▍    | 12/22 [00:03<00:02,  3.84it/s, v_num=1]
Epoch 19:  55%|█████▍    | 12/22 [00:03<00:02,  3.84it/s, v_num=1]
Epoch 19:  59%|█████▉    | 13/22 [00:03<00:02,  3.97it/s, v_num=1]
Epoch 19:  59%|█████▉    | 13/22 [00:03<00:02,  3.97it/s, v_num=1]
Epoch 19:  64%|██████▎   | 14/22 [00:03<00:01,  4.09it/s, v_num=1]
Epoch 19:  64%|██████▎   | 14/22 [00:03<00:01,  4.09it/s, v_num=1]
Epoch 19:  68%|██████▊   | 15/22 [00:03<00:01,  4.20it/s, v_num=1]
Epoch 19:  68%|██████▊   | 15/22 [00:03<00:01,  4.20it/s, v_num=1]
Epoch 19:  73%|███████▎  | 16/22 [00:03<00:01,  4.31it/s, v_num=1]
Epoch 19:  73%|███████▎  | 16/22 [00:03<00:01,  4.31it/s, v_num=1]
Epoch 19:  77%|███████▋  | 17/22 [00:03<00:01,  4.40it/s, v_num=1]
Epoch 19:  77%|███████▋  | 17/22 [00:03<00:01,  4.40it/s, v_num=1]
Epoch 19:  82%|████████▏ | 18/22 [00:04<00:00,  4.49it/s, v_num=1]
Epoch 19:  82%|████████▏ | 18/22 [00:04<00:00,  4.49it/s, v_num=1]
Epoch 19:  86%|████████▋ | 19/22 [00:04<00:00,  4.57it/s, v_num=1]
Epoch 19:  86%|████████▋ | 19/22 [00:04<00:00,  4.57it/s, v_num=1]
Epoch 19:  91%|█████████ | 20/22 [00:04<00:00,  4.64it/s, v_num=1]
Epoch 19:  91%|█████████ | 20/22 [00:04<00:00,  4.64it/s, v_num=1]
Epoch 19:  95%|█████████▌| 21/22 [00:04<00:00,  4.71it/s, v_num=1]
Epoch 19:  95%|█████████▌| 21/22 [00:04<00:00,  4.71it/s, v_num=1]
Epoch 19: 100%|██████████| 22/22 [00:04<00:00,  4.77it/s, v_num=1]
Epoch 19: 100%|██████████| 22/22 [00:04<00:00,  4.77it/s, v_num=1]
Epoch 19: 100%|██████████| 22/22 [00:04<00:00,  4.77it/s, v_num=1]
Epoch 19: 100%|██████████| 22/22 [00:05<00:00,  4.29it/s, v_num=1]

other example

plot_knn_examples(embeddings, filenames)

• 
• 
• 
• 
• 
• 

What’s next?

# You could use the pre-trained model and train a classifier on top.
pretrained_resnet_backbone = model.backbone

# you can also store the backbone and use it in another code
state_dict = {"resnet18_parameters": pretrained_resnet_backbone.state_dict()}
torch.save(state_dict, "model.pth")

THIS COULD BE IN A NEW FILE (e.g. inference.py)

Make sure you place the model.pth file in the same folder as this code

# load the model in a new file for inference
resnet18_new = torchvision.models.resnet18()

# note that we need to create exactly the same backbone in order to load the weights
backbone_new = nn.Sequential(*list(resnet18_new.children())[:-1])

ckpt = torch.load("model.pth")
backbone_new.load_state_dict(ckpt["resnet18_parameters"])

<All keys matched successfully>

Next Steps

Interested in exploring other self-supervised models? Check out our other tutorials:

• Tutorial 2: Train MoCo on CIFAR-10

• Tutorial 4: Train SimSiam on Satellite Images

• Tutorial 5: Custom Augmentations

• Tutorial 6: Pre-train a Detectron2 Backbone with Lightly