Note
Click here to download the full example code
Tutorial 3: Train SimCLR on Clothing
In this tutorial, we will train a SimCLR model using lightly. The model, augmentations and training procedure is from A Simple Framework for Contrastive Learning of Visual Representations.
The paper explores a rather simple training procedure for contrastive learning. Since we use the typical contrastive learning loss based on NCE the method greatly benefits from having larger batch sizes. In this example, we use a batch size of 256 and paired with the input resolution per image of 64x64 pixels and a resnet-18 model this example requires 16GB of GPU memory.
We use the clothing dataset from Alex Grigorev for this tutorial.
In this tutorial you will learn:
How to create a SimCLR model
How to generate image representations
How different augmentations impact the learned representations
Imports
Import the Python frameworks we need for this tutorial.
import os
import matplotlib.pyplot as plt
import numpy as np
import pytorch_lightning as pl
import torch
import torch.nn as nn
import torchvision
from PIL import Image
from sklearn.neighbors import NearestNeighbors
from sklearn.preprocessing import normalize
from lightly.data import LightlyDataset, SimCLRCollateFunction, collate
Configuration
We set some configuration parameters for our experiment. Feel free to change them and analyze the effect.
The default configuration with a batch size of 256 and input resolution of 128 requires 6GB of GPU memory.
num_workers = 8
batch_size = 256
seed = 1
max_epochs = 20
input_size = 128
num_ftrs = 32
Let’s set the seed for our experiments
pl.seed_everything(seed)
1
Make sure path_to_data points to the downloaded clothing dataset. You can download it using git clone https://github.com/alexeygrigorev/clothing-dataset.git
path_to_data = "/datasets/clothing-dataset/images"
Setup data augmentations and loaders
The images from the dataset have been taken from above when the clothing was on a table, bed or floor. Therefore, we can make use of additional augmentations such as vertical flip or random rotation (90 degrees). By adding these augmentations we learn our model invariance regarding the orientation of the clothing piece. E.g. we don’t care if a shirt is upside down but more about the strcture which make it a shirt.
You can learn more about the different augmentations and learned invariances here: Advanced Concepts in Self-Supervised Learning.
collate_fn = SimCLRCollateFunction(input_size=input_size, vf_prob=0.5, rr_prob=0.5)
# We create a torchvision transformation for embedding the dataset after
# training
test_transforms = torchvision.transforms.Compose(
[
torchvision.transforms.Resize((input_size, input_size)),
torchvision.transforms.ToTensor(),
torchvision.transforms.Normalize(
mean=collate.imagenet_normalize["mean"],
std=collate.imagenet_normalize["std"],
),
]
)
dataset_train_simclr = LightlyDataset(input_dir=path_to_data)
dataset_test = LightlyDataset(input_dir=path_to_data, transform=test_transforms)
dataloader_train_simclr = torch.utils.data.DataLoader(
dataset_train_simclr,
batch_size=batch_size,
shuffle=True,
collate_fn=collate_fn,
drop_last=True,
num_workers=num_workers,
)
dataloader_test = torch.utils.data.DataLoader(
dataset_test,
batch_size=batch_size,
shuffle=False,
drop_last=False,
num_workers=num_workers,
)
Create the SimCLR Model
Now we create the SimCLR model. We implement it as a PyTorch Lightning Module and use a ResNet-18 backbone from Torchvision. Lightly provides implementations of the SimCLR projection head and loss function in the SimCLRProjectionHead and NTXentLoss classes. We can simply import them and combine the building blocks in the module.
from lightly.loss import NTXentLoss
from lightly.models.modules.heads import SimCLRProjectionHead
class SimCLRModel(pl.LightningModule):
def __init__(self):
super().__init__()
# create a ResNet backbone and remove the classification head
resnet = torchvision.models.resnet18()
self.backbone = nn.Sequential(*list(resnet.children())[:-1])
hidden_dim = resnet.fc.in_features
self.projection_head = SimCLRProjectionHead(hidden_dim, hidden_dim, 128)
self.criterion = NTXentLoss()
def forward(self, x):
h = self.backbone(x).flatten(start_dim=1)
z = self.projection_head(h)
return z
def training_step(self, batch, batch_idx):
(x0, x1), _, _ = batch
z0 = self.forward(x0)
z1 = self.forward(x1)
loss = self.criterion(z0, z1)
self.log("train_loss_ssl", loss)
return loss
def configure_optimizers(self):
optim = torch.optim.SGD(
self.parameters(), lr=6e-2, momentum=0.9, weight_decay=5e-4
)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim, max_epochs)
return [optim], [scheduler]
Train the module using the PyTorch Lightning Trainer on a single GPU.
model = SimCLRModel()
trainer = pl.Trainer(max_epochs=max_epochs, devices=1, accelerator="gpu")
trainer.fit(model, dataloader_train_simclr)
/opt/runner_04/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1555: PossibleUserWarning: The number of training batches (22) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
rank_zero_warn(
Training: 0it [00:00, ?it/s]
Training: 0%| | 0/22 [00:00<?, ?it/s]
Epoch 0: 0%| | 0/22 [00:00<?, ?it/s]
Epoch 0: 5%|4 | 1/22 [00:04<01:32, 4.40s/it]
Epoch 0: 5%|4 | 1/22 [00:04<01:33, 4.46s/it, loss=6.21, v_num=0]
Epoch 0: 9%|9 | 2/22 [00:04<00:45, 2.28s/it, loss=6.21, v_num=0]
Epoch 0: 9%|9 | 2/22 [00:04<00:46, 2.31s/it, loss=6.2, v_num=0]
Epoch 0: 14%|#3 | 3/22 [00:04<00:30, 1.58s/it, loss=6.2, v_num=0]
Epoch 0: 14%|#3 | 3/22 [00:04<00:30, 1.60s/it, loss=6.19, v_num=0]
Epoch 0: 18%|#8 | 4/22 [00:04<00:21, 1.22s/it, loss=6.19, v_num=0]
Epoch 0: 18%|#8 | 4/22 [00:04<00:22, 1.23s/it, loss=6.16, v_num=0]
Epoch 0: 23%|##2 | 5/22 [00:05<00:17, 1.01s/it, loss=6.16, v_num=0]
Epoch 0: 23%|##2 | 5/22 [00:05<00:17, 1.02s/it, loss=6.12, v_num=0]
Epoch 0: 27%|##7 | 6/22 [00:05<00:13, 1.16it/s, loss=6.12, v_num=0]
Epoch 0: 27%|##7 | 6/22 [00:05<00:13, 1.15it/s, loss=6.09, v_num=0]
Epoch 0: 32%|###1 | 7/22 [00:05<00:11, 1.31it/s, loss=6.09, v_num=0]
Epoch 0: 32%|###1 | 7/22 [00:05<00:11, 1.30it/s, loss=6.05, v_num=0]
Epoch 0: 36%|###6 | 8/22 [00:05<00:09, 1.46it/s, loss=6.05, v_num=0]
Epoch 0: 36%|###6 | 8/22 [00:05<00:09, 1.44it/s, loss=6.01, v_num=0]
Epoch 0: 41%|#### | 9/22 [00:05<00:08, 1.58it/s, loss=6.01, v_num=0]
Epoch 0: 41%|#### | 9/22 [00:05<00:08, 1.57it/s, loss=5.99, v_num=0]
Epoch 0: 45%|####5 | 10/22 [00:05<00:06, 1.72it/s, loss=5.99, v_num=0]
Epoch 0: 45%|####5 | 10/22 [00:05<00:07, 1.70it/s, loss=5.96, v_num=0]
Epoch 0: 50%|##### | 11/22 [00:05<00:05, 1.84it/s, loss=5.96, v_num=0]
Epoch 0: 50%|##### | 11/22 [00:06<00:06, 1.82it/s, loss=5.95, v_num=0]
Epoch 0: 55%|#####4 | 12/22 [00:06<00:05, 1.96it/s, loss=5.95, v_num=0]
Epoch 0: 55%|#####4 | 12/22 [00:06<00:05, 1.94it/s, loss=5.92, v_num=0]
Epoch 0: 59%|#####9 | 13/22 [00:06<00:04, 2.07it/s, loss=5.92, v_num=0]
Epoch 0: 59%|#####9 | 13/22 [00:06<00:04, 2.05it/s, loss=5.9, v_num=0]
Epoch 0: 64%|######3 | 14/22 [00:06<00:03, 2.18it/s, loss=5.9, v_num=0]
Epoch 0: 64%|######3 | 14/22 [00:06<00:03, 2.16it/s, loss=5.87, v_num=0]
Epoch 0: 68%|######8 | 15/22 [00:06<00:03, 2.28it/s, loss=5.87, v_num=0]
Epoch 0: 68%|######8 | 15/22 [00:06<00:03, 2.26it/s, loss=5.86, v_num=0]
Epoch 0: 73%|#######2 | 16/22 [00:06<00:02, 2.38it/s, loss=5.86, v_num=0]
Epoch 0: 73%|#######2 | 16/22 [00:06<00:02, 2.36it/s, loss=5.84, v_num=0]
Epoch 0: 77%|#######7 | 17/22 [00:06<00:02, 2.48it/s, loss=5.84, v_num=0]
Epoch 0: 77%|#######7 | 17/22 [00:06<00:02, 2.46it/s, loss=5.82, v_num=0]
Epoch 0: 82%|########1 | 18/22 [00:07<00:01, 2.57it/s, loss=5.82, v_num=0]
Epoch 0: 82%|########1 | 18/22 [00:07<00:01, 2.55it/s, loss=5.81, v_num=0]
Epoch 0: 86%|########6 | 19/22 [00:07<00:01, 2.66it/s, loss=5.81, v_num=0]
Epoch 0: 86%|########6 | 19/22 [00:07<00:01, 2.63it/s, loss=5.8, v_num=0]
Epoch 0: 91%|######### | 20/22 [00:07<00:00, 2.74it/s, loss=5.8, v_num=0]
Epoch 0: 91%|######### | 20/22 [00:07<00:00, 2.72it/s, loss=5.79, v_num=0]
Epoch 0: 95%|#########5| 21/22 [00:07<00:00, 2.82it/s, loss=5.79, v_num=0]
Epoch 0: 95%|#########5| 21/22 [00:07<00:00, 2.80it/s, loss=5.76, v_num=0]
Epoch 0: 100%|##########| 22/22 [00:07<00:00, 2.90it/s, loss=5.76, v_num=0]
Epoch 0: 100%|##########| 22/22 [00:07<00:00, 2.87it/s, loss=5.72, v_num=0]
Epoch 0: 100%|##########| 22/22 [00:07<00:00, 2.87it/s, loss=5.72, v_num=0]
Epoch 0: 0%| | 0/22 [00:00<?, ?it/s, loss=5.72, v_num=0]
Epoch 1: 0%| | 0/22 [00:00<?, ?it/s, loss=5.72, v_num=0]
Epoch 1: 5%|4 | 1/22 [00:02<00:42, 2.04s/it, loss=5.72, v_num=0]
Epoch 1: 5%|4 | 1/22 [00:02<00:43, 2.09s/it, loss=5.69, v_num=0]
Epoch 1: 9%|9 | 2/22 [00:02<00:21, 1.10s/it, loss=5.69, v_num=0]
Epoch 1: 9%|9 | 2/22 [00:02<00:22, 1.13s/it, loss=5.67, v_num=0]
Epoch 1: 14%|#3 | 3/22 [00:02<00:14, 1.28it/s, loss=5.67, v_num=0]
Epoch 1: 14%|#3 | 3/22 [00:02<00:15, 1.25it/s, loss=5.64, v_num=0]
Epoch 1: 18%|#8 | 4/22 [00:02<00:11, 1.61it/s, loss=5.64, v_num=0]
Epoch 1: 18%|#8 | 4/22 [00:02<00:11, 1.57it/s, loss=5.62, v_num=0]
Epoch 1: 23%|##2 | 5/22 [00:02<00:08, 1.89it/s, loss=5.62, v_num=0]
Epoch 1: 23%|##2 | 5/22 [00:02<00:09, 1.85it/s, loss=5.61, v_num=0]
Epoch 1: 27%|##7 | 6/22 [00:02<00:07, 2.15it/s, loss=5.61, v_num=0]
Epoch 1: 27%|##7 | 6/22 [00:02<00:07, 2.10it/s, loss=5.59, v_num=0]
Epoch 1: 32%|###1 | 7/22 [00:02<00:06, 2.38it/s, loss=5.59, v_num=0]
Epoch 1: 32%|###1 | 7/22 [00:03<00:06, 2.33it/s, loss=5.58, v_num=0]
Epoch 1: 36%|###6 | 8/22 [00:03<00:05, 2.59it/s, loss=5.58, v_num=0]
Epoch 1: 36%|###6 | 8/22 [00:03<00:05, 2.54it/s, loss=5.57, v_num=0]
Epoch 1: 41%|#### | 9/22 [00:03<00:04, 2.70it/s, loss=5.57, v_num=0]
Epoch 1: 41%|#### | 9/22 [00:03<00:04, 2.65it/s, loss=5.55, v_num=0]
Epoch 1: 45%|####5 | 10/22 [00:03<00:04, 2.86it/s, loss=5.55, v_num=0]
Epoch 1: 45%|####5 | 10/22 [00:03<00:04, 2.81it/s, loss=5.55, v_num=0]
Epoch 1: 50%|##### | 11/22 [00:03<00:03, 3.02it/s, loss=5.55, v_num=0]
Epoch 1: 50%|##### | 11/22 [00:03<00:03, 2.97it/s, loss=5.54, v_num=0]
Epoch 1: 55%|#####4 | 12/22 [00:03<00:03, 3.17it/s, loss=5.54, v_num=0]
Epoch 1: 55%|#####4 | 12/22 [00:03<00:03, 3.12it/s, loss=5.54, v_num=0]
Epoch 1: 59%|#####9 | 13/22 [00:03<00:02, 3.30it/s, loss=5.54, v_num=0]
Epoch 1: 59%|#####9 | 13/22 [00:03<00:02, 3.25it/s, loss=5.53, v_num=0]
Epoch 1: 64%|######3 | 14/22 [00:04<00:02, 3.43it/s, loss=5.53, v_num=0]
Epoch 1: 64%|######3 | 14/22 [00:04<00:02, 3.38it/s, loss=5.53, v_num=0]
Epoch 1: 68%|######8 | 15/22 [00:04<00:01, 3.54it/s, loss=5.53, v_num=0]
Epoch 1: 68%|######8 | 15/22 [00:04<00:02, 3.50it/s, loss=5.52, v_num=0]
Epoch 1: 73%|#######2 | 16/22 [00:04<00:01, 3.65it/s, loss=5.52, v_num=0]
Epoch 1: 73%|#######2 | 16/22 [00:04<00:01, 3.60it/s, loss=5.52, v_num=0]
Epoch 1: 77%|#######7 | 17/22 [00:04<00:01, 3.70it/s, loss=5.52, v_num=0]
Epoch 1: 77%|#######7 | 17/22 [00:04<00:01, 3.65it/s, loss=5.51, v_num=0]
Epoch 1: 82%|########1 | 18/22 [00:04<00:01, 3.79it/s, loss=5.51, v_num=0]
Epoch 1: 82%|########1 | 18/22 [00:04<00:01, 3.74it/s, loss=5.5, v_num=0]
Epoch 1: 86%|########6 | 19/22 [00:04<00:00, 3.88it/s, loss=5.5, v_num=0]
Epoch 1: 86%|########6 | 19/22 [00:04<00:00, 3.83it/s, loss=5.5, v_num=0]
Epoch 1: 91%|######### | 20/22 [00:05<00:00, 3.96it/s, loss=5.5, v_num=0]
Epoch 1: 91%|######### | 20/22 [00:05<00:00, 3.92it/s, loss=5.5, v_num=0]
Epoch 1: 95%|#########5| 21/22 [00:05<00:00, 4.04it/s, loss=5.5, v_num=0]
Epoch 1: 95%|#########5| 21/22 [00:05<00:00, 4.00it/s, loss=5.49, v_num=0]
Epoch 1: 100%|##########| 22/22 [00:05<00:00, 4.12it/s, loss=5.49, v_num=0]
Epoch 1: 100%|##########| 22/22 [00:05<00:00, 4.07it/s, loss=5.49, v_num=0]
Epoch 1: 100%|##########| 22/22 [00:05<00:00, 4.07it/s, loss=5.49, v_num=0]
Epoch 1: 0%| | 0/22 [00:00<?, ?it/s, loss=5.49, v_num=0]
Epoch 2: 0%| | 0/22 [00:00<?, ?it/s, loss=5.49, v_num=0]
Epoch 2: 5%|4 | 1/22 [00:02<00:43, 2.09s/it, loss=5.49, v_num=0]
Epoch 2: 5%|4 | 1/22 [00:02<00:45, 2.15s/it, loss=5.48, v_num=0]
Epoch 2: 9%|9 | 2/22 [00:02<00:22, 1.12s/it, loss=5.48, v_num=0]
Epoch 2: 9%|9 | 2/22 [00:02<00:23, 1.15s/it, loss=5.48, v_num=0]
Epoch 2: 14%|#3 | 3/22 [00:02<00:15, 1.25it/s, loss=5.48, v_num=0]
Epoch 2: 14%|#3 | 3/22 [00:02<00:15, 1.22it/s, loss=5.48, v_num=0]
Epoch 2: 18%|#8 | 4/22 [00:02<00:11, 1.57it/s, loss=5.48, v_num=0]
Epoch 2: 18%|#8 | 4/22 [00:02<00:11, 1.53it/s, loss=5.47, v_num=0]
Epoch 2: 23%|##2 | 5/22 [00:02<00:09, 1.85it/s, loss=5.47, v_num=0]
Epoch 2: 23%|##2 | 5/22 [00:02<00:09, 1.81it/s, loss=5.47, v_num=0]
Epoch 2: 27%|##7 | 6/22 [00:02<00:07, 2.11it/s, loss=5.47, v_num=0]
Epoch 2: 27%|##7 | 6/22 [00:02<00:07, 2.06it/s, loss=5.46, v_num=0]
Epoch 2: 32%|###1 | 7/22 [00:02<00:06, 2.34it/s, loss=5.46, v_num=0]
Epoch 2: 32%|###1 | 7/22 [00:03<00:06, 2.29it/s, loss=5.46, v_num=0]
Epoch 2: 36%|###6 | 8/22 [00:03<00:05, 2.54it/s, loss=5.46, v_num=0]
Epoch 2: 36%|###6 | 8/22 [00:03<00:05, 2.49it/s, loss=5.45, v_num=0]
Epoch 2: 41%|#### | 9/22 [00:03<00:04, 2.60it/s, loss=5.45, v_num=0]
Epoch 2: 41%|#### | 9/22 [00:03<00:05, 2.56it/s, loss=5.45, v_num=0]
Epoch 2: 45%|####5 | 10/22 [00:03<00:04, 2.77it/s, loss=5.45, v_num=0]
Epoch 2: 45%|####5 | 10/22 [00:03<00:04, 2.72it/s, loss=5.44, v_num=0]
Epoch 2: 50%|##### | 11/22 [00:03<00:03, 2.92it/s, loss=5.44, v_num=0]
Epoch 2: 50%|##### | 11/22 [00:03<00:03, 2.88it/s, loss=5.44, v_num=0]
Epoch 2: 55%|#####4 | 12/22 [00:03<00:03, 3.07it/s, loss=5.44, v_num=0]
Epoch 2: 55%|#####4 | 12/22 [00:03<00:03, 3.02it/s, loss=5.43, v_num=0]
Epoch 2: 59%|#####9 | 13/22 [00:04<00:02, 3.21it/s, loss=5.43, v_num=0]
Epoch 2: 59%|#####9 | 13/22 [00:04<00:02, 3.16it/s, loss=5.42, v_num=0]
Epoch 2: 64%|######3 | 14/22 [00:04<00:02, 3.33it/s, loss=5.42, v_num=0]
Epoch 2: 64%|######3 | 14/22 [00:04<00:02, 3.28it/s, loss=5.42, v_num=0]
Epoch 2: 68%|######8 | 15/22 [00:04<00:02, 3.45it/s, loss=5.42, v_num=0]
Epoch 2: 68%|######8 | 15/22 [00:04<00:02, 3.40it/s, loss=5.42, v_num=0]
Epoch 2: 73%|#######2 | 16/22 [00:04<00:01, 3.56it/s, loss=5.42, v_num=0]
Epoch 2: 73%|#######2 | 16/22 [00:04<00:01, 3.51it/s, loss=5.41, v_num=0]
Epoch 2: 77%|#######7 | 17/22 [00:04<00:01, 3.56it/s, loss=5.41, v_num=0]
Epoch 2: 77%|#######7 | 17/22 [00:04<00:01, 3.52it/s, loss=5.41, v_num=0]
Epoch 2: 82%|########1 | 18/22 [00:04<00:01, 3.66it/s, loss=5.41, v_num=0]
Epoch 2: 82%|########1 | 18/22 [00:04<00:01, 3.61it/s, loss=5.4, v_num=0]
Epoch 2: 86%|########6 | 19/22 [00:05<00:00, 3.75it/s, loss=5.4, v_num=0]
Epoch 2: 86%|########6 | 19/22 [00:05<00:00, 3.70it/s, loss=5.39, v_num=0]
Epoch 2: 91%|######### | 20/22 [00:05<00:00, 3.83it/s, loss=5.39, v_num=0]
Epoch 2: 91%|######### | 20/22 [00:05<00:00, 3.79it/s, loss=5.39, v_num=0]
Epoch 2: 95%|#########5| 21/22 [00:05<00:00, 3.92it/s, loss=5.39, v_num=0]
Epoch 2: 95%|#########5| 21/22 [00:05<00:00, 3.87it/s, loss=5.38, v_num=0]
Epoch 2: 100%|##########| 22/22 [00:05<00:00, 3.99it/s, loss=5.38, v_num=0]
Epoch 2: 100%|##########| 22/22 [00:05<00:00, 3.95it/s, loss=5.38, v_num=0]
Epoch 2: 100%|##########| 22/22 [00:05<00:00, 3.95it/s, loss=5.38, v_num=0]
Epoch 2: 0%| | 0/22 [00:00<?, ?it/s, loss=5.38, v_num=0]
Epoch 3: 0%| | 0/22 [00:00<?, ?it/s, loss=5.38, v_num=0]
Epoch 3: 5%|4 | 1/22 [00:02<00:44, 2.13s/it, loss=5.38, v_num=0]
Epoch 3: 5%|4 | 1/22 [00:02<00:46, 2.19s/it, loss=5.38, v_num=0]
Epoch 3: 9%|9 | 2/22 [00:02<00:22, 1.15s/it, loss=5.38, v_num=0]
Epoch 3: 9%|9 | 2/22 [00:02<00:23, 1.18s/it, loss=5.38, v_num=0]
Epoch 3: 14%|#3 | 3/22 [00:02<00:15, 1.23it/s, loss=5.38, v_num=0]
Epoch 3: 14%|#3 | 3/22 [00:02<00:15, 1.20it/s, loss=5.37, v_num=0]
Epoch 3: 18%|#8 | 4/22 [00:02<00:11, 1.54it/s, loss=5.37, v_num=0]
Epoch 3: 18%|#8 | 4/22 [00:02<00:11, 1.51it/s, loss=5.38, v_num=0]
Epoch 3: 23%|##2 | 5/22 [00:02<00:09, 1.82it/s, loss=5.38, v_num=0]
Epoch 3: 23%|##2 | 5/22 [00:02<00:09, 1.78it/s, loss=5.37, v_num=0]
Epoch 3: 27%|##7 | 6/22 [00:02<00:07, 2.08it/s, loss=5.37, v_num=0]
Epoch 3: 27%|##7 | 6/22 [00:02<00:07, 2.03it/s, loss=5.37, v_num=0]
Epoch 3: 32%|###1 | 7/22 [00:03<00:06, 2.30it/s, loss=5.37, v_num=0]
Epoch 3: 32%|###1 | 7/22 [00:03<00:06, 2.26it/s, loss=5.37, v_num=0]
Epoch 3: 36%|###6 | 8/22 [00:03<00:05, 2.50it/s, loss=5.37, v_num=0]
Epoch 3: 36%|###6 | 8/22 [00:03<00:05, 2.46it/s, loss=5.37, v_num=0]
Epoch 3: 41%|#### | 9/22 [00:03<00:05, 2.57it/s, loss=5.37, v_num=0]
Epoch 3: 41%|#### | 9/22 [00:03<00:05, 2.53it/s, loss=5.36, v_num=0]
Epoch 3: 45%|####5 | 10/22 [00:03<00:04, 2.74it/s, loss=5.36, v_num=0]
Epoch 3: 45%|####5 | 10/22 [00:03<00:04, 2.69it/s, loss=5.36, v_num=0]
Epoch 3: 50%|##### | 11/22 [00:03<00:03, 2.90it/s, loss=5.36, v_num=0]
Epoch 3: 50%|##### | 11/22 [00:03<00:03, 2.85it/s, loss=5.36, v_num=0]
Epoch 3: 55%|#####4 | 12/22 [00:03<00:03, 3.04it/s, loss=5.36, v_num=0]
Epoch 3: 55%|#####4 | 12/22 [00:04<00:03, 2.99it/s, loss=5.36, v_num=0]
Epoch 3: 59%|#####9 | 13/22 [00:04<00:02, 3.17it/s, loss=5.36, v_num=0]
Epoch 3: 59%|#####9 | 13/22 [00:04<00:02, 3.13it/s, loss=5.35, v_num=0]
Epoch 3: 64%|######3 | 14/22 [00:04<00:02, 3.30it/s, loss=5.35, v_num=0]
Epoch 3: 64%|######3 | 14/22 [00:04<00:02, 3.25it/s, loss=5.35, v_num=0]
Epoch 3: 68%|######8 | 15/22 [00:04<00:02, 3.42it/s, loss=5.35, v_num=0]
Epoch 3: 68%|######8 | 15/22 [00:04<00:02, 3.37it/s, loss=5.34, v_num=0]
Epoch 3: 73%|#######2 | 16/22 [00:04<00:01, 3.52it/s, loss=5.34, v_num=0]
Epoch 3: 73%|#######2 | 16/22 [00:04<00:01, 3.48it/s, loss=5.35, v_num=0]
Epoch 3: 77%|#######7 | 17/22 [00:04<00:01, 3.53it/s, loss=5.35, v_num=0]
Epoch 3: 77%|#######7 | 17/22 [00:04<00:01, 3.49it/s, loss=5.34, v_num=0]
Epoch 3: 82%|########1 | 18/22 [00:04<00:01, 3.63it/s, loss=5.34, v_num=0]
Epoch 3: 82%|########1 | 18/22 [00:05<00:01, 3.58it/s, loss=5.34, v_num=0]
Epoch 3: 86%|########6 | 19/22 [00:05<00:00, 3.72it/s, loss=5.34, v_num=0]
Epoch 3: 86%|########6 | 19/22 [00:05<00:00, 3.67it/s, loss=5.34, v_num=0]
Epoch 3: 91%|######### | 20/22 [00:05<00:00, 3.80it/s, loss=5.34, v_num=0]
Epoch 3: 91%|######### | 20/22 [00:05<00:00, 3.76it/s, loss=5.33, v_num=0]
Epoch 3: 95%|#########5| 21/22 [00:05<00:00, 3.88it/s, loss=5.33, v_num=0]
Epoch 3: 95%|#########5| 21/22 [00:05<00:00, 3.84it/s, loss=5.33, v_num=0]
Epoch 3: 100%|##########| 22/22 [00:05<00:00, 3.96it/s, loss=5.33, v_num=0]
Epoch 3: 100%|##########| 22/22 [00:05<00:00, 3.91it/s, loss=5.32, v_num=0]
Epoch 3: 100%|##########| 22/22 [00:05<00:00, 3.91it/s, loss=5.32, v_num=0]
Epoch 3: 0%| | 0/22 [00:00<?, ?it/s, loss=5.32, v_num=0]
Epoch 4: 0%| | 0/22 [00:00<?, ?it/s, loss=5.32, v_num=0]
Epoch 4: 5%|4 | 1/22 [00:02<00:45, 2.15s/it, loss=5.32, v_num=0]
Epoch 4: 5%|4 | 1/22 [00:02<00:46, 2.21s/it, loss=5.32, v_num=0]
Epoch 4: 9%|9 | 2/22 [00:02<00:23, 1.16s/it, loss=5.32, v_num=0]
Epoch 4: 9%|9 | 2/22 [00:02<00:23, 1.19s/it, loss=5.31, v_num=0]
Epoch 4: 14%|#3 | 3/22 [00:02<00:15, 1.22it/s, loss=5.31, v_num=0]
Epoch 4: 14%|#3 | 3/22 [00:02<00:15, 1.19it/s, loss=5.31, v_num=0]
Epoch 4: 18%|#8 | 4/22 [00:02<00:11, 1.53it/s, loss=5.31, v_num=0]
Epoch 4: 18%|#8 | 4/22 [00:02<00:12, 1.50it/s, loss=5.3, v_num=0]
Epoch 4: 23%|##2 | 5/22 [00:02<00:09, 1.81it/s, loss=5.3, v_num=0]
Epoch 4: 23%|##2 | 5/22 [00:02<00:09, 1.77it/s, loss=5.31, v_num=0]
Epoch 4: 27%|##7 | 6/22 [00:02<00:07, 2.06it/s, loss=5.31, v_num=0]
Epoch 4: 27%|##7 | 6/22 [00:02<00:07, 2.02it/s, loss=5.3, v_num=0]
Epoch 4: 32%|###1 | 7/22 [00:03<00:06, 2.29it/s, loss=5.3, v_num=0]
Epoch 4: 32%|###1 | 7/22 [00:03<00:06, 2.24it/s, loss=5.31, v_num=0]
Epoch 4: 36%|###6 | 8/22 [00:03<00:05, 2.49it/s, loss=5.31, v_num=0]
Epoch 4: 36%|###6 | 8/22 [00:03<00:05, 2.44it/s, loss=5.3, v_num=0]
Epoch 4: 41%|#### | 9/22 [00:03<00:05, 2.55it/s, loss=5.3, v_num=0]
Epoch 4: 41%|#### | 9/22 [00:03<00:05, 2.51it/s, loss=5.3, v_num=0]
Epoch 4: 45%|####5 | 10/22 [00:03<00:04, 2.72it/s, loss=5.3, v_num=0]
Epoch 4: 45%|####5 | 10/22 [00:03<00:04, 2.67it/s, loss=5.3, v_num=0]
Epoch 4: 50%|##### | 11/22 [00:03<00:03, 2.87it/s, loss=5.3, v_num=0]
Epoch 4: 50%|##### | 11/22 [00:03<00:03, 2.83it/s, loss=5.3, v_num=0]
Epoch 4: 55%|#####4 | 12/22 [00:03<00:03, 3.01it/s, loss=5.3, v_num=0]
Epoch 4: 55%|#####4 | 12/22 [00:04<00:03, 2.97it/s, loss=5.29, v_num=0]
Epoch 4: 59%|#####9 | 13/22 [00:04<00:02, 3.14it/s, loss=5.29, v_num=0]
Epoch 4: 59%|#####9 | 13/22 [00:04<00:02, 3.10it/s, loss=5.29, v_num=0]
Epoch 4: 64%|######3 | 14/22 [00:04<00:02, 3.27it/s, loss=5.29, v_num=0]
Epoch 4: 64%|######3 | 14/22 [00:04<00:02, 3.22it/s, loss=5.29, v_num=0]
Epoch 4: 68%|######8 | 15/22 [00:04<00:02, 3.38it/s, loss=5.29, v_num=0]
Epoch 4: 68%|######8 | 15/22 [00:04<00:02, 3.34it/s, loss=5.29, v_num=0]
Epoch 4: 73%|#######2 | 16/22 [00:04<00:01, 3.49it/s, loss=5.29, v_num=0]
Epoch 4: 73%|#######2 | 16/22 [00:04<00:01, 3.45it/s, loss=5.29, v_num=0]
Epoch 4: 77%|#######7 | 17/22 [00:04<00:01, 3.50it/s, loss=5.29, v_num=0]
Epoch 4: 77%|#######7 | 17/22 [00:04<00:01, 3.46it/s, loss=5.28, v_num=0]
Epoch 4: 82%|########1 | 18/22 [00:05<00:01, 3.59it/s, loss=5.28, v_num=0]
Epoch 4: 82%|########1 | 18/22 [00:05<00:01, 3.55it/s, loss=5.29, v_num=0]
Epoch 4: 86%|########6 | 19/22 [00:05<00:00, 3.68it/s, loss=5.29, v_num=0]
Epoch 4: 86%|########6 | 19/22 [00:05<00:00, 3.64it/s, loss=5.28, v_num=0]
Epoch 4: 91%|######### | 20/22 [00:05<00:00, 3.77it/s, loss=5.28, v_num=0]
Epoch 4: 91%|######### | 20/22 [00:05<00:00, 3.73it/s, loss=5.28, v_num=0]
Epoch 4: 95%|#########5| 21/22 [00:05<00:00, 3.85it/s, loss=5.28, v_num=0]
Epoch 4: 95%|#########5| 21/22 [00:05<00:00, 3.81it/s, loss=5.28, v_num=0]
Epoch 4: 100%|##########| 22/22 [00:05<00:00, 3.93it/s, loss=5.28, v_num=0]
Epoch 4: 100%|##########| 22/22 [00:05<00:00, 3.89it/s, loss=5.28, v_num=0]
Epoch 4: 100%|##########| 22/22 [00:05<00:00, 3.89it/s, loss=5.28, v_num=0]
Epoch 4: 0%| | 0/22 [00:00<?, ?it/s, loss=5.28, v_num=0]
Epoch 5: 0%| | 0/22 [00:00<?, ?it/s, loss=5.28, v_num=0]
Epoch 5: 5%|4 | 1/22 [00:02<00:43, 2.07s/it, loss=5.28, v_num=0]
Epoch 5: 5%|4 | 1/22 [00:02<00:44, 2.13s/it, loss=5.28, v_num=0]
Epoch 5: 9%|9 | 2/22 [00:02<00:22, 1.11s/it, loss=5.28, v_num=0]
Epoch 5: 9%|9 | 2/22 [00:02<00:22, 1.14s/it, loss=5.28, v_num=0]
Epoch 5: 14%|#3 | 3/22 [00:02<00:15, 1.26it/s, loss=5.28, v_num=0]
Epoch 5: 14%|#3 | 3/22 [00:02<00:15, 1.23it/s, loss=5.28, v_num=0]
Epoch 5: 18%|#8 | 4/22 [00:02<00:11, 1.58it/s, loss=5.28, v_num=0]
Epoch 5: 18%|#8 | 4/22 [00:02<00:11, 1.55it/s, loss=5.28, v_num=0]
Epoch 5: 23%|##2 | 5/22 [00:02<00:09, 1.87it/s, loss=5.28, v_num=0]
Epoch 5: 23%|##2 | 5/22 [00:02<00:09, 1.83it/s, loss=5.27, v_num=0]
Epoch 5: 27%|##7 | 6/22 [00:02<00:07, 2.12it/s, loss=5.27, v_num=0]
Epoch 5: 27%|##7 | 6/22 [00:02<00:07, 2.08it/s, loss=5.28, v_num=0]
Epoch 5: 32%|###1 | 7/22 [00:02<00:06, 2.35it/s, loss=5.28, v_num=0]
Epoch 5: 32%|###1 | 7/22 [00:03<00:06, 2.30it/s, loss=5.27, v_num=0]
Epoch 5: 36%|###6 | 8/22 [00:03<00:05, 2.55it/s, loss=5.27, v_num=0]
Epoch 5: 36%|###6 | 8/22 [00:03<00:05, 2.51it/s, loss=5.28, v_num=0]
Epoch 5: 41%|#### | 9/22 [00:03<00:04, 2.63it/s, loss=5.28, v_num=0]
Epoch 5: 41%|#### | 9/22 [00:03<00:05, 2.59it/s, loss=5.27, v_num=0]
Epoch 5: 45%|####5 | 10/22 [00:03<00:04, 2.80it/s, loss=5.27, v_num=0]
Epoch 5: 45%|####5 | 10/22 [00:03<00:04, 2.75it/s, loss=5.26, v_num=0]
Epoch 5: 50%|##### | 11/22 [00:03<00:03, 2.96it/s, loss=5.26, v_num=0]
Epoch 5: 50%|##### | 11/22 [00:03<00:03, 2.91it/s, loss=5.26, v_num=0]
Epoch 5: 55%|#####4 | 12/22 [00:03<00:03, 3.10it/s, loss=5.26, v_num=0]
Epoch 5: 55%|#####4 | 12/22 [00:03<00:03, 3.05it/s, loss=5.26, v_num=0]
Epoch 5: 59%|#####9 | 13/22 [00:04<00:02, 3.24it/s, loss=5.26, v_num=0]
Epoch 5: 59%|#####9 | 13/22 [00:04<00:02, 3.19it/s, loss=5.26, v_num=0]
Epoch 5: 64%|######3 | 14/22 [00:04<00:02, 3.36it/s, loss=5.26, v_num=0]
Epoch 5: 64%|######3 | 14/22 [00:04<00:02, 3.31it/s, loss=5.26, v_num=0]
Epoch 5: 68%|######8 | 15/22 [00:04<00:02, 3.48it/s, loss=5.26, v_num=0]
Epoch 5: 68%|######8 | 15/22 [00:04<00:02, 3.43it/s, loss=5.26, v_num=0]
Epoch 5: 73%|#######2 | 16/22 [00:04<00:01, 3.59it/s, loss=5.26, v_num=0]
Epoch 5: 73%|#######2 | 16/22 [00:04<00:01, 3.54it/s, loss=5.26, v_num=0]
Epoch 5: 77%|#######7 | 17/22 [00:04<00:01, 3.60it/s, loss=5.26, v_num=0]
Epoch 5: 77%|#######7 | 17/22 [00:04<00:01, 3.56it/s, loss=5.26, v_num=0]
Epoch 5: 82%|########1 | 18/22 [00:04<00:01, 3.70it/s, loss=5.26, v_num=0]
Epoch 5: 82%|########1 | 18/22 [00:04<00:01, 3.65it/s, loss=5.26, v_num=0]
Epoch 5: 86%|########6 | 19/22 [00:05<00:00, 3.78it/s, loss=5.26, v_num=0]
Epoch 5: 86%|########6 | 19/22 [00:05<00:00, 3.74it/s, loss=5.25, v_num=0]
Epoch 5: 91%|######### | 20/22 [00:05<00:00, 3.87it/s, loss=5.25, v_num=0]
Epoch 5: 91%|######### | 20/22 [00:05<00:00, 3.82it/s, loss=5.25, v_num=0]
Epoch 5: 95%|#########5| 21/22 [00:05<00:00, 3.95it/s, loss=5.25, v_num=0]
Epoch 5: 95%|#########5| 21/22 [00:05<00:00, 3.91it/s, loss=5.25, v_num=0]
Epoch 5: 100%|##########| 22/22 [00:05<00:00, 4.03it/s, loss=5.25, v_num=0]
Epoch 5: 100%|##########| 22/22 [00:05<00:00, 3.98it/s, loss=5.25, v_num=0]
Epoch 5: 100%|##########| 22/22 [00:05<00:00, 3.98it/s, loss=5.25, v_num=0]
Epoch 5: 0%| | 0/22 [00:00<?, ?it/s, loss=5.25, v_num=0]
Epoch 6: 0%| | 0/22 [00:00<?, ?it/s, loss=5.25, v_num=0]
Epoch 6: 5%|4 | 1/22 [00:02<00:45, 2.16s/it, loss=5.25, v_num=0]
Epoch 6: 5%|4 | 1/22 [00:02<00:46, 2.22s/it, loss=5.24, v_num=0]
Epoch 6: 9%|9 | 2/22 [00:02<00:23, 1.16s/it, loss=5.24, v_num=0]
Epoch 6: 9%|9 | 2/22 [00:02<00:23, 1.19s/it, loss=5.24, v_num=0]
Epoch 6: 14%|#3 | 3/22 [00:02<00:15, 1.21it/s, loss=5.24, v_num=0]
Epoch 6: 14%|#3 | 3/22 [00:02<00:16, 1.19it/s, loss=5.23, v_num=0]
Epoch 6: 18%|#8 | 4/22 [00:02<00:11, 1.52it/s, loss=5.23, v_num=0]
Epoch 6: 18%|#8 | 4/22 [00:02<00:12, 1.49it/s, loss=5.23, v_num=0]
Epoch 6: 23%|##2 | 5/22 [00:02<00:09, 1.80it/s, loss=5.23, v_num=0]
Epoch 6: 23%|##2 | 5/22 [00:02<00:09, 1.76it/s, loss=5.23, v_num=0]
Epoch 6: 27%|##7 | 6/22 [00:02<00:07, 2.05it/s, loss=5.23, v_num=0]
Epoch 6: 27%|##7 | 6/22 [00:02<00:07, 2.01it/s, loss=5.22, v_num=0]
Epoch 6: 32%|###1 | 7/22 [00:03<00:06, 2.28it/s, loss=5.22, v_num=0]
Epoch 6: 32%|###1 | 7/22 [00:03<00:06, 2.23it/s, loss=5.22, v_num=0]
Epoch 6: 36%|###6 | 8/22 [00:03<00:05, 2.48it/s, loss=5.22, v_num=0]
Epoch 6: 36%|###6 | 8/22 [00:03<00:05, 2.44it/s, loss=5.22, v_num=0]
Epoch 6: 41%|#### | 9/22 [00:03<00:05, 2.57it/s, loss=5.22, v_num=0]
Epoch 6: 41%|#### | 9/22 [00:03<00:05, 2.53it/s, loss=5.22, v_num=0]
Epoch 6: 45%|####5 | 10/22 [00:03<00:04, 2.73it/s, loss=5.22, v_num=0]
Epoch 6: 45%|####5 | 10/22 [00:03<00:04, 2.69it/s, loss=5.22, v_num=0]
Epoch 6: 50%|##### | 11/22 [00:03<00:03, 2.89it/s, loss=5.22, v_num=0]
Epoch 6: 50%|##### | 11/22 [00:03<00:03, 2.84it/s, loss=5.21, v_num=0]
Epoch 6: 55%|#####4 | 12/22 [00:03<00:03, 3.03it/s, loss=5.21, v_num=0]
Epoch 6: 55%|#####4 | 12/22 [00:04<00:03, 2.99it/s, loss=5.21, v_num=0]
Epoch 6: 59%|#####9 | 13/22 [00:04<00:02, 3.17it/s, loss=5.21, v_num=0]
Epoch 6: 59%|#####9 | 13/22 [00:04<00:02, 3.12it/s, loss=5.21, v_num=0]
Epoch 6: 64%|######3 | 14/22 [00:04<00:02, 3.29it/s, loss=5.21, v_num=0]
Epoch 6: 64%|######3 | 14/22 [00:04<00:02, 3.24it/s, loss=5.22, v_num=0]
Epoch 6: 68%|######8 | 15/22 [00:04<00:02, 3.40it/s, loss=5.22, v_num=0]
Epoch 6: 68%|######8 | 15/22 [00:04<00:02, 3.36it/s, loss=5.21, v_num=0]
Epoch 6: 73%|#######2 | 16/22 [00:04<00:01, 3.51it/s, loss=5.21, v_num=0]
Epoch 6: 73%|#######2 | 16/22 [00:04<00:01, 3.47it/s, loss=5.21, v_num=0]
Epoch 6: 77%|#######7 | 17/22 [00:04<00:01, 3.54it/s, loss=5.21, v_num=0]
Epoch 6: 77%|#######7 | 17/22 [00:04<00:01, 3.49it/s, loss=5.21, v_num=0]
Epoch 6: 82%|########1 | 18/22 [00:04<00:01, 3.63it/s, loss=5.21, v_num=0]
Epoch 6: 82%|########1 | 18/22 [00:05<00:01, 3.59it/s, loss=5.2, v_num=0]
Epoch 6: 86%|########6 | 19/22 [00:05<00:00, 3.72it/s, loss=5.2, v_num=0]
Epoch 6: 86%|########6 | 19/22 [00:05<00:00, 3.68it/s, loss=5.2, v_num=0]
Epoch 6: 91%|######### | 20/22 [00:05<00:00, 3.80it/s, loss=5.2, v_num=0]
Epoch 6: 91%|######### | 20/22 [00:05<00:00, 3.76it/s, loss=5.21, v_num=0]
Epoch 6: 95%|#########5| 21/22 [00:05<00:00, 3.88it/s, loss=5.21, v_num=0]
Epoch 6: 95%|#########5| 21/22 [00:05<00:00, 3.84it/s, loss=5.2, v_num=0]
Epoch 6: 100%|##########| 22/22 [00:05<00:00, 3.96it/s, loss=5.2, v_num=0]
Epoch 6: 100%|##########| 22/22 [00:05<00:00, 3.91it/s, loss=5.2, v_num=0]
Epoch 6: 100%|##########| 22/22 [00:05<00:00, 3.91it/s, loss=5.2, v_num=0]
Epoch 6: 0%| | 0/22 [00:00<?, ?it/s, loss=5.2, v_num=0]
Epoch 7: 0%| | 0/22 [00:00<?, ?it/s, loss=5.2, v_num=0]
Epoch 7: 5%|4 | 1/22 [00:02<00:45, 2.17s/it, loss=5.2, v_num=0]
Epoch 7: 5%|4 | 1/22 [00:02<00:46, 2.23s/it, loss=5.21, v_num=0]
Epoch 7: 9%|9 | 2/22 [00:02<00:23, 1.16s/it, loss=5.21, v_num=0]
Epoch 7: 9%|9 | 2/22 [00:02<00:23, 1.19s/it, loss=5.21, v_num=0]
Epoch 7: 14%|#3 | 3/22 [00:02<00:15, 1.21it/s, loss=5.21, v_num=0]
Epoch 7: 14%|#3 | 3/22 [00:02<00:16, 1.18it/s, loss=5.2, v_num=0]
Epoch 7: 18%|#8 | 4/22 [00:02<00:11, 1.52it/s, loss=5.2, v_num=0]
Epoch 7: 18%|#8 | 4/22 [00:02<00:12, 1.49it/s, loss=5.2, v_num=0]
Epoch 7: 23%|##2 | 5/22 [00:02<00:09, 1.80it/s, loss=5.2, v_num=0]
Epoch 7: 23%|##2 | 5/22 [00:02<00:09, 1.76it/s, loss=5.19, v_num=0]
Epoch 7: 27%|##7 | 6/22 [00:02<00:07, 2.04it/s, loss=5.19, v_num=0]
Epoch 7: 27%|##7 | 6/22 [00:02<00:07, 2.00it/s, loss=5.19, v_num=0]
Epoch 7: 32%|###1 | 7/22 [00:03<00:06, 2.27it/s, loss=5.19, v_num=0]
Epoch 7: 32%|###1 | 7/22 [00:03<00:06, 2.23it/s, loss=5.19, v_num=0]
Epoch 7: 36%|###6 | 8/22 [00:03<00:05, 2.47it/s, loss=5.19, v_num=0]
Epoch 7: 36%|###6 | 8/22 [00:03<00:05, 2.43it/s, loss=5.2, v_num=0]
Epoch 7: 41%|#### | 9/22 [00:03<00:05, 2.51it/s, loss=5.2, v_num=0]
Epoch 7: 41%|#### | 9/22 [00:03<00:05, 2.47it/s, loss=5.2, v_num=0]
Epoch 7: 45%|####5 | 10/22 [00:03<00:04, 2.67it/s, loss=5.2, v_num=0]
Epoch 7: 45%|####5 | 10/22 [00:03<00:04, 2.63it/s, loss=5.2, v_num=0]
Epoch 7: 50%|##### | 11/22 [00:03<00:03, 2.83it/s, loss=5.2, v_num=0]
Epoch 7: 50%|##### | 11/22 [00:03<00:03, 2.79it/s, loss=5.19, v_num=0]
Epoch 7: 55%|#####4 | 12/22 [00:04<00:03, 2.97it/s, loss=5.19, v_num=0]
Epoch 7: 55%|#####4 | 12/22 [00:04<00:03, 2.93it/s, loss=5.19, v_num=0]
Epoch 7: 59%|#####9 | 13/22 [00:04<00:02, 3.10it/s, loss=5.19, v_num=0]
Epoch 7: 59%|#####9 | 13/22 [00:04<00:02, 3.06it/s, loss=5.19, v_num=0]
Epoch 7: 64%|######3 | 14/22 [00:04<00:02, 3.22it/s, loss=5.19, v_num=0]
Epoch 7: 64%|######3 | 14/22 [00:04<00:02, 3.18it/s, loss=5.18, v_num=0]
Epoch 7: 68%|######8 | 15/22 [00:04<00:02, 3.34it/s, loss=5.18, v_num=0]
Epoch 7: 68%|######8 | 15/22 [00:04<00:02, 3.30it/s, loss=5.18, v_num=0]
Epoch 7: 73%|#######2 | 16/22 [00:04<00:01, 3.45it/s, loss=5.18, v_num=0]
Epoch 7: 73%|#######2 | 16/22 [00:04<00:01, 3.40it/s, loss=5.18, v_num=0]
Epoch 7: 77%|#######7 | 17/22 [00:04<00:01, 3.40it/s, loss=5.18, v_num=0]
Epoch 7: 77%|#######7 | 17/22 [00:05<00:01, 3.36it/s, loss=5.18, v_num=0]
Epoch 7: 82%|########1 | 18/22 [00:05<00:01, 3.50it/s, loss=5.18, v_num=0]
Epoch 7: 82%|########1 | 18/22 [00:05<00:01, 3.46it/s, loss=5.18, v_num=0]
Epoch 7: 86%|########6 | 19/22 [00:05<00:00, 3.59it/s, loss=5.18, v_num=0]
Epoch 7: 86%|########6 | 19/22 [00:05<00:00, 3.55it/s, loss=5.18, v_num=0]
Epoch 7: 91%|######### | 20/22 [00:05<00:00, 3.68it/s, loss=5.18, v_num=0]
Epoch 7: 91%|######### | 20/22 [00:05<00:00, 3.64it/s, loss=5.18, v_num=0]
Epoch 7: 95%|#########5| 21/22 [00:05<00:00, 3.76it/s, loss=5.18, v_num=0]
Epoch 7: 95%|#########5| 21/22 [00:05<00:00, 3.72it/s, loss=5.18, v_num=0]
Epoch 7: 100%|##########| 22/22 [00:05<00:00, 3.83it/s, loss=5.18, v_num=0]
Epoch 7: 100%|##########| 22/22 [00:05<00:00, 3.80it/s, loss=5.17, v_num=0]
Epoch 7: 100%|##########| 22/22 [00:05<00:00, 3.79it/s, loss=5.17, v_num=0]
Epoch 7: 0%| | 0/22 [00:00<?, ?it/s, loss=5.17, v_num=0]
Epoch 8: 0%| | 0/22 [00:00<?, ?it/s, loss=5.17, v_num=0]
Epoch 8: 5%|4 | 1/22 [00:02<00:45, 2.16s/it, loss=5.17, v_num=0]
Epoch 8: 5%|4 | 1/22 [00:02<00:46, 2.22s/it, loss=5.18, v_num=0]
Epoch 8: 9%|9 | 2/22 [00:02<00:23, 1.16s/it, loss=5.18, v_num=0]
Epoch 8: 9%|9 | 2/22 [00:02<00:23, 1.19s/it, loss=5.18, v_num=0]
Epoch 8: 14%|#3 | 3/22 [00:02<00:15, 1.21it/s, loss=5.18, v_num=0]
Epoch 8: 14%|#3 | 3/22 [00:02<00:16, 1.18it/s, loss=5.18, v_num=0]
Epoch 8: 18%|#8 | 4/22 [00:02<00:11, 1.52it/s, loss=5.18, v_num=0]
Epoch 8: 18%|#8 | 4/22 [00:02<00:12, 1.49it/s, loss=5.18, v_num=0]
Epoch 8: 23%|##2 | 5/22 [00:02<00:09, 1.80it/s, loss=5.18, v_num=0]
Epoch 8: 23%|##2 | 5/22 [00:02<00:09, 1.76it/s, loss=5.18, v_num=0]
Epoch 8: 27%|##7 | 6/22 [00:02<00:07, 2.04it/s, loss=5.18, v_num=0]
Epoch 8: 27%|##7 | 6/22 [00:02<00:07, 2.00it/s, loss=5.17, v_num=0]
Epoch 8: 32%|###1 | 7/22 [00:03<00:06, 2.26it/s, loss=5.17, v_num=0]
Epoch 8: 32%|###1 | 7/22 [00:03<00:06, 2.22it/s, loss=5.17, v_num=0]
Epoch 8: 36%|###6 | 8/22 [00:03<00:05, 2.47it/s, loss=5.17, v_num=0]
Epoch 8: 36%|###6 | 8/22 [00:03<00:05, 2.42it/s, loss=5.17, v_num=0]
Epoch 8: 41%|#### | 9/22 [00:03<00:05, 2.56it/s, loss=5.17, v_num=0]
Epoch 8: 41%|#### | 9/22 [00:03<00:05, 2.52it/s, loss=5.16, v_num=0]
Epoch 8: 45%|####5 | 10/22 [00:03<00:04, 2.72it/s, loss=5.16, v_num=0]
Epoch 8: 45%|####5 | 10/22 [00:03<00:04, 2.68it/s, loss=5.16, v_num=0]
Epoch 8: 50%|##### | 11/22 [00:03<00:03, 2.87it/s, loss=5.16, v_num=0]
Epoch 8: 50%|##### | 11/22 [00:03<00:03, 2.83it/s, loss=5.16, v_num=0]
Epoch 8: 55%|#####4 | 12/22 [00:03<00:03, 3.01it/s, loss=5.16, v_num=0]
Epoch 8: 55%|#####4 | 12/22 [00:04<00:03, 2.97it/s, loss=5.16, v_num=0]
Epoch 8: 59%|#####9 | 13/22 [00:04<00:02, 3.15it/s, loss=5.16, v_num=0]
Epoch 8: 59%|#####9 | 13/22 [00:04<00:02, 3.10it/s, loss=5.16, v_num=0]
Epoch 8: 64%|######3 | 14/22 [00:04<00:02, 3.27it/s, loss=5.16, v_num=0]
Epoch 8: 64%|######3 | 14/22 [00:04<00:02, 3.23it/s, loss=5.16, v_num=0]
Epoch 8: 68%|######8 | 15/22 [00:04<00:02, 3.39it/s, loss=5.16, v_num=0]
Epoch 8: 68%|######8 | 15/22 [00:04<00:02, 3.34it/s, loss=5.16, v_num=0]
Epoch 8: 73%|#######2 | 16/22 [00:04<00:01, 3.49it/s, loss=5.16, v_num=0]
Epoch 8: 73%|#######2 | 16/22 [00:04<00:01, 3.45it/s, loss=5.15, v_num=0]
Epoch 8: 77%|#######7 | 17/22 [00:04<00:01, 3.50it/s, loss=5.15, v_num=0]
Epoch 8: 77%|#######7 | 17/22 [00:04<00:01, 3.46it/s, loss=5.15, v_num=0]
Epoch 8: 82%|########1 | 18/22 [00:05<00:01, 3.60it/s, loss=5.15, v_num=0]
Epoch 8: 82%|########1 | 18/22 [00:05<00:01, 3.55it/s, loss=5.15, v_num=0]
Epoch 8: 86%|########6 | 19/22 [00:05<00:00, 3.69it/s, loss=5.15, v_num=0]
Epoch 8: 86%|########6 | 19/22 [00:05<00:00, 3.64it/s, loss=5.14, v_num=0]
Epoch 8: 91%|######### | 20/22 [00:05<00:00, 3.77it/s, loss=5.14, v_num=0]
Epoch 8: 91%|######### | 20/22 [00:05<00:00, 3.73it/s, loss=5.14, v_num=0]
Epoch 8: 95%|#########5| 21/22 [00:05<00:00, 3.85it/s, loss=5.14, v_num=0]
Epoch 8: 95%|#########5| 21/22 [00:05<00:00, 3.81it/s, loss=5.14, v_num=0]
Epoch 8: 100%|##########| 22/22 [00:05<00:00, 3.93it/s, loss=5.14, v_num=0]
Epoch 8: 100%|##########| 22/22 [00:05<00:00, 3.89it/s, loss=5.14, v_num=0]
Epoch 8: 100%|##########| 22/22 [00:05<00:00, 3.88it/s, loss=5.14, v_num=0]
Epoch 8: 0%| | 0/22 [00:00<?, ?it/s, loss=5.14, v_num=0]
Epoch 9: 0%| | 0/22 [00:00<?, ?it/s, loss=5.14, v_num=0]
Epoch 9: 5%|4 | 1/22 [00:02<00:43, 2.08s/it, loss=5.14, v_num=0]
Epoch 9: 5%|4 | 1/22 [00:02<00:44, 2.14s/it, loss=5.14, v_num=0]
Epoch 9: 9%|9 | 2/22 [00:02<00:22, 1.12s/it, loss=5.14, v_num=0]
Epoch 9: 9%|9 | 2/22 [00:02<00:22, 1.15s/it, loss=5.14, v_num=0]
Epoch 9: 14%|#3 | 3/22 [00:02<00:15, 1.25it/s, loss=5.14, v_num=0]
Epoch 9: 14%|#3 | 3/22 [00:02<00:15, 1.22it/s, loss=5.14, v_num=0]
Epoch 9: 18%|#8 | 4/22 [00:02<00:11, 1.56it/s, loss=5.14, v_num=0]
Epoch 9: 18%|#8 | 4/22 [00:02<00:11, 1.53it/s, loss=5.14, v_num=0]
Epoch 9: 23%|##2 | 5/22 [00:02<00:09, 1.85it/s, loss=5.14, v_num=0]
Epoch 9: 23%|##2 | 5/22 [00:02<00:09, 1.81it/s, loss=5.14, v_num=0]
Epoch 9: 27%|##7 | 6/22 [00:02<00:07, 2.10it/s, loss=5.14, v_num=0]
Epoch 9: 27%|##7 | 6/22 [00:02<00:07, 2.06it/s, loss=5.14, v_num=0]
Epoch 9: 32%|###1 | 7/22 [00:03<00:06, 2.33it/s, loss=5.14, v_num=0]
Epoch 9: 32%|###1 | 7/22 [00:03<00:06, 2.28it/s, loss=5.14, v_num=0]
Epoch 9: 36%|###6 | 8/22 [00:03<00:05, 2.53it/s, loss=5.14, v_num=0]
Epoch 9: 36%|###6 | 8/22 [00:03<00:05, 2.48it/s, loss=5.14, v_num=0]
Epoch 9: 41%|#### | 9/22 [00:03<00:04, 2.61it/s, loss=5.14, v_num=0]
Epoch 9: 41%|#### | 9/22 [00:03<00:05, 2.56it/s, loss=5.15, v_num=0]
Epoch 9: 45%|####5 | 10/22 [00:03<00:04, 2.77it/s, loss=5.15, v_num=0]
Epoch 9: 45%|####5 | 10/22 [00:03<00:04, 2.72it/s, loss=5.14, v_num=0]
Epoch 9: 50%|##### | 11/22 [00:03<00:03, 2.92it/s, loss=5.14, v_num=0]
Epoch 9: 50%|##### | 11/22 [00:03<00:03, 2.88it/s, loss=5.14, v_num=0]
Epoch 9: 55%|#####4 | 12/22 [00:03<00:03, 3.06it/s, loss=5.14, v_num=0]
Epoch 9: 55%|#####4 | 12/22 [00:03<00:03, 3.01it/s, loss=5.14, v_num=0]
Epoch 9: 59%|#####9 | 13/22 [00:04<00:02, 3.19it/s, loss=5.14, v_num=0]
Epoch 9: 59%|#####9 | 13/22 [00:04<00:02, 3.14it/s, loss=5.14, v_num=0]
Epoch 9: 64%|######3 | 14/22 [00:04<00:02, 3.31it/s, loss=5.14, v_num=0]
Epoch 9: 64%|######3 | 14/22 [00:04<00:02, 3.27it/s, loss=5.14, v_num=0]
Epoch 9: 68%|######8 | 15/22 [00:04<00:02, 3.43it/s, loss=5.14, v_num=0]
Epoch 9: 68%|######8 | 15/22 [00:04<00:02, 3.38it/s, loss=5.13, v_num=0]
Epoch 9: 73%|#######2 | 16/22 [00:04<00:01, 3.54it/s, loss=5.13, v_num=0]
Epoch 9: 73%|#######2 | 16/22 [00:04<00:01, 3.49it/s, loss=5.13, v_num=0]
Epoch 9: 77%|#######7 | 17/22 [00:04<00:01, 3.54it/s, loss=5.13, v_num=0]
Epoch 9: 77%|#######7 | 17/22 [00:04<00:01, 3.50it/s, loss=5.13, v_num=0]
Epoch 9: 82%|########1 | 18/22 [00:04<00:01, 3.63it/s, loss=5.13, v_num=0]
Epoch 9: 82%|########1 | 18/22 [00:05<00:01, 3.59it/s, loss=5.13, v_num=0]
Epoch 9: 86%|########6 | 19/22 [00:05<00:00, 3.72it/s, loss=5.13, v_num=0]
Epoch 9: 86%|########6 | 19/22 [00:05<00:00, 3.68it/s, loss=5.13, v_num=0]
Epoch 9: 91%|######### | 20/22 [00:05<00:00, 3.80it/s, loss=5.13, v_num=0]
Epoch 9: 91%|######### | 20/22 [00:05<00:00, 3.76it/s, loss=5.12, v_num=0]
Epoch 9: 95%|#########5| 21/22 [00:05<00:00, 3.89it/s, loss=5.12, v_num=0]
Epoch 9: 95%|#########5| 21/22 [00:05<00:00, 3.84it/s, loss=5.12, v_num=0]
Epoch 9: 100%|##########| 22/22 [00:05<00:00, 3.96it/s, loss=5.12, v_num=0]
Epoch 9: 100%|##########| 22/22 [00:05<00:00, 3.92it/s, loss=5.12, v_num=0]
Epoch 9: 100%|##########| 22/22 [00:05<00:00, 3.92it/s, loss=5.12, v_num=0]
Epoch 9: 0%| | 0/22 [00:00<?, ?it/s, loss=5.12, v_num=0]
Epoch 10: 0%| | 0/22 [00:00<?, ?it/s, loss=5.12, v_num=0]
Epoch 10: 5%|4 | 1/22 [00:02<00:46, 2.21s/it, loss=5.12, v_num=0]
Epoch 10: 5%|4 | 1/22 [00:02<00:47, 2.27s/it, loss=5.12, v_num=0]
Epoch 10: 9%|9 | 2/22 [00:02<00:23, 1.18s/it, loss=5.12, v_num=0]
Epoch 10: 9%|9 | 2/22 [00:02<00:24, 1.21s/it, loss=5.12, v_num=0]
Epoch 10: 14%|#3 | 3/22 [00:02<00:15, 1.19it/s, loss=5.12, v_num=0]
Epoch 10: 14%|#3 | 3/22 [00:02<00:16, 1.17it/s, loss=5.11, v_num=0]
Epoch 10: 18%|#8 | 4/22 [00:02<00:11, 1.50it/s, loss=5.11, v_num=0]
Epoch 10: 18%|#8 | 4/22 [00:02<00:12, 1.47it/s, loss=5.11, v_num=0]
Epoch 10: 23%|##2 | 5/22 [00:02<00:09, 1.77it/s, loss=5.11, v_num=0]
Epoch 10: 23%|##2 | 5/22 [00:02<00:09, 1.74it/s, loss=5.11, v_num=0]
Epoch 10: 27%|##7 | 6/22 [00:02<00:07, 2.02it/s, loss=5.11, v_num=0]
Epoch 10: 27%|##7 | 6/22 [00:03<00:08, 1.98it/s, loss=5.11, v_num=0]
Epoch 10: 32%|###1 | 7/22 [00:03<00:06, 2.24it/s, loss=5.11, v_num=0]
Epoch 10: 32%|###1 | 7/22 [00:03<00:06, 2.20it/s, loss=5.11, v_num=0]
Epoch 10: 36%|###6 | 8/22 [00:03<00:05, 2.45it/s, loss=5.11, v_num=0]
Epoch 10: 36%|###6 | 8/22 [00:03<00:05, 2.40it/s, loss=5.11, v_num=0]
Epoch 10: 41%|#### | 9/22 [00:03<00:05, 2.52it/s, loss=5.11, v_num=0]
Epoch 10: 41%|#### | 9/22 [00:03<00:05, 2.48it/s, loss=5.11, v_num=0]
Epoch 10: 45%|####5 | 10/22 [00:03<00:04, 2.68it/s, loss=5.11, v_num=0]
Epoch 10: 45%|####5 | 10/22 [00:03<00:04, 2.64it/s, loss=5.11, v_num=0]
Epoch 10: 50%|##### | 11/22 [00:03<00:03, 2.84it/s, loss=5.11, v_num=0]
Epoch 10: 50%|##### | 11/22 [00:03<00:03, 2.80it/s, loss=5.11, v_num=0]
Epoch 10: 55%|#####4 | 12/22 [00:04<00:03, 2.98it/s, loss=5.11, v_num=0]
Epoch 10: 55%|#####4 | 12/22 [00:04<00:03, 2.94it/s, loss=5.11, v_num=0]
Epoch 10: 59%|#####9 | 13/22 [00:04<00:02, 3.11it/s, loss=5.11, v_num=0]
Epoch 10: 59%|#####9 | 13/22 [00:04<00:02, 3.07it/s, loss=5.11, v_num=0]
Epoch 10: 64%|######3 | 14/22 [00:04<00:02, 3.24it/s, loss=5.11, v_num=0]
Epoch 10: 64%|######3 | 14/22 [00:04<00:02, 3.19it/s, loss=5.12, v_num=0]
Epoch 10: 68%|######8 | 15/22 [00:04<00:02, 3.35it/s, loss=5.12, v_num=0]
Epoch 10: 68%|######8 | 15/22 [00:04<00:02, 3.31it/s, loss=5.12, v_num=0]
Epoch 10: 73%|#######2 | 16/22 [00:04<00:01, 3.46it/s, loss=5.12, v_num=0]
Epoch 10: 73%|#######2 | 16/22 [00:04<00:01, 3.42it/s, loss=5.12, v_num=0]
Epoch 10: 77%|#######7 | 17/22 [00:04<00:01, 3.46it/s, loss=5.12, v_num=0]
Epoch 10: 77%|#######7 | 17/22 [00:04<00:01, 3.42it/s, loss=5.12, v_num=0]
Epoch 10: 82%|########1 | 18/22 [00:05<00:01, 3.55it/s, loss=5.12, v_num=0]
Epoch 10: 82%|########1 | 18/22 [00:05<00:01, 3.51it/s, loss=5.12, v_num=0]
Epoch 10: 86%|########6 | 19/22 [00:05<00:00, 3.64it/s, loss=5.12, v_num=0]
Epoch 10: 86%|########6 | 19/22 [00:05<00:00, 3.60it/s, loss=5.12, v_num=0]
Epoch 10: 91%|######### | 20/22 [00:05<00:00, 3.73it/s, loss=5.12, v_num=0]
Epoch 10: 91%|######### | 20/22 [00:05<00:00, 3.69it/s, loss=5.12, v_num=0]
Epoch 10: 95%|#########5| 21/22 [00:05<00:00, 3.81it/s, loss=5.12, v_num=0]
Epoch 10: 95%|#########5| 21/22 [00:05<00:00, 3.77it/s, loss=5.12, v_num=0]
Epoch 10: 100%|##########| 22/22 [00:05<00:00, 3.88it/s, loss=5.12, v_num=0]
Epoch 10: 100%|##########| 22/22 [00:05<00:00, 3.84it/s, loss=5.12, v_num=0]
Epoch 10: 100%|##########| 22/22 [00:05<00:00, 3.84it/s, loss=5.12, v_num=0]
Epoch 10: 0%| | 0/22 [00:00<?, ?it/s, loss=5.12, v_num=0]
Epoch 11: 0%| | 0/22 [00:00<?, ?it/s, loss=5.12, v_num=0]
Epoch 11: 5%|4 | 1/22 [00:02<00:44, 2.11s/it, loss=5.12, v_num=0]
Epoch 11: 5%|4 | 1/22 [00:02<00:45, 2.17s/it, loss=5.12, v_num=0]
Epoch 11: 9%|9 | 2/22 [00:02<00:22, 1.13s/it, loss=5.12, v_num=0]
Epoch 11: 9%|9 | 2/22 [00:02<00:23, 1.16s/it, loss=5.12, v_num=0]
Epoch 11: 14%|#3 | 3/22 [00:02<00:15, 1.24it/s, loss=5.12, v_num=0]
Epoch 11: 14%|#3 | 3/22 [00:02<00:15, 1.21it/s, loss=5.12, v_num=0]
Epoch 11: 18%|#8 | 4/22 [00:02<00:11, 1.55it/s, loss=5.12, v_num=0]
Epoch 11: 18%|#8 | 4/22 [00:02<00:11, 1.52it/s, loss=5.11, v_num=0]
Epoch 11: 23%|##2 | 5/22 [00:02<00:09, 1.84it/s, loss=5.11, v_num=0]
Epoch 11: 23%|##2 | 5/22 [00:02<00:09, 1.80it/s, loss=5.11, v_num=0]
Epoch 11: 27%|##7 | 6/22 [00:02<00:07, 2.09it/s, loss=5.11, v_num=0]
Epoch 11: 27%|##7 | 6/22 [00:02<00:07, 2.05it/s, loss=5.11, v_num=0]
Epoch 11: 32%|###1 | 7/22 [00:03<00:06, 2.32it/s, loss=5.11, v_num=0]
Epoch 11: 32%|###1 | 7/22 [00:03<00:06, 2.27it/s, loss=5.11, v_num=0]
Epoch 11: 36%|###6 | 8/22 [00:03<00:05, 2.52it/s, loss=5.11, v_num=0]
Epoch 11: 36%|###6 | 8/22 [00:03<00:05, 2.48it/s, loss=5.1, v_num=0]
Epoch 11: 41%|#### | 9/22 [00:03<00:04, 2.61it/s, loss=5.1, v_num=0]
Epoch 11: 41%|#### | 9/22 [00:03<00:05, 2.57it/s, loss=5.1, v_num=0]
Epoch 11: 45%|####5 | 10/22 [00:03<00:04, 2.78it/s, loss=5.1, v_num=0]
Epoch 11: 45%|####5 | 10/22 [00:03<00:04, 2.74it/s, loss=5.1, v_num=0]
Epoch 11: 50%|##### | 11/22 [00:03<00:03, 2.94it/s, loss=5.1, v_num=0]
Epoch 11: 50%|##### | 11/22 [00:03<00:03, 2.89it/s, loss=5.1, v_num=0]
Epoch 11: 55%|#####4 | 12/22 [00:03<00:03, 3.08it/s, loss=5.1, v_num=0]
Epoch 11: 55%|#####4 | 12/22 [00:03<00:03, 3.03it/s, loss=5.1, v_num=0]
Epoch 11: 59%|#####9 | 13/22 [00:04<00:02, 3.21it/s, loss=5.1, v_num=0]
Epoch 11: 59%|#####9 | 13/22 [00:04<00:02, 3.17it/s, loss=5.1, v_num=0]
Epoch 11: 64%|######3 | 14/22 [00:04<00:02, 3.34it/s, loss=5.1, v_num=0]
Epoch 11: 64%|######3 | 14/22 [00:04<00:02, 3.29it/s, loss=5.1, v_num=0]
Epoch 11: 68%|######8 | 15/22 [00:04<00:02, 3.45it/s, loss=5.1, v_num=0]
Epoch 11: 68%|######8 | 15/22 [00:04<00:02, 3.41it/s, loss=5.1, v_num=0]
Epoch 11: 73%|#######2 | 16/22 [00:04<00:01, 3.56it/s, loss=5.1, v_num=0]
Epoch 11: 73%|#######2 | 16/22 [00:04<00:01, 3.51it/s, loss=5.1, v_num=0]
Epoch 11: 77%|#######7 | 17/22 [00:04<00:01, 3.58it/s, loss=5.1, v_num=0]
Epoch 11: 77%|#######7 | 17/22 [00:04<00:01, 3.54it/s, loss=5.09, v_num=0]
Epoch 11: 82%|########1 | 18/22 [00:04<00:01, 3.68it/s, loss=5.09, v_num=0]
Epoch 11: 82%|########1 | 18/22 [00:04<00:01, 3.63it/s, loss=5.09, v_num=0]
Epoch 11: 86%|########6 | 19/22 [00:05<00:00, 3.77it/s, loss=5.09, v_num=0]
Epoch 11: 86%|########6 | 19/22 [00:05<00:00, 3.72it/s, loss=5.09, v_num=0]
Epoch 11: 91%|######### | 20/22 [00:05<00:00, 3.85it/s, loss=5.09, v_num=0]
Epoch 11: 91%|######### | 20/22 [00:05<00:00, 3.80it/s, loss=5.09, v_num=0]
Epoch 11: 95%|#########5| 21/22 [00:05<00:00, 3.93it/s, loss=5.09, v_num=0]
Epoch 11: 95%|#########5| 21/22 [00:05<00:00, 3.88it/s, loss=5.09, v_num=0]
Epoch 11: 100%|##########| 22/22 [00:05<00:00, 4.00it/s, loss=5.09, v_num=0]
Epoch 11: 100%|##########| 22/22 [00:05<00:00, 3.96it/s, loss=5.09, v_num=0]
Epoch 11: 100%|##########| 22/22 [00:05<00:00, 3.96it/s, loss=5.09, v_num=0]
Epoch 11: 0%| | 0/22 [00:00<?, ?it/s, loss=5.09, v_num=0]
Epoch 12: 0%| | 0/22 [00:00<?, ?it/s, loss=5.09, v_num=0]
Epoch 12: 5%|4 | 1/22 [00:02<00:45, 2.15s/it, loss=5.09, v_num=0]
Epoch 12: 5%|4 | 1/22 [00:02<00:46, 2.21s/it, loss=5.08, v_num=0]
Epoch 12: 9%|9 | 2/22 [00:02<00:23, 1.16s/it, loss=5.08, v_num=0]
Epoch 12: 9%|9 | 2/22 [00:02<00:23, 1.19s/it, loss=5.09, v_num=0]
Epoch 12: 14%|#3 | 3/22 [00:02<00:15, 1.22it/s, loss=5.09, v_num=0]
Epoch 12: 14%|#3 | 3/22 [00:02<00:15, 1.19it/s, loss=5.08, v_num=0]
Epoch 12: 18%|#8 | 4/22 [00:02<00:11, 1.53it/s, loss=5.08, v_num=0]
Epoch 12: 18%|#8 | 4/22 [00:02<00:12, 1.49it/s, loss=5.08, v_num=0]
Epoch 12: 23%|##2 | 5/22 [00:02<00:09, 1.80it/s, loss=5.08, v_num=0]
Epoch 12: 23%|##2 | 5/22 [00:02<00:09, 1.77it/s, loss=5.08, v_num=0]
Epoch 12: 27%|##7 | 6/22 [00:02<00:07, 2.05it/s, loss=5.08, v_num=0]
Epoch 12: 27%|##7 | 6/22 [00:02<00:07, 2.01it/s, loss=5.08, v_num=0]
Epoch 12: 32%|###1 | 7/22 [00:03<00:06, 2.27it/s, loss=5.08, v_num=0]
Epoch 12: 32%|###1 | 7/22 [00:03<00:06, 2.23it/s, loss=5.08, v_num=0]
Epoch 12: 36%|###6 | 8/22 [00:03<00:05, 2.47it/s, loss=5.08, v_num=0]
Epoch 12: 36%|###6 | 8/22 [00:03<00:05, 2.43it/s, loss=5.08, v_num=0]
Epoch 12: 41%|#### | 9/22 [00:03<00:05, 2.58it/s, loss=5.08, v_num=0]
Epoch 12: 41%|#### | 9/22 [00:03<00:05, 2.54it/s, loss=5.08, v_num=0]
Epoch 12: 45%|####5 | 10/22 [00:03<00:04, 2.73it/s, loss=5.08, v_num=0]
Epoch 12: 45%|####5 | 10/22 [00:03<00:04, 2.68it/s, loss=5.08, v_num=0]
Epoch 12: 50%|##### | 11/22 [00:03<00:03, 2.88it/s, loss=5.08, v_num=0]
Epoch 12: 50%|##### | 11/22 [00:03<00:03, 2.84it/s, loss=5.08, v_num=0]
Epoch 12: 55%|#####4 | 12/22 [00:03<00:03, 3.03it/s, loss=5.08, v_num=0]
Epoch 12: 55%|#####4 | 12/22 [00:04<00:03, 2.98it/s, loss=5.08, v_num=0]
Epoch 12: 59%|#####9 | 13/22 [00:04<00:02, 3.16it/s, loss=5.08, v_num=0]
Epoch 12: 59%|#####9 | 13/22 [00:04<00:02, 3.11it/s, loss=5.07, v_num=0]
Epoch 12: 64%|######3 | 14/22 [00:04<00:02, 3.28it/s, loss=5.07, v_num=0]
Epoch 12: 64%|######3 | 14/22 [00:04<00:02, 3.23it/s, loss=5.07, v_num=0]
Epoch 12: 68%|######8 | 15/22 [00:04<00:02, 3.39it/s, loss=5.07, v_num=0]
Epoch 12: 68%|######8 | 15/22 [00:04<00:02, 3.35it/s, loss=5.07, v_num=0]
Epoch 12: 73%|#######2 | 16/22 [00:04<00:01, 3.50it/s, loss=5.07, v_num=0]
Epoch 12: 73%|#######2 | 16/22 [00:04<00:01, 3.45it/s, loss=5.07, v_num=0]
Epoch 12: 77%|#######7 | 17/22 [00:04<00:01, 3.45it/s, loss=5.07, v_num=0]
Epoch 12: 77%|#######7 | 17/22 [00:04<00:01, 3.41it/s, loss=5.07, v_num=0]
Epoch 12: 82%|########1 | 18/22 [00:05<00:01, 3.54it/s, loss=5.07, v_num=0]
Epoch 12: 82%|########1 | 18/22 [00:05<00:01, 3.50it/s, loss=5.07, v_num=0]
Epoch 12: 86%|########6 | 19/22 [00:05<00:00, 3.63it/s, loss=5.07, v_num=0]
Epoch 12: 86%|########6 | 19/22 [00:05<00:00, 3.59it/s, loss=5.07, v_num=0]
Epoch 12: 91%|######### | 20/22 [00:05<00:00, 3.71it/s, loss=5.07, v_num=0]
Epoch 12: 91%|######### | 20/22 [00:05<00:00, 3.67it/s, loss=5.07, v_num=0]
Epoch 12: 95%|#########5| 21/22 [00:05<00:00, 3.79it/s, loss=5.07, v_num=0]
Epoch 12: 95%|#########5| 21/22 [00:05<00:00, 3.75it/s, loss=5.07, v_num=0]
Epoch 12: 100%|##########| 22/22 [00:05<00:00, 3.86it/s, loss=5.07, v_num=0]
Epoch 12: 100%|##########| 22/22 [00:05<00:00, 3.82it/s, loss=5.07, v_num=0]
Epoch 12: 100%|##########| 22/22 [00:05<00:00, 3.82it/s, loss=5.07, v_num=0]
Epoch 12: 0%| | 0/22 [00:00<?, ?it/s, loss=5.07, v_num=0]
Epoch 13: 0%| | 0/22 [00:00<?, ?it/s, loss=5.07, v_num=0]
Epoch 13: 5%|4 | 1/22 [00:02<00:45, 2.15s/it, loss=5.07, v_num=0]
Epoch 13: 5%|4 | 1/22 [00:02<00:46, 2.22s/it, loss=5.07, v_num=0]
Epoch 13: 9%|9 | 2/22 [00:02<00:23, 1.16s/it, loss=5.07, v_num=0]
Epoch 13: 9%|9 | 2/22 [00:02<00:23, 1.19s/it, loss=5.07, v_num=0]
Epoch 13: 14%|#3 | 3/22 [00:02<00:15, 1.22it/s, loss=5.07, v_num=0]
Epoch 13: 14%|#3 | 3/22 [00:02<00:16, 1.19it/s, loss=5.07, v_num=0]
Epoch 13: 18%|#8 | 4/22 [00:02<00:11, 1.53it/s, loss=5.07, v_num=0]
Epoch 13: 18%|#8 | 4/22 [00:02<00:12, 1.49it/s, loss=5.07, v_num=0]
Epoch 13: 23%|##2 | 5/22 [00:02<00:09, 1.80it/s, loss=5.07, v_num=0]
Epoch 13: 23%|##2 | 5/22 [00:02<00:09, 1.77it/s, loss=5.07, v_num=0]
Epoch 13: 27%|##7 | 6/22 [00:02<00:07, 2.05it/s, loss=5.07, v_num=0]
Epoch 13: 27%|##7 | 6/22 [00:02<00:07, 2.01it/s, loss=5.07, v_num=0]
Epoch 13: 32%|###1 | 7/22 [00:03<00:06, 2.28it/s, loss=5.07, v_num=0]
Epoch 13: 32%|###1 | 7/22 [00:03<00:06, 2.23it/s, loss=5.06, v_num=0]
Epoch 13: 36%|###6 | 8/22 [00:03<00:05, 2.47it/s, loss=5.06, v_num=0]
Epoch 13: 36%|###6 | 8/22 [00:03<00:05, 2.42it/s, loss=5.06, v_num=0]
Epoch 13: 41%|#### | 9/22 [00:03<00:05, 2.52it/s, loss=5.06, v_num=0]
Epoch 13: 41%|#### | 9/22 [00:03<00:05, 2.48it/s, loss=5.07, v_num=0]
Epoch 13: 45%|####5 | 10/22 [00:03<00:04, 2.69it/s, loss=5.07, v_num=0]
Epoch 13: 45%|####5 | 10/22 [00:03<00:04, 2.64it/s, loss=5.06, v_num=0]
Epoch 13: 50%|##### | 11/22 [00:03<00:03, 2.84it/s, loss=5.06, v_num=0]
Epoch 13: 50%|##### | 11/22 [00:03<00:03, 2.80it/s, loss=5.06, v_num=0]
Epoch 13: 55%|#####4 | 12/22 [00:04<00:03, 2.98it/s, loss=5.06, v_num=0]
Epoch 13: 55%|#####4 | 12/22 [00:04<00:03, 2.94it/s, loss=5.06, v_num=0]
Epoch 13: 59%|#####9 | 13/22 [00:04<00:02, 3.12it/s, loss=5.06, v_num=0]
Epoch 13: 59%|#####9 | 13/22 [00:04<00:02, 3.07it/s, loss=5.06, v_num=0]
Epoch 13: 64%|######3 | 14/22 [00:04<00:02, 3.24it/s, loss=5.06, v_num=0]
Epoch 13: 64%|######3 | 14/22 [00:04<00:02, 3.19it/s, loss=5.06, v_num=0]
Epoch 13: 68%|######8 | 15/22 [00:04<00:02, 3.35it/s, loss=5.06, v_num=0]
Epoch 13: 68%|######8 | 15/22 [00:04<00:02, 3.30it/s, loss=5.06, v_num=0]
Epoch 13: 73%|#######2 | 16/22 [00:04<00:01, 3.45it/s, loss=5.06, v_num=0]
Epoch 13: 73%|#######2 | 16/22 [00:04<00:01, 3.41it/s, loss=5.07, v_num=0]
Epoch 13: 77%|#######7 | 17/22 [00:04<00:01, 3.45it/s, loss=5.07, v_num=0]
Epoch 13: 77%|#######7 | 17/22 [00:04<00:01, 3.41it/s, loss=5.07, v_num=0]
Epoch 13: 82%|########1 | 18/22 [00:05<00:01, 3.55it/s, loss=5.07, v_num=0]
Epoch 13: 82%|########1 | 18/22 [00:05<00:01, 3.50it/s, loss=5.06, v_num=0]
Epoch 13: 86%|########6 | 19/22 [00:05<00:00, 3.63it/s, loss=5.06, v_num=0]
Epoch 13: 86%|########6 | 19/22 [00:05<00:00, 3.59it/s, loss=5.07, v_num=0]
Epoch 13: 91%|######### | 20/22 [00:05<00:00, 3.72it/s, loss=5.07, v_num=0]
Epoch 13: 91%|######### | 20/22 [00:05<00:00, 3.68it/s, loss=5.06, v_num=0]
Epoch 13: 95%|#########5| 21/22 [00:05<00:00, 3.80it/s, loss=5.06, v_num=0]
Epoch 13: 95%|#########5| 21/22 [00:05<00:00, 3.76it/s, loss=5.06, v_num=0]
Epoch 13: 100%|##########| 22/22 [00:05<00:00, 3.88it/s, loss=5.06, v_num=0]
Epoch 13: 100%|##########| 22/22 [00:05<00:00, 3.83it/s, loss=5.06, v_num=0]
Epoch 13: 100%|##########| 22/22 [00:05<00:00, 3.83it/s, loss=5.06, v_num=0]
Epoch 13: 0%| | 0/22 [00:00<?, ?it/s, loss=5.06, v_num=0]
Epoch 14: 0%| | 0/22 [00:00<?, ?it/s, loss=5.06, v_num=0]
Epoch 14: 5%|4 | 1/22 [00:02<00:46, 2.21s/it, loss=5.06, v_num=0]
Epoch 14: 5%|4 | 1/22 [00:02<00:47, 2.27s/it, loss=5.06, v_num=0]
Epoch 14: 9%|9 | 2/22 [00:02<00:23, 1.18s/it, loss=5.06, v_num=0]
Epoch 14: 9%|9 | 2/22 [00:02<00:24, 1.21s/it, loss=5.07, v_num=0]
Epoch 14: 14%|#3 | 3/22 [00:02<00:15, 1.19it/s, loss=5.07, v_num=0]
Epoch 14: 14%|#3 | 3/22 [00:02<00:16, 1.16it/s, loss=5.07, v_num=0]
Epoch 14: 18%|#8 | 4/22 [00:02<00:12, 1.50it/s, loss=5.07, v_num=0]
Epoch 14: 18%|#8 | 4/22 [00:02<00:12, 1.46it/s, loss=5.07, v_num=0]
Epoch 14: 23%|##2 | 5/22 [00:02<00:09, 1.77it/s, loss=5.07, v_num=0]
Epoch 14: 23%|##2 | 5/22 [00:02<00:09, 1.73it/s, loss=5.07, v_num=0]
Epoch 14: 27%|##7 | 6/22 [00:02<00:07, 2.01it/s, loss=5.07, v_num=0]
Epoch 14: 27%|##7 | 6/22 [00:03<00:08, 1.97it/s, loss=5.07, v_num=0]
Epoch 14: 32%|###1 | 7/22 [00:03<00:06, 2.24it/s, loss=5.07, v_num=0]
Epoch 14: 32%|###1 | 7/22 [00:03<00:06, 2.19it/s, loss=5.07, v_num=0]
Epoch 14: 36%|###6 | 8/22 [00:03<00:05, 2.44it/s, loss=5.07, v_num=0]
Epoch 14: 36%|###6 | 8/22 [00:03<00:05, 2.39it/s, loss=5.06, v_num=0]
Epoch 14: 41%|#### | 9/22 [00:03<00:05, 2.49it/s, loss=5.06, v_num=0]
Epoch 14: 41%|#### | 9/22 [00:03<00:05, 2.44it/s, loss=5.06, v_num=0]
Epoch 14: 45%|####5 | 10/22 [00:03<00:04, 2.65it/s, loss=5.06, v_num=0]
Epoch 14: 45%|####5 | 10/22 [00:03<00:04, 2.61it/s, loss=5.06, v_num=0]
Epoch 14: 50%|##### | 11/22 [00:03<00:03, 2.80it/s, loss=5.06, v_num=0]
Epoch 14: 50%|##### | 11/22 [00:03<00:03, 2.76it/s, loss=5.06, v_num=0]
Epoch 14: 55%|#####4 | 12/22 [00:04<00:03, 2.94it/s, loss=5.06, v_num=0]
Epoch 14: 55%|#####4 | 12/22 [00:04<00:03, 2.90it/s, loss=5.06, v_num=0]
Epoch 14: 59%|#####9 | 13/22 [00:04<00:02, 3.08it/s, loss=5.06, v_num=0]
Epoch 14: 59%|#####9 | 13/22 [00:04<00:02, 3.03it/s, loss=5.06, v_num=0]
Epoch 14: 64%|######3 | 14/22 [00:04<00:02, 3.20it/s, loss=5.06, v_num=0]
Epoch 14: 64%|######3 | 14/22 [00:04<00:02, 3.15it/s, loss=5.06, v_num=0]
Epoch 14: 68%|######8 | 15/22 [00:04<00:02, 3.32it/s, loss=5.06, v_num=0]
Epoch 14: 68%|######8 | 15/22 [00:04<00:02, 3.27it/s, loss=5.06, v_num=0]
Epoch 14: 73%|#######2 | 16/22 [00:04<00:01, 3.42it/s, loss=5.06, v_num=0]
Epoch 14: 73%|#######2 | 16/22 [00:04<00:01, 3.38it/s, loss=5.06, v_num=0]
Epoch 14: 77%|#######7 | 17/22 [00:04<00:01, 3.42it/s, loss=5.06, v_num=0]
Epoch 14: 77%|#######7 | 17/22 [00:05<00:01, 3.38it/s, loss=5.05, v_num=0]
Epoch 14: 82%|########1 | 18/22 [00:05<00:01, 3.52it/s, loss=5.05, v_num=0]
Epoch 14: 82%|########1 | 18/22 [00:05<00:01, 3.48it/s, loss=5.05, v_num=0]
Epoch 14: 86%|########6 | 19/22 [00:05<00:00, 3.61it/s, loss=5.05, v_num=0]
Epoch 14: 86%|########6 | 19/22 [00:05<00:00, 3.57it/s, loss=5.05, v_num=0]
Epoch 14: 91%|######### | 20/22 [00:05<00:00, 3.69it/s, loss=5.05, v_num=0]
Epoch 14: 91%|######### | 20/22 [00:05<00:00, 3.65it/s, loss=5.05, v_num=0]
Epoch 14: 95%|#########5| 21/22 [00:05<00:00, 3.78it/s, loss=5.05, v_num=0]
Epoch 14: 95%|#########5| 21/22 [00:05<00:00, 3.73it/s, loss=5.05, v_num=0]
Epoch 14: 100%|##########| 22/22 [00:05<00:00, 3.85it/s, loss=5.05, v_num=0]
Epoch 14: 100%|##########| 22/22 [00:05<00:00, 3.81it/s, loss=5.05, v_num=0]
Epoch 14: 100%|##########| 22/22 [00:05<00:00, 3.81it/s, loss=5.05, v_num=0]
Epoch 14: 0%| | 0/22 [00:00<?, ?it/s, loss=5.05, v_num=0]
Epoch 15: 0%| | 0/22 [00:00<?, ?it/s, loss=5.05, v_num=0]
Epoch 15: 5%|4 | 1/22 [00:02<00:45, 2.17s/it, loss=5.05, v_num=0]
Epoch 15: 5%|4 | 1/22 [00:02<00:46, 2.23s/it, loss=5.05, v_num=0]
Epoch 15: 9%|9 | 2/22 [00:02<00:23, 1.16s/it, loss=5.05, v_num=0]
Epoch 15: 9%|9 | 2/22 [00:02<00:23, 1.19s/it, loss=5.05, v_num=0]
Epoch 15: 14%|#3 | 3/22 [00:02<00:15, 1.21it/s, loss=5.05, v_num=0]
Epoch 15: 14%|#3 | 3/22 [00:02<00:16, 1.18it/s, loss=5.05, v_num=0]
Epoch 15: 18%|#8 | 4/22 [00:02<00:11, 1.52it/s, loss=5.05, v_num=0]
Epoch 15: 18%|#8 | 4/22 [00:02<00:12, 1.49it/s, loss=5.06, v_num=0]
Epoch 15: 23%|##2 | 5/22 [00:02<00:09, 1.80it/s, loss=5.06, v_num=0]
Epoch 15: 23%|##2 | 5/22 [00:02<00:09, 1.76it/s, loss=5.05, v_num=0]
Epoch 15: 27%|##7 | 6/22 [00:02<00:07, 2.04it/s, loss=5.05, v_num=0]
Epoch 15: 27%|##7 | 6/22 [00:02<00:07, 2.00it/s, loss=5.05, v_num=0]
Epoch 15: 32%|###1 | 7/22 [00:03<00:06, 2.27it/s, loss=5.05, v_num=0]
Epoch 15: 32%|###1 | 7/22 [00:03<00:06, 2.22it/s, loss=5.05, v_num=0]
Epoch 15: 36%|###6 | 8/22 [00:03<00:05, 2.47it/s, loss=5.05, v_num=0]
Epoch 15: 36%|###6 | 8/22 [00:03<00:05, 2.42it/s, loss=5.05, v_num=0]
Epoch 15: 41%|#### | 9/22 [00:03<00:05, 2.55it/s, loss=5.05, v_num=0]
Epoch 15: 41%|#### | 9/22 [00:03<00:05, 2.50it/s, loss=5.05, v_num=0]
Epoch 15: 45%|####5 | 10/22 [00:03<00:04, 2.71it/s, loss=5.05, v_num=0]
Epoch 15: 45%|####5 | 10/22 [00:03<00:04, 2.67it/s, loss=5.05, v_num=0]
Epoch 15: 50%|##### | 11/22 [00:03<00:03, 2.87it/s, loss=5.05, v_num=0]
Epoch 15: 50%|##### | 11/22 [00:03<00:03, 2.82it/s, loss=5.05, v_num=0]
Epoch 15: 55%|#####4 | 12/22 [00:03<00:03, 3.01it/s, loss=5.05, v_num=0]
Epoch 15: 55%|#####4 | 12/22 [00:04<00:03, 2.96it/s, loss=5.05, v_num=0]
Epoch 15: 59%|#####9 | 13/22 [00:04<00:02, 3.14it/s, loss=5.05, v_num=0]
Epoch 15: 59%|#####9 | 13/22 [00:04<00:02, 3.09it/s, loss=5.05, v_num=0]
Epoch 15: 64%|######3 | 14/22 [00:04<00:02, 3.26it/s, loss=5.05, v_num=0]
Epoch 15: 64%|######3 | 14/22 [00:04<00:02, 3.22it/s, loss=5.05, v_num=0]
Epoch 15: 68%|######8 | 15/22 [00:04<00:02, 3.38it/s, loss=5.05, v_num=0]
Epoch 15: 68%|######8 | 15/22 [00:04<00:02, 3.33it/s, loss=5.06, v_num=0]
Epoch 15: 73%|#######2 | 16/22 [00:04<00:01, 3.49it/s, loss=5.06, v_num=0]
Epoch 15: 73%|#######2 | 16/22 [00:04<00:01, 3.44it/s, loss=5.05, v_num=0]
Epoch 15: 77%|#######7 | 17/22 [00:04<00:01, 3.48it/s, loss=5.05, v_num=0]
Epoch 15: 77%|#######7 | 17/22 [00:04<00:01, 3.44it/s, loss=5.06, v_num=0]
Epoch 15: 82%|########1 | 18/22 [00:05<00:01, 3.58it/s, loss=5.06, v_num=0]
Epoch 15: 82%|########1 | 18/22 [00:05<00:01, 3.53it/s, loss=5.05, v_num=0]
Epoch 15: 86%|########6 | 19/22 [00:05<00:00, 3.67it/s, loss=5.05, v_num=0]
Epoch 15: 86%|########6 | 19/22 [00:05<00:00, 3.62it/s, loss=5.05, v_num=0]
Epoch 15: 91%|######### | 20/22 [00:05<00:00, 3.75it/s, loss=5.05, v_num=0]
Epoch 15: 91%|######### | 20/22 [00:05<00:00, 3.70it/s, loss=5.05, v_num=0]
Epoch 15: 95%|#########5| 21/22 [00:05<00:00, 3.83it/s, loss=5.05, v_num=0]
Epoch 15: 95%|#########5| 21/22 [00:05<00:00, 3.79it/s, loss=5.05, v_num=0]
Epoch 15: 100%|##########| 22/22 [00:05<00:00, 3.90it/s, loss=5.05, v_num=0]
Epoch 15: 100%|##########| 22/22 [00:05<00:00, 3.86it/s, loss=5.05, v_num=0]
Epoch 15: 100%|##########| 22/22 [00:05<00:00, 3.86it/s, loss=5.05, v_num=0]
Epoch 15: 0%| | 0/22 [00:00<?, ?it/s, loss=5.05, v_num=0]
Epoch 16: 0%| | 0/22 [00:00<?, ?it/s, loss=5.05, v_num=0]
Epoch 16: 5%|4 | 1/22 [00:02<00:44, 2.12s/it, loss=5.05, v_num=0]
Epoch 16: 5%|4 | 1/22 [00:02<00:45, 2.18s/it, loss=5.05, v_num=0]
Epoch 16: 9%|9 | 2/22 [00:02<00:22, 1.14s/it, loss=5.05, v_num=0]
Epoch 16: 9%|9 | 2/22 [00:02<00:23, 1.17s/it, loss=5.05, v_num=0]
Epoch 16: 14%|#3 | 3/22 [00:02<00:15, 1.23it/s, loss=5.05, v_num=0]
Epoch 16: 14%|#3 | 3/22 [00:02<00:15, 1.20it/s, loss=5.05, v_num=0]
Epoch 16: 18%|#8 | 4/22 [00:02<00:11, 1.55it/s, loss=5.05, v_num=0]
Epoch 16: 18%|#8 | 4/22 [00:02<00:11, 1.51it/s, loss=5.05, v_num=0]
Epoch 16: 23%|##2 | 5/22 [00:02<00:09, 1.83it/s, loss=5.05, v_num=0]
Epoch 16: 23%|##2 | 5/22 [00:02<00:09, 1.79it/s, loss=5.05, v_num=0]
Epoch 16: 27%|##7 | 6/22 [00:02<00:07, 2.08it/s, loss=5.05, v_num=0]
Epoch 16: 27%|##7 | 6/22 [00:02<00:07, 2.03it/s, loss=5.05, v_num=0]
Epoch 16: 32%|###1 | 7/22 [00:03<00:06, 2.30it/s, loss=5.05, v_num=0]
Epoch 16: 32%|###1 | 7/22 [00:03<00:06, 2.26it/s, loss=5.05, v_num=0]
Epoch 16: 36%|###6 | 8/22 [00:03<00:05, 2.51it/s, loss=5.05, v_num=0]
Epoch 16: 36%|###6 | 8/22 [00:03<00:05, 2.46it/s, loss=5.05, v_num=0]
Epoch 16: 41%|#### | 9/22 [00:03<00:04, 2.60it/s, loss=5.05, v_num=0]
Epoch 16: 41%|#### | 9/22 [00:03<00:05, 2.56it/s, loss=5.05, v_num=0]
Epoch 16: 45%|####5 | 10/22 [00:03<00:04, 2.77it/s, loss=5.05, v_num=0]
Epoch 16: 45%|####5 | 10/22 [00:03<00:04, 2.72it/s, loss=5.05, v_num=0]
Epoch 16: 50%|##### | 11/22 [00:03<00:03, 2.93it/s, loss=5.05, v_num=0]
Epoch 16: 50%|##### | 11/22 [00:03<00:03, 2.88it/s, loss=5.05, v_num=0]
Epoch 16: 55%|#####4 | 12/22 [00:03<00:03, 3.06it/s, loss=5.05, v_num=0]
Epoch 16: 55%|#####4 | 12/22 [00:03<00:03, 3.02it/s, loss=5.05, v_num=0]
Epoch 16: 59%|#####9 | 13/22 [00:04<00:02, 3.20it/s, loss=5.05, v_num=0]
Epoch 16: 59%|#####9 | 13/22 [00:04<00:02, 3.15it/s, loss=5.05, v_num=0]
Epoch 16: 64%|######3 | 14/22 [00:04<00:02, 3.32it/s, loss=5.05, v_num=0]
Epoch 16: 64%|######3 | 14/22 [00:04<00:02, 3.27it/s, loss=5.05, v_num=0]
Epoch 16: 68%|######8 | 15/22 [00:04<00:02, 3.43it/s, loss=5.05, v_num=0]
Epoch 16: 68%|######8 | 15/22 [00:04<00:02, 3.38it/s, loss=5.05, v_num=0]
Epoch 16: 73%|#######2 | 16/22 [00:04<00:01, 3.54it/s, loss=5.05, v_num=0]
Epoch 16: 73%|#######2 | 16/22 [00:04<00:01, 3.49it/s, loss=5.05, v_num=0]
Epoch 16: 77%|#######7 | 17/22 [00:04<00:01, 3.59it/s, loss=5.05, v_num=0]
Epoch 16: 77%|#######7 | 17/22 [00:04<00:01, 3.54it/s, loss=5.04, v_num=0]
Epoch 16: 82%|########1 | 18/22 [00:04<00:01, 3.68it/s, loss=5.04, v_num=0]
Epoch 16: 82%|########1 | 18/22 [00:04<00:01, 3.63it/s, loss=5.05, v_num=0]
Epoch 16: 86%|########6 | 19/22 [00:05<00:00, 3.77it/s, loss=5.05, v_num=0]
Epoch 16: 86%|########6 | 19/22 [00:05<00:00, 3.72it/s, loss=5.04, v_num=0]
Epoch 16: 91%|######### | 20/22 [00:05<00:00, 3.86it/s, loss=5.04, v_num=0]
Epoch 16: 91%|######### | 20/22 [00:05<00:00, 3.81it/s, loss=5.04, v_num=0]
Epoch 16: 95%|#########5| 21/22 [00:05<00:00, 3.93it/s, loss=5.04, v_num=0]
Epoch 16: 95%|#########5| 21/22 [00:05<00:00, 3.89it/s, loss=5.04, v_num=0]
Epoch 16: 100%|##########| 22/22 [00:05<00:00, 4.01it/s, loss=5.04, v_num=0]
Epoch 16: 100%|##########| 22/22 [00:05<00:00, 3.96it/s, loss=5.04, v_num=0]
Epoch 16: 100%|##########| 22/22 [00:05<00:00, 3.96it/s, loss=5.04, v_num=0]
Epoch 16: 0%| | 0/22 [00:00<?, ?it/s, loss=5.04, v_num=0]
Epoch 17: 0%| | 0/22 [00:00<?, ?it/s, loss=5.04, v_num=0]
Epoch 17: 5%|4 | 1/22 [00:02<00:45, 2.17s/it, loss=5.04, v_num=0]
Epoch 17: 5%|4 | 1/22 [00:02<00:46, 2.23s/it, loss=5.04, v_num=0]
Epoch 17: 9%|9 | 2/22 [00:02<00:23, 1.17s/it, loss=5.04, v_num=0]
Epoch 17: 9%|9 | 2/22 [00:02<00:23, 1.19s/it, loss=5.04, v_num=0]
Epoch 17: 14%|#3 | 3/22 [00:02<00:15, 1.21it/s, loss=5.04, v_num=0]
Epoch 17: 14%|#3 | 3/22 [00:02<00:16, 1.18it/s, loss=5.04, v_num=0]
Epoch 17: 18%|#8 | 4/22 [00:02<00:11, 1.52it/s, loss=5.04, v_num=0]
Epoch 17: 18%|#8 | 4/22 [00:02<00:12, 1.48it/s, loss=5.04, v_num=0]
Epoch 17: 23%|##2 | 5/22 [00:02<00:09, 1.79it/s, loss=5.04, v_num=0]
Epoch 17: 23%|##2 | 5/22 [00:02<00:09, 1.76it/s, loss=5.04, v_num=0]
Epoch 17: 27%|##7 | 6/22 [00:02<00:07, 2.04it/s, loss=5.04, v_num=0]
Epoch 17: 27%|##7 | 6/22 [00:02<00:07, 2.00it/s, loss=5.04, v_num=0]
Epoch 17: 32%|###1 | 7/22 [00:03<00:06, 2.27it/s, loss=5.04, v_num=0]
Epoch 17: 32%|###1 | 7/22 [00:03<00:06, 2.22it/s, loss=5.04, v_num=0]
Epoch 17: 36%|###6 | 8/22 [00:03<00:05, 2.47it/s, loss=5.04, v_num=0]
Epoch 17: 36%|###6 | 8/22 [00:03<00:05, 2.42it/s, loss=5.04, v_num=0]
Epoch 17: 41%|#### | 9/22 [00:03<00:05, 2.55it/s, loss=5.04, v_num=0]
Epoch 17: 41%|#### | 9/22 [00:03<00:05, 2.51it/s, loss=5.04, v_num=0]
Epoch 17: 45%|####5 | 10/22 [00:03<00:04, 2.72it/s, loss=5.04, v_num=0]
Epoch 17: 45%|####5 | 10/22 [00:03<00:04, 2.67it/s, loss=5.04, v_num=0]
Epoch 17: 50%|##### | 11/22 [00:03<00:03, 2.87it/s, loss=5.04, v_num=0]
Epoch 17: 50%|##### | 11/22 [00:03<00:03, 2.83it/s, loss=5.04, v_num=0]
Epoch 17: 55%|#####4 | 12/22 [00:03<00:03, 3.02it/s, loss=5.04, v_num=0]
Epoch 17: 55%|#####4 | 12/22 [00:04<00:03, 2.97it/s, loss=5.04, v_num=0]
Epoch 17: 59%|#####9 | 13/22 [00:04<00:02, 3.14it/s, loss=5.04, v_num=0]
Epoch 17: 59%|#####9 | 13/22 [00:04<00:02, 3.10it/s, loss=5.04, v_num=0]
Epoch 17: 64%|######3 | 14/22 [00:04<00:02, 3.27it/s, loss=5.04, v_num=0]
Epoch 17: 64%|######3 | 14/22 [00:04<00:02, 3.22it/s, loss=5.03, v_num=0]
Epoch 17: 68%|######8 | 15/22 [00:04<00:02, 3.38it/s, loss=5.03, v_num=0]
Epoch 17: 68%|######8 | 15/22 [00:04<00:02, 3.34it/s, loss=5.04, v_num=0]
Epoch 17: 73%|#######2 | 16/22 [00:04<00:01, 3.49it/s, loss=5.04, v_num=0]
Epoch 17: 73%|#######2 | 16/22 [00:04<00:01, 3.44it/s, loss=5.03, v_num=0]
Epoch 17: 77%|#######7 | 17/22 [00:04<00:01, 3.51it/s, loss=5.03, v_num=0]
Epoch 17: 77%|#######7 | 17/22 [00:04<00:01, 3.47it/s, loss=5.03, v_num=0]
Epoch 17: 82%|########1 | 18/22 [00:04<00:01, 3.60it/s, loss=5.03, v_num=0]
Epoch 17: 82%|########1 | 18/22 [00:05<00:01, 3.56it/s, loss=5.04, v_num=0]
Epoch 17: 86%|########6 | 19/22 [00:05<00:00, 3.69it/s, loss=5.04, v_num=0]
Epoch 17: 86%|########6 | 19/22 [00:05<00:00, 3.65it/s, loss=5.04, v_num=0]
Epoch 17: 91%|######### | 20/22 [00:05<00:00, 3.78it/s, loss=5.04, v_num=0]
Epoch 17: 91%|######### | 20/22 [00:05<00:00, 3.73it/s, loss=5.04, v_num=0]
Epoch 17: 95%|#########5| 21/22 [00:05<00:00, 3.86it/s, loss=5.04, v_num=0]
Epoch 17: 95%|#########5| 21/22 [00:05<00:00, 3.81it/s, loss=5.03, v_num=0]
Epoch 17: 100%|##########| 22/22 [00:05<00:00, 3.93it/s, loss=5.03, v_num=0]
Epoch 17: 100%|##########| 22/22 [00:05<00:00, 3.89it/s, loss=5.03, v_num=0]
Epoch 17: 100%|##########| 22/22 [00:05<00:00, 3.89it/s, loss=5.03, v_num=0]
Epoch 17: 0%| | 0/22 [00:00<?, ?it/s, loss=5.03, v_num=0]
Epoch 18: 0%| | 0/22 [00:00<?, ?it/s, loss=5.03, v_num=0]
Epoch 18: 5%|4 | 1/22 [00:02<00:44, 2.12s/it, loss=5.03, v_num=0]
Epoch 18: 5%|4 | 1/22 [00:02<00:45, 2.19s/it, loss=5.03, v_num=0]
Epoch 18: 9%|9 | 2/22 [00:02<00:22, 1.14s/it, loss=5.03, v_num=0]
Epoch 18: 9%|9 | 2/22 [00:02<00:23, 1.17s/it, loss=5.03, v_num=0]
Epoch 18: 14%|#3 | 3/22 [00:02<00:15, 1.23it/s, loss=5.03, v_num=0]
Epoch 18: 14%|#3 | 3/22 [00:02<00:15, 1.20it/s, loss=5.03, v_num=0]
Epoch 18: 18%|#8 | 4/22 [00:02<00:11, 1.54it/s, loss=5.03, v_num=0]
Epoch 18: 18%|#8 | 4/22 [00:02<00:11, 1.51it/s, loss=5.03, v_num=0]
Epoch 18: 23%|##2 | 5/22 [00:02<00:09, 1.82it/s, loss=5.03, v_num=0]
Epoch 18: 23%|##2 | 5/22 [00:02<00:09, 1.78it/s, loss=5.03, v_num=0]
Epoch 18: 27%|##7 | 6/22 [00:02<00:07, 2.08it/s, loss=5.03, v_num=0]
Epoch 18: 27%|##7 | 6/22 [00:02<00:07, 2.03it/s, loss=5.03, v_num=0]
Epoch 18: 32%|###1 | 7/22 [00:03<00:06, 2.30it/s, loss=5.03, v_num=0]
Epoch 18: 32%|###1 | 7/22 [00:03<00:06, 2.26it/s, loss=5.03, v_num=0]
Epoch 18: 36%|###6 | 8/22 [00:03<00:05, 2.51it/s, loss=5.03, v_num=0]
Epoch 18: 36%|###6 | 8/22 [00:03<00:05, 2.46it/s, loss=5.02, v_num=0]
Epoch 18: 41%|#### | 9/22 [00:03<00:05, 2.56it/s, loss=5.02, v_num=0]
Epoch 18: 41%|#### | 9/22 [00:03<00:05, 2.52it/s, loss=5.02, v_num=0]
Epoch 18: 45%|####5 | 10/22 [00:03<00:04, 2.73it/s, loss=5.02, v_num=0]
Epoch 18: 45%|####5 | 10/22 [00:03<00:04, 2.68it/s, loss=5.02, v_num=0]
Epoch 18: 50%|##### | 11/22 [00:03<00:03, 2.88it/s, loss=5.02, v_num=0]
Epoch 18: 50%|##### | 11/22 [00:03<00:03, 2.84it/s, loss=5.03, v_num=0]
Epoch 18: 55%|#####4 | 12/22 [00:03<00:03, 3.02it/s, loss=5.03, v_num=0]
Epoch 18: 55%|#####4 | 12/22 [00:04<00:03, 2.98it/s, loss=5.03, v_num=0]
Epoch 18: 59%|#####9 | 13/22 [00:04<00:02, 3.16it/s, loss=5.03, v_num=0]
Epoch 18: 59%|#####9 | 13/22 [00:04<00:02, 3.11it/s, loss=5.03, v_num=0]
Epoch 18: 64%|######3 | 14/22 [00:04<00:02, 3.27it/s, loss=5.03, v_num=0]
Epoch 18: 64%|######3 | 14/22 [00:04<00:02, 3.23it/s, loss=5.03, v_num=0]
Epoch 18: 68%|######8 | 15/22 [00:04<00:02, 3.39it/s, loss=5.03, v_num=0]
Epoch 18: 68%|######8 | 15/22 [00:04<00:02, 3.34it/s, loss=5.03, v_num=0]
Epoch 18: 73%|#######2 | 16/22 [00:04<00:01, 3.49it/s, loss=5.03, v_num=0]
Epoch 18: 73%|#######2 | 16/22 [00:04<00:01, 3.44it/s, loss=5.03, v_num=0]
Epoch 18: 77%|#######7 | 17/22 [00:04<00:01, 3.48it/s, loss=5.03, v_num=0]
Epoch 18: 77%|#######7 | 17/22 [00:04<00:01, 3.43it/s, loss=5.02, v_num=0]
Epoch 18: 82%|########1 | 18/22 [00:05<00:01, 3.57it/s, loss=5.02, v_num=0]
Epoch 18: 82%|########1 | 18/22 [00:05<00:01, 3.53it/s, loss=5.03, v_num=0]
Epoch 18: 86%|########6 | 19/22 [00:05<00:00, 3.66it/s, loss=5.03, v_num=0]
Epoch 18: 86%|########6 | 19/22 [00:05<00:00, 3.62it/s, loss=5.03, v_num=0]
Epoch 18: 91%|######### | 20/22 [00:05<00:00, 3.74it/s, loss=5.03, v_num=0]
Epoch 18: 91%|######### | 20/22 [00:05<00:00, 3.70it/s, loss=5.03, v_num=0]
Epoch 18: 95%|#########5| 21/22 [00:05<00:00, 3.82it/s, loss=5.03, v_num=0]
Epoch 18: 95%|#########5| 21/22 [00:05<00:00, 3.78it/s, loss=5.03, v_num=0]
Epoch 18: 100%|##########| 22/22 [00:05<00:00, 3.90it/s, loss=5.03, v_num=0]
Epoch 18: 100%|##########| 22/22 [00:05<00:00, 3.86it/s, loss=5.03, v_num=0]
Epoch 18: 100%|##########| 22/22 [00:05<00:00, 3.86it/s, loss=5.03, v_num=0]
Epoch 18: 0%| | 0/22 [00:00<?, ?it/s, loss=5.03, v_num=0]
Epoch 19: 0%| | 0/22 [00:00<?, ?it/s, loss=5.03, v_num=0]
Epoch 19: 5%|4 | 1/22 [00:02<00:45, 2.16s/it, loss=5.03, v_num=0]
Epoch 19: 5%|4 | 1/22 [00:02<00:46, 2.22s/it, loss=5.02, v_num=0]
Epoch 19: 9%|9 | 2/22 [00:02<00:23, 1.16s/it, loss=5.02, v_num=0]
Epoch 19: 9%|9 | 2/22 [00:02<00:23, 1.19s/it, loss=5.02, v_num=0]
Epoch 19: 14%|#3 | 3/22 [00:02<00:15, 1.22it/s, loss=5.02, v_num=0]
Epoch 19: 14%|#3 | 3/22 [00:02<00:16, 1.19it/s, loss=5.02, v_num=0]
Epoch 19: 18%|#8 | 4/22 [00:02<00:11, 1.53it/s, loss=5.02, v_num=0]
Epoch 19: 18%|#8 | 4/22 [00:02<00:12, 1.49it/s, loss=5.02, v_num=0]
Epoch 19: 23%|##2 | 5/22 [00:02<00:09, 1.81it/s, loss=5.02, v_num=0]
Epoch 19: 23%|##2 | 5/22 [00:02<00:09, 1.77it/s, loss=5.02, v_num=0]
Epoch 19: 27%|##7 | 6/22 [00:02<00:07, 2.06it/s, loss=5.02, v_num=0]
Epoch 19: 27%|##7 | 6/22 [00:02<00:07, 2.01it/s, loss=5.02, v_num=0]
Epoch 19: 32%|###1 | 7/22 [00:03<00:06, 2.28it/s, loss=5.02, v_num=0]
Epoch 19: 32%|###1 | 7/22 [00:03<00:06, 2.24it/s, loss=5.02, v_num=0]
Epoch 19: 36%|###6 | 8/22 [00:03<00:05, 2.48it/s, loss=5.02, v_num=0]
Epoch 19: 36%|###6 | 8/22 [00:03<00:05, 2.44it/s, loss=5.02, v_num=0]
Epoch 19: 41%|#### | 9/22 [00:03<00:05, 2.57it/s, loss=5.02, v_num=0]
Epoch 19: 41%|#### | 9/22 [00:03<00:05, 2.53it/s, loss=5.02, v_num=0]
Epoch 19: 45%|####5 | 10/22 [00:03<00:04, 2.74it/s, loss=5.02, v_num=0]
Epoch 19: 45%|####5 | 10/22 [00:03<00:04, 2.69it/s, loss=5.02, v_num=0]
Epoch 19: 50%|##### | 11/22 [00:03<00:03, 2.89it/s, loss=5.02, v_num=0]
Epoch 19: 50%|##### | 11/22 [00:03<00:03, 2.84it/s, loss=5.02, v_num=0]
Epoch 19: 55%|#####4 | 12/22 [00:03<00:03, 3.03it/s, loss=5.02, v_num=0]
Epoch 19: 55%|#####4 | 12/22 [00:04<00:03, 2.99it/s, loss=5.02, v_num=0]
Epoch 19: 59%|#####9 | 13/22 [00:04<00:02, 3.17it/s, loss=5.02, v_num=0]
Epoch 19: 59%|#####9 | 13/22 [00:04<00:02, 3.12it/s, loss=5.02, v_num=0]
Epoch 19: 64%|######3 | 14/22 [00:04<00:02, 3.29it/s, loss=5.02, v_num=0]
Epoch 19: 64%|######3 | 14/22 [00:04<00:02, 3.24it/s, loss=5.02, v_num=0]
Epoch 19: 68%|######8 | 15/22 [00:04<00:02, 3.41it/s, loss=5.02, v_num=0]
Epoch 19: 68%|######8 | 15/22 [00:04<00:02, 3.36it/s, loss=5.02, v_num=0]
Epoch 19: 73%|#######2 | 16/22 [00:04<00:01, 3.52it/s, loss=5.02, v_num=0]
Epoch 19: 73%|#######2 | 16/22 [00:04<00:01, 3.47it/s, loss=5.01, v_num=0]
Epoch 19: 77%|#######7 | 17/22 [00:04<00:01, 3.52it/s, loss=5.01, v_num=0]
Epoch 19: 77%|#######7 | 17/22 [00:04<00:01, 3.47it/s, loss=5.02, v_num=0]
Epoch 19: 82%|########1 | 18/22 [00:04<00:01, 3.61it/s, loss=5.02, v_num=0]
Epoch 19: 82%|########1 | 18/22 [00:05<00:01, 3.57it/s, loss=5.02, v_num=0]
Epoch 19: 86%|########6 | 19/22 [00:05<00:00, 3.70it/s, loss=5.02, v_num=0]
Epoch 19: 86%|########6 | 19/22 [00:05<00:00, 3.66it/s, loss=5.02, v_num=0]
Epoch 19: 91%|######### | 20/22 [00:05<00:00, 3.78it/s, loss=5.02, v_num=0]
Epoch 19: 91%|######### | 20/22 [00:05<00:00, 3.74it/s, loss=5.02, v_num=0]
Epoch 19: 95%|#########5| 21/22 [00:05<00:00, 3.86it/s, loss=5.02, v_num=0]
Epoch 19: 95%|#########5| 21/22 [00:05<00:00, 3.82it/s, loss=5.02, v_num=0]
Epoch 19: 100%|##########| 22/22 [00:05<00:00, 3.94it/s, loss=5.02, v_num=0]
Epoch 19: 100%|##########| 22/22 [00:05<00:00, 3.90it/s, loss=5.03, v_num=0]
Epoch 19: 100%|##########| 22/22 [00:05<00:00, 3.90it/s, loss=5.03, v_num=0]
Epoch 19: 100%|##########| 22/22 [00:05<00:00, 3.80it/s, loss=5.03, v_num=0]
Next we create a helper function to generate embeddings from our test images using the model we just trained. Note that only the backbone is needed to generate embeddings, the projection head is only required for the training. Make sure to put the model into eval mode for this part!
def generate_embeddings(model, dataloader):
"""Generates representations for all images in the dataloader with
the given model
"""
embeddings = []
filenames = []
with torch.no_grad():
for img, label, fnames in dataloader:
img = img.to(model.device)
emb = model.backbone(img).flatten(start_dim=1)
embeddings.append(emb)
filenames.extend(fnames)
embeddings = torch.cat(embeddings, 0)
embeddings = normalize(embeddings)
return embeddings, filenames
model.eval()
embeddings, filenames = generate_embeddings(model, dataloader_test)
Visualize Nearest Neighbors
Let’s look at the trained embedding and visualize the nearest neighbors for a few random samples.
We create some helper functions to simplify the work
def get_image_as_np_array(filename: str):
"""Returns an image as an numpy array"""
img = Image.open(filename)
return np.asarray(img)
def plot_knn_examples(embeddings, filenames, n_neighbors=3, num_examples=6):
"""Plots multiple rows of random images with their nearest neighbors"""
# lets look at the nearest neighbors for some samples
# we use the sklearn library
nbrs = NearestNeighbors(n_neighbors=n_neighbors).fit(embeddings)
distances, indices = nbrs.kneighbors(embeddings)
# get 5 random samples
samples_idx = np.random.choice(len(indices), size=num_examples, replace=False)
# loop through our randomly picked samples
for idx in samples_idx:
fig = plt.figure()
# loop through their nearest neighbors
for plot_x_offset, neighbor_idx in enumerate(indices[idx]):
# add the subplot
ax = fig.add_subplot(1, len(indices[idx]), plot_x_offset + 1)
# get the correponding filename for the current index
fname = os.path.join(path_to_data, filenames[neighbor_idx])
# plot the image
plt.imshow(get_image_as_np_array(fname))
# set the title to the distance of the neighbor
ax.set_title(f"d={distances[idx][plot_x_offset]:.3f}")
# let's disable the axis
plt.axis("off")
Let’s do the plot of the images. The leftmost image is the query image whereas the ones next to it on the same row are the nearest neighbors. In the title we see the distance of the neigbor.
plot_knn_examples(embeddings, filenames)
Color Invariance
Let’s train again without color augmentation. This will force our model to respect the colors in the images.
# Set color jitter and gray scale probability to 0
new_collate_fn = SimCLRCollateFunction(
input_size=input_size, vf_prob=0.5, rr_prob=0.5, cj_prob=0.0, random_gray_scale=0.0
)
# let's update our collate method and reuse our dataloader
dataloader_train_simclr.collate_fn = new_collate_fn
# then train a new model
model = SimCLRModel()
trainer = pl.Trainer(max_epochs=max_epochs, devices=1, accelerator="gpu")
trainer.fit(model, dataloader_train_simclr)
# and generate again embeddings from the test set
model.eval()
embeddings, filenames = generate_embeddings(model, dataloader_test)
/opt/runner_04/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1555: PossibleUserWarning: The number of training batches (22) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
rank_zero_warn(
Training: 0it [00:00, ?it/s]
Training: 0%| | 0/22 [00:00<?, ?it/s]
Epoch 0: 0%| | 0/22 [00:00<?, ?it/s]
Epoch 0: 5%|4 | 1/22 [00:01<00:32, 1.54s/it]
Epoch 0: 5%|4 | 1/22 [00:01<00:33, 1.59s/it, loss=6.17, v_num=1]
Epoch 0: 9%|9 | 2/22 [00:01<00:16, 1.18it/s, loss=6.17, v_num=1]
Epoch 0: 9%|9 | 2/22 [00:01<00:17, 1.14it/s, loss=6.15, v_num=1]
Epoch 0: 14%|#3 | 3/22 [00:01<00:11, 1.63it/s, loss=6.15, v_num=1]
Epoch 0: 14%|#3 | 3/22 [00:01<00:12, 1.57it/s, loss=6.06, v_num=1]
Epoch 0: 18%|#8 | 4/22 [00:01<00:08, 2.00it/s, loss=6.06, v_num=1]
Epoch 0: 18%|#8 | 4/22 [00:02<00:09, 1.94it/s, loss=5.97, v_num=1]
Epoch 0: 23%|##2 | 5/22 [00:02<00:07, 2.33it/s, loss=5.97, v_num=1]
Epoch 0: 23%|##2 | 5/22 [00:02<00:07, 2.26it/s, loss=5.88, v_num=1]
Epoch 0: 27%|##7 | 6/22 [00:02<00:06, 2.61it/s, loss=5.88, v_num=1]
Epoch 0: 27%|##7 | 6/22 [00:02<00:06, 2.54it/s, loss=5.79, v_num=1]
Epoch 0: 32%|###1 | 7/22 [00:02<00:05, 2.85it/s, loss=5.79, v_num=1]
Epoch 0: 32%|###1 | 7/22 [00:02<00:05, 2.78it/s, loss=5.72, v_num=1]
Epoch 0: 36%|###6 | 8/22 [00:02<00:04, 3.07it/s, loss=5.72, v_num=1]
Epoch 0: 36%|###6 | 8/22 [00:02<00:04, 3.00it/s, loss=5.65, v_num=1]
Epoch 0: 41%|#### | 9/22 [00:02<00:03, 3.26it/s, loss=5.65, v_num=1]
Epoch 0: 41%|#### | 9/22 [00:02<00:04, 3.19it/s, loss=5.59, v_num=1]
Epoch 0: 45%|####5 | 10/22 [00:02<00:03, 3.44it/s, loss=5.59, v_num=1]
Epoch 0: 45%|####5 | 10/22 [00:02<00:03, 3.37it/s, loss=5.54, v_num=1]
Epoch 0: 50%|##### | 11/22 [00:03<00:03, 3.59it/s, loss=5.54, v_num=1]
Epoch 0: 50%|##### | 11/22 [00:03<00:03, 3.52it/s, loss=5.49, v_num=1]
Epoch 0: 55%|#####4 | 12/22 [00:03<00:02, 3.73it/s, loss=5.49, v_num=1]
Epoch 0: 55%|#####4 | 12/22 [00:03<00:02, 3.66it/s, loss=5.45, v_num=1]
Epoch 0: 59%|#####9 | 13/22 [00:03<00:02, 3.86it/s, loss=5.45, v_num=1]
Epoch 0: 59%|#####9 | 13/22 [00:03<00:02, 3.79it/s, loss=5.41, v_num=1]
Epoch 0: 64%|######3 | 14/22 [00:03<00:02, 3.98it/s, loss=5.41, v_num=1]
Epoch 0: 64%|######3 | 14/22 [00:03<00:02, 3.91it/s, loss=5.38, v_num=1]
Epoch 0: 68%|######8 | 15/22 [00:03<00:01, 4.09it/s, loss=5.38, v_num=1]
Epoch 0: 68%|######8 | 15/22 [00:03<00:01, 4.02it/s, loss=5.35, v_num=1]
Epoch 0: 73%|#######2 | 16/22 [00:03<00:01, 4.19it/s, loss=5.35, v_num=1]
Epoch 0: 73%|#######2 | 16/22 [00:03<00:01, 4.13it/s, loss=5.32, v_num=1]
Epoch 0: 77%|#######7 | 17/22 [00:03<00:01, 4.29it/s, loss=5.32, v_num=1]
Epoch 0: 77%|#######7 | 17/22 [00:04<00:01, 4.22it/s, loss=5.3, v_num=1]
Epoch 0: 82%|########1 | 18/22 [00:04<00:00, 4.38it/s, loss=5.3, v_num=1]
Epoch 0: 82%|########1 | 18/22 [00:04<00:00, 4.31it/s, loss=5.28, v_num=1]
Epoch 0: 86%|########6 | 19/22 [00:04<00:00, 4.46it/s, loss=5.28, v_num=1]
Epoch 0: 86%|########6 | 19/22 [00:04<00:00, 4.39it/s, loss=5.25, v_num=1]
Epoch 0: 91%|######### | 20/22 [00:04<00:00, 4.54it/s, loss=5.25, v_num=1]
Epoch 0: 91%|######### | 20/22 [00:04<00:00, 4.47it/s, loss=5.23, v_num=1]
Epoch 0: 95%|#########5| 21/22 [00:04<00:00, 4.61it/s, loss=5.23, v_num=1]
Epoch 0: 95%|#########5| 21/22 [00:04<00:00, 4.55it/s, loss=5.17, v_num=1]
Epoch 0: 100%|##########| 22/22 [00:04<00:00, 4.68it/s, loss=5.17, v_num=1]
Epoch 0: 100%|##########| 22/22 [00:04<00:00, 4.62it/s, loss=5.1, v_num=1]
Epoch 0: 100%|##########| 22/22 [00:04<00:00, 4.62it/s, loss=5.1, v_num=1]
Epoch 0: 0%| | 0/22 [00:00<?, ?it/s, loss=5.1, v_num=1]
Epoch 1: 0%| | 0/22 [00:00<?, ?it/s, loss=5.1, v_num=1]
Epoch 1: 5%|4 | 1/22 [00:01<00:36, 1.74s/it, loss=5.1, v_num=1]
Epoch 1: 5%|4 | 1/22 [00:01<00:37, 1.80s/it, loss=5.05, v_num=1]
Epoch 1: 9%|9 | 2/22 [00:01<00:18, 1.06it/s, loss=5.05, v_num=1]
Epoch 1: 9%|9 | 2/22 [00:01<00:19, 1.02it/s, loss=5, v_num=1]
Epoch 1: 14%|#3 | 3/22 [00:02<00:12, 1.47it/s, loss=5, v_num=1]
Epoch 1: 14%|#3 | 3/22 [00:02<00:13, 1.43it/s, loss=4.97, v_num=1]
Epoch 1: 18%|#8 | 4/22 [00:02<00:09, 1.82it/s, loss=4.97, v_num=1]
Epoch 1: 18%|#8 | 4/22 [00:02<00:10, 1.77it/s, loss=4.94, v_num=1]
Epoch 1: 23%|##2 | 5/22 [00:02<00:08, 2.12it/s, loss=4.94, v_num=1]
Epoch 1: 23%|##2 | 5/22 [00:02<00:08, 2.07it/s, loss=4.92, v_num=1]
Epoch 1: 27%|##7 | 6/22 [00:02<00:06, 2.39it/s, loss=4.92, v_num=1]
Epoch 1: 27%|##7 | 6/22 [00:02<00:06, 2.33it/s, loss=4.9, v_num=1]
Epoch 1: 32%|###1 | 7/22 [00:02<00:05, 2.63it/s, loss=4.9, v_num=1]
Epoch 1: 32%|###1 | 7/22 [00:02<00:05, 2.57it/s, loss=4.88, v_num=1]
Epoch 1: 36%|###6 | 8/22 [00:02<00:04, 2.84it/s, loss=4.88, v_num=1]
Epoch 1: 36%|###6 | 8/22 [00:02<00:05, 2.78it/s, loss=4.87, v_num=1]
Epoch 1: 41%|#### | 9/22 [00:02<00:04, 3.03it/s, loss=4.87, v_num=1]
Epoch 1: 41%|#### | 9/22 [00:03<00:04, 2.97it/s, loss=4.86, v_num=1]
Epoch 1: 45%|####5 | 10/22 [00:03<00:03, 3.21it/s, loss=4.86, v_num=1]
Epoch 1: 45%|####5 | 10/22 [00:03<00:03, 3.15it/s, loss=4.85, v_num=1]
Epoch 1: 50%|##### | 11/22 [00:03<00:03, 3.37it/s, loss=4.85, v_num=1]
Epoch 1: 50%|##### | 11/22 [00:03<00:03, 3.30it/s, loss=4.84, v_num=1]
Epoch 1: 55%|#####4 | 12/22 [00:03<00:02, 3.50it/s, loss=4.84, v_num=1]
Epoch 1: 55%|#####4 | 12/22 [00:03<00:02, 3.44it/s, loss=4.83, v_num=1]
Epoch 1: 59%|#####9 | 13/22 [00:03<00:02, 3.63it/s, loss=4.83, v_num=1]
Epoch 1: 59%|#####9 | 13/22 [00:03<00:02, 3.57it/s, loss=4.82, v_num=1]
Epoch 1: 64%|######3 | 14/22 [00:03<00:02, 3.76it/s, loss=4.82, v_num=1]
Epoch 1: 64%|######3 | 14/22 [00:03<00:02, 3.69it/s, loss=4.82, v_num=1]
Epoch 1: 68%|######8 | 15/22 [00:03<00:01, 3.87it/s, loss=4.82, v_num=1]
Epoch 1: 68%|######8 | 15/22 [00:03<00:01, 3.81it/s, loss=4.81, v_num=1]
Epoch 1: 73%|#######2 | 16/22 [00:04<00:01, 3.98it/s, loss=4.81, v_num=1]
Epoch 1: 73%|#######2 | 16/22 [00:04<00:01, 3.92it/s, loss=4.8, v_num=1]
Epoch 1: 77%|#######7 | 17/22 [00:04<00:01, 4.07it/s, loss=4.8, v_num=1]
Epoch 1: 77%|#######7 | 17/22 [00:04<00:01, 4.01it/s, loss=4.8, v_num=1]
Epoch 1: 82%|########1 | 18/22 [00:04<00:00, 4.16it/s, loss=4.8, v_num=1]
Epoch 1: 82%|########1 | 18/22 [00:04<00:00, 4.10it/s, loss=4.79, v_num=1]
Epoch 1: 86%|########6 | 19/22 [00:04<00:00, 4.25it/s, loss=4.79, v_num=1]
Epoch 1: 86%|########6 | 19/22 [00:04<00:00, 4.19it/s, loss=4.79, v_num=1]
Epoch 1: 91%|######### | 20/22 [00:04<00:00, 4.32it/s, loss=4.79, v_num=1]
Epoch 1: 91%|######### | 20/22 [00:04<00:00, 4.27it/s, loss=4.78, v_num=1]
Epoch 1: 95%|#########5| 21/22 [00:04<00:00, 4.40it/s, loss=4.78, v_num=1]
Epoch 1: 95%|#########5| 21/22 [00:04<00:00, 4.34it/s, loss=4.78, v_num=1]
Epoch 1: 100%|##########| 22/22 [00:04<00:00, 4.47it/s, loss=4.78, v_num=1]
Epoch 1: 100%|##########| 22/22 [00:04<00:00, 4.41it/s, loss=4.78, v_num=1]
Epoch 1: 100%|##########| 22/22 [00:04<00:00, 4.41it/s, loss=4.78, v_num=1]
Epoch 1: 0%| | 0/22 [00:00<?, ?it/s, loss=4.78, v_num=1]
Epoch 2: 0%| | 0/22 [00:00<?, ?it/s, loss=4.78, v_num=1]
Epoch 2: 5%|4 | 1/22 [00:01<00:36, 1.74s/it, loss=4.78, v_num=1]
Epoch 2: 5%|4 | 1/22 [00:01<00:37, 1.80s/it, loss=4.77, v_num=1]
Epoch 2: 9%|9 | 2/22 [00:01<00:18, 1.05it/s, loss=4.77, v_num=1]
Epoch 2: 9%|9 | 2/22 [00:01<00:19, 1.02it/s, loss=4.77, v_num=1]
Epoch 2: 14%|#3 | 3/22 [00:02<00:13, 1.46it/s, loss=4.77, v_num=1]
Epoch 2: 14%|#3 | 3/22 [00:02<00:13, 1.42it/s, loss=4.77, v_num=1]
Epoch 2: 18%|#8 | 4/22 [00:02<00:09, 1.81it/s, loss=4.77, v_num=1]
Epoch 2: 18%|#8 | 4/22 [00:02<00:10, 1.77it/s, loss=4.76, v_num=1]
Epoch 2: 23%|##2 | 5/22 [00:02<00:07, 2.13it/s, loss=4.76, v_num=1]
Epoch 2: 23%|##2 | 5/22 [00:02<00:08, 2.07it/s, loss=4.76, v_num=1]
Epoch 2: 27%|##7 | 6/22 [00:02<00:06, 2.38it/s, loss=4.76, v_num=1]
Epoch 2: 27%|##7 | 6/22 [00:02<00:06, 2.32it/s, loss=4.75, v_num=1]
Epoch 2: 32%|###1 | 7/22 [00:02<00:05, 2.61it/s, loss=4.75, v_num=1]
Epoch 2: 32%|###1 | 7/22 [00:02<00:05, 2.56it/s, loss=4.75, v_num=1]
Epoch 2: 36%|###6 | 8/22 [00:02<00:04, 2.83it/s, loss=4.75, v_num=1]
Epoch 2: 36%|###6 | 8/22 [00:02<00:05, 2.77it/s, loss=4.75, v_num=1]
Epoch 2: 41%|#### | 9/22 [00:02<00:04, 3.01it/s, loss=4.75, v_num=1]
Epoch 2: 41%|#### | 9/22 [00:03<00:04, 2.96it/s, loss=4.75, v_num=1]
Epoch 2: 45%|####5 | 10/22 [00:03<00:03, 3.18it/s, loss=4.75, v_num=1]
Epoch 2: 45%|####5 | 10/22 [00:03<00:03, 3.12it/s, loss=4.74, v_num=1]
Epoch 2: 50%|##### | 11/22 [00:03<00:03, 3.33it/s, loss=4.74, v_num=1]
Epoch 2: 50%|##### | 11/22 [00:03<00:03, 3.27it/s, loss=4.74, v_num=1]
Epoch 2: 55%|#####4 | 12/22 [00:03<00:02, 3.45it/s, loss=4.74, v_num=1]
Epoch 2: 55%|#####4 | 12/22 [00:03<00:02, 3.39it/s, loss=4.73, v_num=1]
Epoch 2: 59%|#####9 | 13/22 [00:03<00:02, 3.58it/s, loss=4.73, v_num=1]
Epoch 2: 59%|#####9 | 13/22 [00:03<00:02, 3.52it/s, loss=4.72, v_num=1]
Epoch 2: 64%|######3 | 14/22 [00:03<00:02, 3.69it/s, loss=4.72, v_num=1]
Epoch 2: 64%|######3 | 14/22 [00:03<00:02, 3.64it/s, loss=4.72, v_num=1]
Epoch 2: 68%|######8 | 15/22 [00:03<00:01, 3.80it/s, loss=4.72, v_num=1]
Epoch 2: 68%|######8 | 15/22 [00:04<00:01, 3.75it/s, loss=4.72, v_num=1]
Epoch 2: 73%|#######2 | 16/22 [00:04<00:01, 3.91it/s, loss=4.72, v_num=1]
Epoch 2: 73%|#######2 | 16/22 [00:04<00:01, 3.85it/s, loss=4.72, v_num=1]
Epoch 2: 77%|#######7 | 17/22 [00:04<00:01, 4.00it/s, loss=4.72, v_num=1]
Epoch 2: 77%|#######7 | 17/22 [00:04<00:01, 3.95it/s, loss=4.71, v_num=1]
Epoch 2: 82%|########1 | 18/22 [00:04<00:00, 4.09it/s, loss=4.71, v_num=1]
Epoch 2: 82%|########1 | 18/22 [00:04<00:00, 4.04it/s, loss=4.71, v_num=1]
Epoch 2: 86%|########6 | 19/22 [00:04<00:00, 4.17it/s, loss=4.71, v_num=1]
Epoch 2: 86%|########6 | 19/22 [00:04<00:00, 4.12it/s, loss=4.71, v_num=1]
Epoch 2: 91%|######### | 20/22 [00:04<00:00, 4.24it/s, loss=4.71, v_num=1]
Epoch 2: 91%|######### | 20/22 [00:04<00:00, 4.20it/s, loss=4.71, v_num=1]
Epoch 2: 95%|#########5| 21/22 [00:04<00:00, 4.32it/s, loss=4.71, v_num=1]
Epoch 2: 95%|#########5| 21/22 [00:04<00:00, 4.27it/s, loss=4.7, v_num=1]
Epoch 2: 100%|##########| 22/22 [00:05<00:00, 4.39it/s, loss=4.7, v_num=1]
Epoch 2: 100%|##########| 22/22 [00:05<00:00, 4.34it/s, loss=4.7, v_num=1]
Epoch 2: 100%|##########| 22/22 [00:05<00:00, 4.34it/s, loss=4.7, v_num=1]
Epoch 2: 0%| | 0/22 [00:00<?, ?it/s, loss=4.7, v_num=1]
Epoch 3: 0%| | 0/22 [00:00<?, ?it/s, loss=4.7, v_num=1]
Epoch 3: 5%|4 | 1/22 [00:01<00:37, 1.81s/it, loss=4.7, v_num=1]
Epoch 3: 5%|4 | 1/22 [00:01<00:39, 1.87s/it, loss=4.7, v_num=1]
Epoch 3: 9%|9 | 2/22 [00:01<00:19, 1.02it/s, loss=4.7, v_num=1]
Epoch 3: 9%|9 | 2/22 [00:02<00:20, 1.01s/it, loss=4.69, v_num=1]
Epoch 3: 14%|#3 | 3/22 [00:02<00:13, 1.41it/s, loss=4.69, v_num=1]
Epoch 3: 14%|#3 | 3/22 [00:02<00:13, 1.37it/s, loss=4.69, v_num=1]
Epoch 3: 18%|#8 | 4/22 [00:02<00:10, 1.75it/s, loss=4.69, v_num=1]
Epoch 3: 18%|#8 | 4/22 [00:02<00:10, 1.71it/s, loss=4.69, v_num=1]
Epoch 3: 23%|##2 | 5/22 [00:02<00:08, 2.05it/s, loss=4.69, v_num=1]
Epoch 3: 23%|##2 | 5/22 [00:02<00:08, 2.00it/s, loss=4.69, v_num=1]
Epoch 3: 27%|##7 | 6/22 [00:02<00:06, 2.32it/s, loss=4.69, v_num=1]
Epoch 3: 27%|##7 | 6/22 [00:02<00:07, 2.26it/s, loss=4.69, v_num=1]
Epoch 3: 32%|###1 | 7/22 [00:02<00:05, 2.54it/s, loss=4.69, v_num=1]
Epoch 3: 32%|###1 | 7/22 [00:02<00:06, 2.49it/s, loss=4.69, v_num=1]
Epoch 3: 36%|###6 | 8/22 [00:02<00:05, 2.75it/s, loss=4.69, v_num=1]
Epoch 3: 36%|###6 | 8/22 [00:02<00:05, 2.70it/s, loss=4.69, v_num=1]
Epoch 3: 41%|#### | 9/22 [00:03<00:04, 2.94it/s, loss=4.69, v_num=1]
Epoch 3: 41%|#### | 9/22 [00:03<00:04, 2.88it/s, loss=4.69, v_num=1]
Epoch 3: 45%|####5 | 10/22 [00:03<00:03, 3.11it/s, loss=4.69, v_num=1]
Epoch 3: 45%|####5 | 10/22 [00:03<00:03, 3.06it/s, loss=4.69, v_num=1]
Epoch 3: 50%|##### | 11/22 [00:03<00:03, 3.27it/s, loss=4.69, v_num=1]
Epoch 3: 50%|##### | 11/22 [00:03<00:03, 3.21it/s, loss=4.69, v_num=1]
Epoch 3: 55%|#####4 | 12/22 [00:03<00:02, 3.40it/s, loss=4.69, v_num=1]
Epoch 3: 55%|#####4 | 12/22 [00:03<00:02, 3.35it/s, loss=4.68, v_num=1]
Epoch 3: 59%|#####9 | 13/22 [00:03<00:02, 3.53it/s, loss=4.68, v_num=1]
Epoch 3: 59%|#####9 | 13/22 [00:03<00:02, 3.48it/s, loss=4.68, v_num=1]
Epoch 3: 64%|######3 | 14/22 [00:03<00:02, 3.65it/s, loss=4.68, v_num=1]
Epoch 3: 64%|######3 | 14/22 [00:03<00:02, 3.60it/s, loss=4.68, v_num=1]
Epoch 3: 68%|######8 | 15/22 [00:03<00:01, 3.77it/s, loss=4.68, v_num=1]
Epoch 3: 68%|######8 | 15/22 [00:04<00:01, 3.71it/s, loss=4.68, v_num=1]
Epoch 3: 73%|#######2 | 16/22 [00:04<00:01, 3.88it/s, loss=4.68, v_num=1]
Epoch 3: 73%|#######2 | 16/22 [00:04<00:01, 3.82it/s, loss=4.68, v_num=1]
Epoch 3: 77%|#######7 | 17/22 [00:04<00:01, 3.97it/s, loss=4.68, v_num=1]
Epoch 3: 77%|#######7 | 17/22 [00:04<00:01, 3.92it/s, loss=4.67, v_num=1]
Epoch 3: 82%|########1 | 18/22 [00:04<00:00, 4.06it/s, loss=4.67, v_num=1]
Epoch 3: 82%|########1 | 18/22 [00:04<00:00, 4.01it/s, loss=4.67, v_num=1]
Epoch 3: 86%|########6 | 19/22 [00:04<00:00, 4.15it/s, loss=4.67, v_num=1]
Epoch 3: 86%|########6 | 19/22 [00:04<00:00, 4.09it/s, loss=4.67, v_num=1]
Epoch 3: 91%|######### | 20/22 [00:04<00:00, 4.23it/s, loss=4.67, v_num=1]
Epoch 3: 91%|######### | 20/22 [00:04<00:00, 4.17it/s, loss=4.67, v_num=1]
Epoch 3: 95%|#########5| 21/22 [00:04<00:00, 4.30it/s, loss=4.67, v_num=1]
Epoch 3: 95%|#########5| 21/22 [00:04<00:00, 4.25it/s, loss=4.67, v_num=1]
Epoch 3: 100%|##########| 22/22 [00:05<00:00, 4.38it/s, loss=4.67, v_num=1]
Epoch 3: 100%|##########| 22/22 [00:05<00:00, 4.32it/s, loss=4.67, v_num=1]
Epoch 3: 100%|##########| 22/22 [00:05<00:00, 4.32it/s, loss=4.67, v_num=1]
Epoch 3: 0%| | 0/22 [00:00<?, ?it/s, loss=4.67, v_num=1]
Epoch 4: 0%| | 0/22 [00:00<?, ?it/s, loss=4.67, v_num=1]
Epoch 4: 5%|4 | 1/22 [00:01<00:36, 1.74s/it, loss=4.67, v_num=1]
Epoch 4: 5%|4 | 1/22 [00:01<00:37, 1.80s/it, loss=4.67, v_num=1]
Epoch 4: 9%|9 | 2/22 [00:01<00:19, 1.05it/s, loss=4.67, v_num=1]
Epoch 4: 9%|9 | 2/22 [00:01<00:19, 1.01it/s, loss=4.66, v_num=1]
Epoch 4: 14%|#3 | 3/22 [00:02<00:13, 1.45it/s, loss=4.66, v_num=1]
Epoch 4: 14%|#3 | 3/22 [00:02<00:13, 1.41it/s, loss=4.66, v_num=1]
Epoch 4: 18%|#8 | 4/22 [00:02<00:10, 1.80it/s, loss=4.66, v_num=1]
Epoch 4: 18%|#8 | 4/22 [00:02<00:10, 1.75it/s, loss=4.65, v_num=1]
Epoch 4: 23%|##2 | 5/22 [00:02<00:08, 2.10it/s, loss=4.65, v_num=1]
Epoch 4: 23%|##2 | 5/22 [00:02<00:08, 2.05it/s, loss=4.65, v_num=1]
Epoch 4: 27%|##7 | 6/22 [00:02<00:06, 2.35it/s, loss=4.65, v_num=1]
Epoch 4: 27%|##7 | 6/22 [00:02<00:06, 2.30it/s, loss=4.65, v_num=1]
Epoch 4: 32%|###1 | 7/22 [00:02<00:05, 2.59it/s, loss=4.65, v_num=1]
Epoch 4: 32%|###1 | 7/22 [00:02<00:05, 2.53it/s, loss=4.65, v_num=1]
Epoch 4: 36%|###6 | 8/22 [00:02<00:04, 2.80it/s, loss=4.65, v_num=1]
Epoch 4: 36%|###6 | 8/22 [00:02<00:05, 2.74it/s, loss=4.65, v_num=1]
Epoch 4: 41%|#### | 9/22 [00:03<00:04, 2.99it/s, loss=4.65, v_num=1]
Epoch 4: 41%|#### | 9/22 [00:03<00:04, 2.93it/s, loss=4.64, v_num=1]
Epoch 4: 45%|####5 | 10/22 [00:03<00:03, 3.15it/s, loss=4.64, v_num=1]
Epoch 4: 45%|####5 | 10/22 [00:03<00:03, 3.10it/s, loss=4.64, v_num=1]
Epoch 4: 50%|##### | 11/22 [00:03<00:03, 3.31it/s, loss=4.64, v_num=1]
Epoch 4: 50%|##### | 11/22 [00:03<00:03, 3.25it/s, loss=4.64, v_num=1]
Epoch 4: 55%|#####4 | 12/22 [00:03<00:02, 3.46it/s, loss=4.64, v_num=1]
Epoch 4: 55%|#####4 | 12/22 [00:03<00:02, 3.39it/s, loss=4.64, v_num=1]
Epoch 4: 59%|#####9 | 13/22 [00:03<00:02, 3.59it/s, loss=4.64, v_num=1]
Epoch 4: 59%|#####9 | 13/22 [00:03<00:02, 3.53it/s, loss=4.64, v_num=1]
Epoch 4: 64%|######3 | 14/22 [00:03<00:02, 3.71it/s, loss=4.64, v_num=1]
Epoch 4: 64%|######3 | 14/22 [00:03<00:02, 3.65it/s, loss=4.64, v_num=1]
Epoch 4: 68%|######8 | 15/22 [00:03<00:01, 3.82it/s, loss=4.64, v_num=1]
Epoch 4: 68%|######8 | 15/22 [00:03<00:01, 3.76it/s, loss=4.64, v_num=1]
Epoch 4: 73%|#######2 | 16/22 [00:04<00:01, 3.93it/s, loss=4.64, v_num=1]
Epoch 4: 73%|#######2 | 16/22 [00:04<00:01, 3.87it/s, loss=4.64, v_num=1]
Epoch 4: 77%|#######7 | 17/22 [00:04<00:01, 4.02it/s, loss=4.64, v_num=1]
Epoch 4: 77%|#######7 | 17/22 [00:04<00:01, 3.97it/s, loss=4.64, v_num=1]
Epoch 4: 82%|########1 | 18/22 [00:04<00:00, 4.11it/s, loss=4.64, v_num=1]
Epoch 4: 82%|########1 | 18/22 [00:04<00:00, 4.06it/s, loss=4.64, v_num=1]
Epoch 4: 86%|########6 | 19/22 [00:04<00:00, 4.20it/s, loss=4.64, v_num=1]
Epoch 4: 86%|########6 | 19/22 [00:04<00:00, 4.14it/s, loss=4.64, v_num=1]
Epoch 4: 91%|######### | 20/22 [00:04<00:00, 4.28it/s, loss=4.64, v_num=1]
Epoch 4: 91%|######### | 20/22 [00:04<00:00, 4.22it/s, loss=4.64, v_num=1]
Epoch 4: 95%|#########5| 21/22 [00:04<00:00, 4.35it/s, loss=4.64, v_num=1]
Epoch 4: 95%|#########5| 21/22 [00:04<00:00, 4.30it/s, loss=4.64, v_num=1]
Epoch 4: 100%|##########| 22/22 [00:04<00:00, 4.42it/s, loss=4.64, v_num=1]
Epoch 4: 100%|##########| 22/22 [00:05<00:00, 4.37it/s, loss=4.64, v_num=1]
Epoch 4: 100%|##########| 22/22 [00:05<00:00, 4.37it/s, loss=4.64, v_num=1]
Epoch 4: 0%| | 0/22 [00:00<?, ?it/s, loss=4.64, v_num=1]
Epoch 5: 0%| | 0/22 [00:00<?, ?it/s, loss=4.64, v_num=1]
Epoch 5: 5%|4 | 1/22 [00:01<00:36, 1.74s/it, loss=4.64, v_num=1]
Epoch 5: 5%|4 | 1/22 [00:01<00:37, 1.80s/it, loss=4.64, v_num=1]
Epoch 5: 9%|9 | 2/22 [00:01<00:18, 1.05it/s, loss=4.64, v_num=1]
Epoch 5: 9%|9 | 2/22 [00:01<00:19, 1.02it/s, loss=4.64, v_num=1]
Epoch 5: 14%|#3 | 3/22 [00:02<00:12, 1.47it/s, loss=4.64, v_num=1]
Epoch 5: 14%|#3 | 3/22 [00:02<00:13, 1.42it/s, loss=4.64, v_num=1]
Epoch 5: 18%|#8 | 4/22 [00:02<00:09, 1.82it/s, loss=4.64, v_num=1]
Epoch 5: 18%|#8 | 4/22 [00:02<00:10, 1.77it/s, loss=4.64, v_num=1]
Epoch 5: 23%|##2 | 5/22 [00:02<00:08, 2.12it/s, loss=4.64, v_num=1]
Epoch 5: 23%|##2 | 5/22 [00:02<00:08, 2.07it/s, loss=4.64, v_num=1]
Epoch 5: 27%|##7 | 6/22 [00:02<00:06, 2.38it/s, loss=4.64, v_num=1]
Epoch 5: 27%|##7 | 6/22 [00:02<00:06, 2.33it/s, loss=4.64, v_num=1]
Epoch 5: 32%|###1 | 7/22 [00:02<00:05, 2.62it/s, loss=4.64, v_num=1]
Epoch 5: 32%|###1 | 7/22 [00:02<00:05, 2.56it/s, loss=4.64, v_num=1]
Epoch 5: 36%|###6 | 8/22 [00:02<00:04, 2.84it/s, loss=4.64, v_num=1]
Epoch 5: 36%|###6 | 8/22 [00:02<00:05, 2.78it/s, loss=4.64, v_num=1]
Epoch 5: 41%|#### | 9/22 [00:02<00:04, 3.03it/s, loss=4.64, v_num=1]
Epoch 5: 41%|#### | 9/22 [00:03<00:04, 2.97it/s, loss=4.63, v_num=1]
Epoch 5: 45%|####5 | 10/22 [00:03<00:03, 3.20it/s, loss=4.63, v_num=1]
Epoch 5: 45%|####5 | 10/22 [00:03<00:03, 3.14it/s, loss=4.63, v_num=1]
Epoch 5: 50%|##### | 11/22 [00:03<00:03, 3.36it/s, loss=4.63, v_num=1]
Epoch 5: 50%|##### | 11/22 [00:03<00:03, 3.30it/s, loss=4.63, v_num=1]
Epoch 5: 55%|#####4 | 12/22 [00:03<00:02, 3.45it/s, loss=4.63, v_num=1]
Epoch 5: 55%|#####4 | 12/22 [00:03<00:02, 3.38it/s, loss=4.63, v_num=1]
Epoch 5: 59%|#####9 | 13/22 [00:03<00:02, 3.57it/s, loss=4.63, v_num=1]
Epoch 5: 59%|#####9 | 13/22 [00:03<00:02, 3.52it/s, loss=4.63, v_num=1]
Epoch 5: 64%|######3 | 14/22 [00:03<00:02, 3.70it/s, loss=4.63, v_num=1]
Epoch 5: 64%|######3 | 14/22 [00:03<00:02, 3.64it/s, loss=4.63, v_num=1]
Epoch 5: 68%|######8 | 15/22 [00:03<00:01, 3.81it/s, loss=4.63, v_num=1]
Epoch 5: 68%|######8 | 15/22 [00:03<00:01, 3.75it/s, loss=4.63, v_num=1]
Epoch 5: 73%|#######2 | 16/22 [00:04<00:01, 3.92it/s, loss=4.63, v_num=1]
Epoch 5: 73%|#######2 | 16/22 [00:04<00:01, 3.86it/s, loss=4.63, v_num=1]
Epoch 5: 77%|#######7 | 17/22 [00:04<00:01, 4.01it/s, loss=4.63, v_num=1]
Epoch 5: 77%|#######7 | 17/22 [00:04<00:01, 3.96it/s, loss=4.63, v_num=1]
Epoch 5: 82%|########1 | 18/22 [00:04<00:00, 4.10it/s, loss=4.63, v_num=1]
Epoch 5: 82%|########1 | 18/22 [00:04<00:00, 4.05it/s, loss=4.63, v_num=1]
Epoch 5: 86%|########6 | 19/22 [00:04<00:00, 4.19it/s, loss=4.63, v_num=1]
Epoch 5: 86%|########6 | 19/22 [00:04<00:00, 4.13it/s, loss=4.63, v_num=1]
Epoch 5: 91%|######### | 20/22 [00:04<00:00, 4.27it/s, loss=4.63, v_num=1]
Epoch 5: 91%|######### | 20/22 [00:04<00:00, 4.21it/s, loss=4.63, v_num=1]
Epoch 5: 95%|#########5| 21/22 [00:04<00:00, 4.34it/s, loss=4.63, v_num=1]
Epoch 5: 95%|#########5| 21/22 [00:04<00:00, 4.29it/s, loss=4.63, v_num=1]
Epoch 5: 100%|##########| 22/22 [00:04<00:00, 4.41it/s, loss=4.63, v_num=1]
Epoch 5: 100%|##########| 22/22 [00:05<00:00, 4.36it/s, loss=4.63, v_num=1]
Epoch 5: 100%|##########| 22/22 [00:05<00:00, 4.36it/s, loss=4.63, v_num=1]
Epoch 5: 0%| | 0/22 [00:00<?, ?it/s, loss=4.63, v_num=1]
Epoch 6: 0%| | 0/22 [00:00<?, ?it/s, loss=4.63, v_num=1]
Epoch 6: 5%|4 | 1/22 [00:01<00:37, 1.79s/it, loss=4.63, v_num=1]
Epoch 6: 5%|4 | 1/22 [00:01<00:38, 1.85s/it, loss=4.63, v_num=1]
Epoch 6: 9%|9 | 2/22 [00:01<00:19, 1.03it/s, loss=4.63, v_num=1]
Epoch 6: 9%|9 | 2/22 [00:02<00:20, 1.00s/it, loss=4.63, v_num=1]
Epoch 6: 14%|#3 | 3/22 [00:02<00:13, 1.42it/s, loss=4.63, v_num=1]
Epoch 6: 14%|#3 | 3/22 [00:02<00:13, 1.38it/s, loss=4.63, v_num=1]
Epoch 6: 18%|#8 | 4/22 [00:02<00:10, 1.77it/s, loss=4.63, v_num=1]
Epoch 6: 18%|#8 | 4/22 [00:02<00:10, 1.72it/s, loss=4.63, v_num=1]
Epoch 6: 23%|##2 | 5/22 [00:02<00:08, 2.07it/s, loss=4.63, v_num=1]
Epoch 6: 23%|##2 | 5/22 [00:02<00:08, 2.02it/s, loss=4.62, v_num=1]
Epoch 6: 27%|##7 | 6/22 [00:02<00:06, 2.32it/s, loss=4.62, v_num=1]
Epoch 6: 27%|##7 | 6/22 [00:02<00:07, 2.27it/s, loss=4.62, v_num=1]
Epoch 6: 32%|###1 | 7/22 [00:02<00:05, 2.56it/s, loss=4.62, v_num=1]
Epoch 6: 32%|###1 | 7/22 [00:02<00:05, 2.50it/s, loss=4.62, v_num=1]
Epoch 6: 36%|###6 | 8/22 [00:02<00:05, 2.77it/s, loss=4.62, v_num=1]
Epoch 6: 36%|###6 | 8/22 [00:02<00:05, 2.71it/s, loss=4.62, v_num=1]
Epoch 6: 41%|#### | 9/22 [00:03<00:04, 2.95it/s, loss=4.62, v_num=1]
Epoch 6: 41%|#### | 9/22 [00:03<00:04, 2.90it/s, loss=4.62, v_num=1]
Epoch 6: 45%|####5 | 10/22 [00:03<00:03, 3.12it/s, loss=4.62, v_num=1]
Epoch 6: 45%|####5 | 10/22 [00:03<00:03, 3.07it/s, loss=4.62, v_num=1]
Epoch 6: 50%|##### | 11/22 [00:03<00:03, 3.28it/s, loss=4.62, v_num=1]
Epoch 6: 50%|##### | 11/22 [00:03<00:03, 3.22it/s, loss=4.62, v_num=1]
Epoch 6: 55%|#####4 | 12/22 [00:03<00:02, 3.42it/s, loss=4.62, v_num=1]
Epoch 6: 55%|#####4 | 12/22 [00:03<00:02, 3.36it/s, loss=4.62, v_num=1]
Epoch 6: 59%|#####9 | 13/22 [00:03<00:02, 3.55it/s, loss=4.62, v_num=1]
Epoch 6: 59%|#####9 | 13/22 [00:03<00:02, 3.49it/s, loss=4.62, v_num=1]
Epoch 6: 64%|######3 | 14/22 [00:03<00:02, 3.67it/s, loss=4.62, v_num=1]
Epoch 6: 64%|######3 | 14/22 [00:03<00:02, 3.61it/s, loss=4.62, v_num=1]
Epoch 6: 68%|######8 | 15/22 [00:03<00:01, 3.78it/s, loss=4.62, v_num=1]
Epoch 6: 68%|######8 | 15/22 [00:04<00:01, 3.73it/s, loss=4.62, v_num=1]
Epoch 6: 73%|#######2 | 16/22 [00:04<00:01, 3.89it/s, loss=4.62, v_num=1]
Epoch 6: 73%|#######2 | 16/22 [00:04<00:01, 3.83it/s, loss=4.61, v_num=1]
Epoch 6: 77%|#######7 | 17/22 [00:04<00:01, 3.98it/s, loss=4.61, v_num=1]
Epoch 6: 77%|#######7 | 17/22 [00:04<00:01, 3.92it/s, loss=4.61, v_num=1]
Epoch 6: 82%|########1 | 18/22 [00:04<00:00, 4.07it/s, loss=4.61, v_num=1]
Epoch 6: 82%|########1 | 18/22 [00:04<00:00, 4.01it/s, loss=4.61, v_num=1]
Epoch 6: 86%|########6 | 19/22 [00:04<00:00, 4.15it/s, loss=4.61, v_num=1]
Epoch 6: 86%|########6 | 19/22 [00:04<00:00, 4.09it/s, loss=4.61, v_num=1]
Epoch 6: 91%|######### | 20/22 [00:04<00:00, 4.23it/s, loss=4.61, v_num=1]
Epoch 6: 91%|######### | 20/22 [00:04<00:00, 4.17it/s, loss=4.61, v_num=1]
Epoch 6: 95%|#########5| 21/22 [00:04<00:00, 4.29it/s, loss=4.61, v_num=1]
Epoch 6: 95%|#########5| 21/22 [00:04<00:00, 4.24it/s, loss=4.61, v_num=1]
Epoch 6: 100%|##########| 22/22 [00:05<00:00, 4.36it/s, loss=4.61, v_num=1]
Epoch 6: 100%|##########| 22/22 [00:05<00:00, 4.31it/s, loss=4.61, v_num=1]
Epoch 6: 100%|##########| 22/22 [00:05<00:00, 4.31it/s, loss=4.61, v_num=1]
Epoch 6: 0%| | 0/22 [00:00<?, ?it/s, loss=4.61, v_num=1]
Epoch 7: 0%| | 0/22 [00:00<?, ?it/s, loss=4.61, v_num=1]
Epoch 7: 5%|4 | 1/22 [00:01<00:38, 1.82s/it, loss=4.61, v_num=1]
Epoch 7: 5%|4 | 1/22 [00:01<00:39, 1.88s/it, loss=4.61, v_num=1]
Epoch 7: 9%|9 | 2/22 [00:01<00:19, 1.01it/s, loss=4.61, v_num=1]
Epoch 7: 9%|9 | 2/22 [00:02<00:20, 1.02s/it, loss=4.61, v_num=1]
Epoch 7: 14%|#3 | 3/22 [00:02<00:13, 1.40it/s, loss=4.61, v_num=1]
Epoch 7: 14%|#3 | 3/22 [00:02<00:13, 1.37it/s, loss=4.61, v_num=1]
Epoch 7: 18%|#8 | 4/22 [00:02<00:10, 1.74it/s, loss=4.61, v_num=1]
Epoch 7: 18%|#8 | 4/22 [00:02<00:10, 1.70it/s, loss=4.61, v_num=1]
Epoch 7: 23%|##2 | 5/22 [00:02<00:08, 2.04it/s, loss=4.61, v_num=1]
Epoch 7: 23%|##2 | 5/22 [00:02<00:08, 1.99it/s, loss=4.6, v_num=1]
Epoch 7: 27%|##7 | 6/22 [00:02<00:06, 2.29it/s, loss=4.6, v_num=1]
Epoch 7: 27%|##7 | 6/22 [00:02<00:07, 2.24it/s, loss=4.61, v_num=1]
Epoch 7: 32%|###1 | 7/22 [00:02<00:05, 2.53it/s, loss=4.61, v_num=1]
Epoch 7: 32%|###1 | 7/22 [00:02<00:06, 2.47it/s, loss=4.61, v_num=1]
Epoch 7: 36%|###6 | 8/22 [00:02<00:05, 2.74it/s, loss=4.61, v_num=1]
Epoch 7: 36%|###6 | 8/22 [00:02<00:05, 2.68it/s, loss=4.6, v_num=1]
Epoch 7: 41%|#### | 9/22 [00:03<00:04, 2.92it/s, loss=4.6, v_num=1]
Epoch 7: 41%|#### | 9/22 [00:03<00:04, 2.87it/s, loss=4.6, v_num=1]
Epoch 7: 45%|####5 | 10/22 [00:03<00:03, 3.10it/s, loss=4.6, v_num=1]
Epoch 7: 45%|####5 | 10/22 [00:03<00:03, 3.04it/s, loss=4.6, v_num=1]
Epoch 7: 50%|##### | 11/22 [00:03<00:03, 3.26it/s, loss=4.6, v_num=1]
Epoch 7: 50%|##### | 11/22 [00:03<00:03, 3.19it/s, loss=4.6, v_num=1]
Epoch 7: 55%|#####4 | 12/22 [00:03<00:02, 3.39it/s, loss=4.6, v_num=1]
Epoch 7: 55%|#####4 | 12/22 [00:03<00:03, 3.33it/s, loss=4.6, v_num=1]
Epoch 7: 59%|#####9 | 13/22 [00:03<00:02, 3.51it/s, loss=4.6, v_num=1]
Epoch 7: 59%|#####9 | 13/22 [00:03<00:02, 3.46it/s, loss=4.6, v_num=1]
Epoch 7: 64%|######3 | 14/22 [00:03<00:02, 3.64it/s, loss=4.6, v_num=1]
Epoch 7: 64%|######3 | 14/22 [00:03<00:02, 3.58it/s, loss=4.6, v_num=1]
Epoch 7: 68%|######8 | 15/22 [00:03<00:01, 3.75it/s, loss=4.6, v_num=1]
Epoch 7: 68%|######8 | 15/22 [00:04<00:01, 3.70it/s, loss=4.6, v_num=1]
Epoch 7: 73%|#######2 | 16/22 [00:04<00:01, 3.86it/s, loss=4.6, v_num=1]
Epoch 7: 73%|#######2 | 16/22 [00:04<00:01, 3.80it/s, loss=4.6, v_num=1]
Epoch 7: 77%|#######7 | 17/22 [00:04<00:01, 3.95it/s, loss=4.6, v_num=1]
Epoch 7: 77%|#######7 | 17/22 [00:04<00:01, 3.90it/s, loss=4.6, v_num=1]
Epoch 7: 82%|########1 | 18/22 [00:04<00:00, 4.04it/s, loss=4.6, v_num=1]
Epoch 7: 82%|########1 | 18/22 [00:04<00:01, 3.99it/s, loss=4.6, v_num=1]
Epoch 7: 86%|########6 | 19/22 [00:04<00:00, 4.13it/s, loss=4.6, v_num=1]
Epoch 7: 86%|########6 | 19/22 [00:04<00:00, 4.08it/s, loss=4.6, v_num=1]
Epoch 7: 91%|######### | 20/22 [00:04<00:00, 4.20it/s, loss=4.6, v_num=1]
Epoch 7: 91%|######### | 20/22 [00:04<00:00, 4.15it/s, loss=4.6, v_num=1]
Epoch 7: 95%|#########5| 21/22 [00:04<00:00, 4.27it/s, loss=4.6, v_num=1]
Epoch 7: 95%|#########5| 21/22 [00:04<00:00, 4.22it/s, loss=4.6, v_num=1]
Epoch 7: 100%|##########| 22/22 [00:05<00:00, 4.34it/s, loss=4.6, v_num=1]
Epoch 7: 100%|##########| 22/22 [00:05<00:00, 4.29it/s, loss=4.6, v_num=1]
Epoch 7: 100%|##########| 22/22 [00:05<00:00, 4.29it/s, loss=4.6, v_num=1]
Epoch 7: 0%| | 0/22 [00:00<?, ?it/s, loss=4.6, v_num=1]
Epoch 8: 0%| | 0/22 [00:00<?, ?it/s, loss=4.6, v_num=1]
Epoch 8: 5%|4 | 1/22 [00:01<00:37, 1.78s/it, loss=4.6, v_num=1]
Epoch 8: 5%|4 | 1/22 [00:01<00:38, 1.84s/it, loss=4.6, v_num=1]
Epoch 8: 9%|9 | 2/22 [00:01<00:19, 1.03it/s, loss=4.6, v_num=1]
Epoch 8: 9%|9 | 2/22 [00:02<00:20, 1.00s/it, loss=4.6, v_num=1]
Epoch 8: 14%|#3 | 3/22 [00:02<00:13, 1.43it/s, loss=4.6, v_num=1]
Epoch 8: 14%|#3 | 3/22 [00:02<00:13, 1.39it/s, loss=4.6, v_num=1]
Epoch 8: 18%|#8 | 4/22 [00:02<00:10, 1.77it/s, loss=4.6, v_num=1]
Epoch 8: 18%|#8 | 4/22 [00:02<00:10, 1.72it/s, loss=4.59, v_num=1]
Epoch 8: 23%|##2 | 5/22 [00:02<00:08, 2.07it/s, loss=4.59, v_num=1]
Epoch 8: 23%|##2 | 5/22 [00:02<00:08, 2.02it/s, loss=4.59, v_num=1]
Epoch 8: 27%|##7 | 6/22 [00:02<00:06, 2.31it/s, loss=4.59, v_num=1]
Epoch 8: 27%|##7 | 6/22 [00:02<00:07, 2.26it/s, loss=4.59, v_num=1]
Epoch 8: 32%|###1 | 7/22 [00:02<00:05, 2.54it/s, loss=4.59, v_num=1]
Epoch 8: 32%|###1 | 7/22 [00:02<00:06, 2.48it/s, loss=4.59, v_num=1]
Epoch 8: 36%|###6 | 8/22 [00:02<00:05, 2.75it/s, loss=4.59, v_num=1]
Epoch 8: 36%|###6 | 8/22 [00:02<00:05, 2.69it/s, loss=4.59, v_num=1]
Epoch 8: 41%|#### | 9/22 [00:03<00:04, 2.93it/s, loss=4.59, v_num=1]
Epoch 8: 41%|#### | 9/22 [00:03<00:04, 2.88it/s, loss=4.59, v_num=1]
Epoch 8: 45%|####5 | 10/22 [00:03<00:03, 3.10it/s, loss=4.59, v_num=1]
Epoch 8: 45%|####5 | 10/22 [00:03<00:03, 3.05it/s, loss=4.59, v_num=1]
Epoch 8: 50%|##### | 11/22 [00:03<00:03, 3.26it/s, loss=4.59, v_num=1]
Epoch 8: 50%|##### | 11/22 [00:03<00:03, 3.20it/s, loss=4.59, v_num=1]
Epoch 8: 55%|#####4 | 12/22 [00:03<00:02, 3.38it/s, loss=4.59, v_num=1]
Epoch 8: 55%|#####4 | 12/22 [00:03<00:03, 3.33it/s, loss=4.59, v_num=1]
Epoch 8: 59%|#####9 | 13/22 [00:03<00:02, 3.51it/s, loss=4.59, v_num=1]
Epoch 8: 59%|#####9 | 13/22 [00:03<00:02, 3.46it/s, loss=4.58, v_num=1]
Epoch 8: 64%|######3 | 14/22 [00:03<00:02, 3.64it/s, loss=4.58, v_num=1]
Epoch 8: 64%|######3 | 14/22 [00:03<00:02, 3.58it/s, loss=4.59, v_num=1]
Epoch 8: 68%|######8 | 15/22 [00:04<00:01, 3.75it/s, loss=4.59, v_num=1]
Epoch 8: 68%|######8 | 15/22 [00:04<00:01, 3.69it/s, loss=4.59, v_num=1]
Epoch 8: 73%|#######2 | 16/22 [00:04<00:01, 3.85it/s, loss=4.59, v_num=1]
Epoch 8: 73%|#######2 | 16/22 [00:04<00:01, 3.79it/s, loss=4.59, v_num=1]
Epoch 8: 77%|#######7 | 17/22 [00:04<00:01, 3.94it/s, loss=4.59, v_num=1]
Epoch 8: 77%|#######7 | 17/22 [00:04<00:01, 3.89it/s, loss=4.58, v_num=1]
Epoch 8: 82%|########1 | 18/22 [00:04<00:00, 4.03it/s, loss=4.58, v_num=1]
Epoch 8: 82%|########1 | 18/22 [00:04<00:01, 3.98it/s, loss=4.58, v_num=1]
Epoch 8: 86%|########6 | 19/22 [00:04<00:00, 4.12it/s, loss=4.58, v_num=1]
Epoch 8: 86%|########6 | 19/22 [00:04<00:00, 4.06it/s, loss=4.58, v_num=1]
Epoch 8: 91%|######### | 20/22 [00:04<00:00, 4.20it/s, loss=4.58, v_num=1]
Epoch 8: 91%|######### | 20/22 [00:04<00:00, 4.14it/s, loss=4.58, v_num=1]
Epoch 8: 95%|#########5| 21/22 [00:04<00:00, 4.27it/s, loss=4.58, v_num=1]
Epoch 8: 95%|#########5| 21/22 [00:04<00:00, 4.22it/s, loss=4.58, v_num=1]
Epoch 8: 100%|##########| 22/22 [00:05<00:00, 4.34it/s, loss=4.58, v_num=1]
Epoch 8: 100%|##########| 22/22 [00:05<00:00, 4.29it/s, loss=4.58, v_num=1]
Epoch 8: 100%|##########| 22/22 [00:05<00:00, 4.29it/s, loss=4.58, v_num=1]
Epoch 8: 0%| | 0/22 [00:00<?, ?it/s, loss=4.58, v_num=1]
Epoch 9: 0%| | 0/22 [00:00<?, ?it/s, loss=4.58, v_num=1]
Epoch 9: 5%|4 | 1/22 [00:01<00:38, 1.82s/it, loss=4.58, v_num=1]
Epoch 9: 5%|4 | 1/22 [00:01<00:39, 1.88s/it, loss=4.58, v_num=1]
Epoch 9: 9%|9 | 2/22 [00:01<00:19, 1.01it/s, loss=4.58, v_num=1]
Epoch 9: 9%|9 | 2/22 [00:02<00:20, 1.02s/it, loss=4.58, v_num=1]
Epoch 9: 14%|#3 | 3/22 [00:02<00:13, 1.40it/s, loss=4.58, v_num=1]
Epoch 9: 14%|#3 | 3/22 [00:02<00:13, 1.37it/s, loss=4.58, v_num=1]
Epoch 9: 18%|#8 | 4/22 [00:02<00:10, 1.74it/s, loss=4.58, v_num=1]
Epoch 9: 18%|#8 | 4/22 [00:02<00:10, 1.70it/s, loss=4.58, v_num=1]
Epoch 9: 23%|##2 | 5/22 [00:02<00:08, 2.04it/s, loss=4.58, v_num=1]
Epoch 9: 23%|##2 | 5/22 [00:02<00:08, 1.99it/s, loss=4.58, v_num=1]
Epoch 9: 27%|##7 | 6/22 [00:02<00:06, 2.30it/s, loss=4.58, v_num=1]
Epoch 9: 27%|##7 | 6/22 [00:02<00:07, 2.24it/s, loss=4.58, v_num=1]
Epoch 9: 32%|###1 | 7/22 [00:02<00:05, 2.52it/s, loss=4.58, v_num=1]
Epoch 9: 32%|###1 | 7/22 [00:02<00:06, 2.47it/s, loss=4.58, v_num=1]
Epoch 9: 36%|###6 | 8/22 [00:02<00:05, 2.73it/s, loss=4.58, v_num=1]
Epoch 9: 36%|###6 | 8/22 [00:02<00:05, 2.68it/s, loss=4.58, v_num=1]
Epoch 9: 41%|#### | 9/22 [00:03<00:04, 2.92it/s, loss=4.58, v_num=1]
Epoch 9: 41%|#### | 9/22 [00:03<00:04, 2.86it/s, loss=4.58, v_num=1]
Epoch 9: 45%|####5 | 10/22 [00:03<00:03, 3.09it/s, loss=4.58, v_num=1]
Epoch 9: 45%|####5 | 10/22 [00:03<00:03, 3.03it/s, loss=4.58, v_num=1]
Epoch 9: 50%|##### | 11/22 [00:03<00:03, 3.24it/s, loss=4.58, v_num=1]
Epoch 9: 50%|##### | 11/22 [00:03<00:03, 3.18it/s, loss=4.58, v_num=1]
Epoch 9: 55%|#####4 | 12/22 [00:03<00:02, 3.35it/s, loss=4.58, v_num=1]
Epoch 9: 55%|#####4 | 12/22 [00:03<00:03, 3.29it/s, loss=4.57, v_num=1]
Epoch 9: 59%|#####9 | 13/22 [00:03<00:02, 3.48it/s, loss=4.57, v_num=1]
Epoch 9: 59%|#####9 | 13/22 [00:03<00:02, 3.42it/s, loss=4.57, v_num=1]
Epoch 9: 64%|######3 | 14/22 [00:03<00:02, 3.60it/s, loss=4.57, v_num=1]
Epoch 9: 64%|######3 | 14/22 [00:03<00:02, 3.54it/s, loss=4.57, v_num=1]
Epoch 9: 68%|######8 | 15/22 [00:04<00:01, 3.71it/s, loss=4.57, v_num=1]
Epoch 9: 68%|######8 | 15/22 [00:04<00:01, 3.65it/s, loss=4.58, v_num=1]
Epoch 9: 73%|#######2 | 16/22 [00:04<00:01, 3.81it/s, loss=4.58, v_num=1]
Epoch 9: 73%|#######2 | 16/22 [00:04<00:01, 3.76it/s, loss=4.58, v_num=1]
Epoch 9: 77%|#######7 | 17/22 [00:04<00:01, 3.91it/s, loss=4.58, v_num=1]
Epoch 9: 77%|#######7 | 17/22 [00:04<00:01, 3.86it/s, loss=4.58, v_num=1]
Epoch 9: 82%|########1 | 18/22 [00:04<00:00, 4.00it/s, loss=4.58, v_num=1]
Epoch 9: 82%|########1 | 18/22 [00:04<00:01, 3.95it/s, loss=4.58, v_num=1]
Epoch 9: 86%|########6 | 19/22 [00:04<00:00, 4.09it/s, loss=4.58, v_num=1]
Epoch 9: 86%|########6 | 19/22 [00:04<00:00, 4.03it/s, loss=4.58, v_num=1]
Epoch 9: 91%|######### | 20/22 [00:04<00:00, 4.17it/s, loss=4.58, v_num=1]
Epoch 9: 91%|######### | 20/22 [00:04<00:00, 4.12it/s, loss=4.58, v_num=1]
Epoch 9: 95%|#########5| 21/22 [00:04<00:00, 4.24it/s, loss=4.58, v_num=1]
Epoch 9: 95%|#########5| 21/22 [00:05<00:00, 4.19it/s, loss=4.58, v_num=1]
Epoch 9: 100%|##########| 22/22 [00:05<00:00, 4.32it/s, loss=4.58, v_num=1]
Epoch 9: 100%|##########| 22/22 [00:05<00:00, 4.27it/s, loss=4.58, v_num=1]
Epoch 9: 100%|##########| 22/22 [00:05<00:00, 4.27it/s, loss=4.58, v_num=1]
Epoch 9: 0%| | 0/22 [00:00<?, ?it/s, loss=4.58, v_num=1]
Epoch 10: 0%| | 0/22 [00:00<?, ?it/s, loss=4.58, v_num=1]
Epoch 10: 5%|4 | 1/22 [00:01<00:37, 1.79s/it, loss=4.58, v_num=1]
Epoch 10: 5%|4 | 1/22 [00:01<00:38, 1.84s/it, loss=4.58, v_num=1]
Epoch 10: 9%|9 | 2/22 [00:01<00:19, 1.03it/s, loss=4.58, v_num=1]
Epoch 10: 9%|9 | 2/22 [00:02<00:20, 1.00s/it, loss=4.58, v_num=1]
Epoch 10: 14%|#3 | 3/22 [00:02<00:13, 1.43it/s, loss=4.58, v_num=1]
Epoch 10: 14%|#3 | 3/22 [00:02<00:13, 1.39it/s, loss=4.58, v_num=1]
Epoch 10: 18%|#8 | 4/22 [00:02<00:10, 1.77it/s, loss=4.58, v_num=1]
Epoch 10: 18%|#8 | 4/22 [00:02<00:10, 1.73it/s, loss=4.58, v_num=1]
Epoch 10: 23%|##2 | 5/22 [00:02<00:08, 2.08it/s, loss=4.58, v_num=1]
Epoch 10: 23%|##2 | 5/22 [00:02<00:08, 2.03it/s, loss=4.57, v_num=1]
Epoch 10: 27%|##7 | 6/22 [00:02<00:06, 2.33it/s, loss=4.57, v_num=1]
Epoch 10: 27%|##7 | 6/22 [00:02<00:07, 2.28it/s, loss=4.57, v_num=1]
Epoch 10: 32%|###1 | 7/22 [00:02<00:05, 2.57it/s, loss=4.57, v_num=1]
Epoch 10: 32%|###1 | 7/22 [00:02<00:05, 2.51it/s, loss=4.57, v_num=1]
Epoch 10: 36%|###6 | 8/22 [00:02<00:05, 2.78it/s, loss=4.57, v_num=1]
Epoch 10: 36%|###6 | 8/22 [00:02<00:05, 2.72it/s, loss=4.57, v_num=1]
Epoch 10: 41%|#### | 9/22 [00:03<00:04, 2.96it/s, loss=4.57, v_num=1]
Epoch 10: 41%|#### | 9/22 [00:03<00:04, 2.90it/s, loss=4.57, v_num=1]
Epoch 10: 45%|####5 | 10/22 [00:03<00:03, 3.13it/s, loss=4.57, v_num=1]
Epoch 10: 45%|####5 | 10/22 [00:03<00:03, 3.07it/s, loss=4.57, v_num=1]
Epoch 10: 50%|##### | 11/22 [00:03<00:03, 3.29it/s, loss=4.57, v_num=1]
Epoch 10: 50%|##### | 11/22 [00:03<00:03, 3.23it/s, loss=4.57, v_num=1]
Epoch 10: 55%|#####4 | 12/22 [00:03<00:02, 3.41it/s, loss=4.57, v_num=1]
Epoch 10: 55%|#####4 | 12/22 [00:03<00:02, 3.36it/s, loss=4.57, v_num=1]
Epoch 10: 59%|#####9 | 13/22 [00:03<00:02, 3.55it/s, loss=4.57, v_num=1]
Epoch 10: 59%|#####9 | 13/22 [00:03<00:02, 3.49it/s, loss=4.57, v_num=1]
Epoch 10: 64%|######3 | 14/22 [00:03<00:02, 3.67it/s, loss=4.57, v_num=1]
Epoch 10: 64%|######3 | 14/22 [00:03<00:02, 3.61it/s, loss=4.57, v_num=1]
Epoch 10: 68%|######8 | 15/22 [00:03<00:01, 3.78it/s, loss=4.57, v_num=1]
Epoch 10: 68%|######8 | 15/22 [00:04<00:01, 3.73it/s, loss=4.56, v_num=1]
Epoch 10: 73%|#######2 | 16/22 [00:04<00:01, 3.89it/s, loss=4.56, v_num=1]
Epoch 10: 73%|#######2 | 16/22 [00:04<00:01, 3.83it/s, loss=4.57, v_num=1]
Epoch 10: 77%|#######7 | 17/22 [00:04<00:01, 3.98it/s, loss=4.57, v_num=1]
Epoch 10: 77%|#######7 | 17/22 [00:04<00:01, 3.92it/s, loss=4.56, v_num=1]
Epoch 10: 82%|########1 | 18/22 [00:04<00:00, 4.08it/s, loss=4.56, v_num=1]
Epoch 10: 82%|########1 | 18/22 [00:04<00:00, 4.02it/s, loss=4.56, v_num=1]
Epoch 10: 86%|########6 | 19/22 [00:04<00:00, 4.16it/s, loss=4.56, v_num=1]
Epoch 10: 86%|########6 | 19/22 [00:04<00:00, 4.10it/s, loss=4.56, v_num=1]
Epoch 10: 91%|######### | 20/22 [00:04<00:00, 4.24it/s, loss=4.56, v_num=1]
Epoch 10: 91%|######### | 20/22 [00:04<00:00, 4.18it/s, loss=4.57, v_num=1]
Epoch 10: 95%|#########5| 21/22 [00:04<00:00, 4.31it/s, loss=4.57, v_num=1]
Epoch 10: 95%|#########5| 21/22 [00:04<00:00, 4.26it/s, loss=4.57, v_num=1]
Epoch 10: 100%|##########| 22/22 [00:05<00:00, 4.39it/s, loss=4.57, v_num=1]
Epoch 10: 100%|##########| 22/22 [00:05<00:00, 4.33it/s, loss=4.57, v_num=1]
Epoch 10: 100%|##########| 22/22 [00:05<00:00, 4.33it/s, loss=4.57, v_num=1]
Epoch 10: 0%| | 0/22 [00:00<?, ?it/s, loss=4.57, v_num=1]
Epoch 11: 0%| | 0/22 [00:00<?, ?it/s, loss=4.57, v_num=1]
Epoch 11: 5%|4 | 1/22 [00:01<00:37, 1.80s/it, loss=4.57, v_num=1]
Epoch 11: 5%|4 | 1/22 [00:01<00:39, 1.86s/it, loss=4.57, v_num=1]
Epoch 11: 9%|9 | 2/22 [00:01<00:19, 1.02it/s, loss=4.57, v_num=1]
Epoch 11: 9%|9 | 2/22 [00:02<00:20, 1.01s/it, loss=4.57, v_num=1]
Epoch 11: 14%|#3 | 3/22 [00:02<00:13, 1.41it/s, loss=4.57, v_num=1]
Epoch 11: 14%|#3 | 3/22 [00:02<00:13, 1.38it/s, loss=4.57, v_num=1]
Epoch 11: 18%|#8 | 4/22 [00:02<00:10, 1.76it/s, loss=4.57, v_num=1]
Epoch 11: 18%|#8 | 4/22 [00:02<00:10, 1.72it/s, loss=4.57, v_num=1]
Epoch 11: 23%|##2 | 5/22 [00:02<00:08, 2.06it/s, loss=4.57, v_num=1]
Epoch 11: 23%|##2 | 5/22 [00:02<00:08, 2.01it/s, loss=4.57, v_num=1]
Epoch 11: 27%|##7 | 6/22 [00:02<00:06, 2.33it/s, loss=4.57, v_num=1]
Epoch 11: 27%|##7 | 6/22 [00:02<00:07, 2.27it/s, loss=4.57, v_num=1]
Epoch 11: 32%|###1 | 7/22 [00:02<00:05, 2.55it/s, loss=4.57, v_num=1]
Epoch 11: 32%|###1 | 7/22 [00:02<00:06, 2.49it/s, loss=4.57, v_num=1]
Epoch 11: 36%|###6 | 8/22 [00:02<00:05, 2.76it/s, loss=4.57, v_num=1]
Epoch 11: 36%|###6 | 8/22 [00:02<00:05, 2.70it/s, loss=4.57, v_num=1]
Epoch 11: 41%|#### | 9/22 [00:03<00:04, 2.94it/s, loss=4.57, v_num=1]
Epoch 11: 41%|#### | 9/22 [00:03<00:04, 2.89it/s, loss=4.57, v_num=1]
Epoch 11: 45%|####5 | 10/22 [00:03<00:03, 3.12it/s, loss=4.57, v_num=1]
Epoch 11: 45%|####5 | 10/22 [00:03<00:03, 3.06it/s, loss=4.57, v_num=1]
Epoch 11: 50%|##### | 11/22 [00:03<00:03, 3.28it/s, loss=4.57, v_num=1]
Epoch 11: 50%|##### | 11/22 [00:03<00:03, 3.22it/s, loss=4.57, v_num=1]
Epoch 11: 55%|#####4 | 12/22 [00:03<00:02, 3.41it/s, loss=4.57, v_num=1]
Epoch 11: 55%|#####4 | 12/22 [00:03<00:02, 3.36it/s, loss=4.57, v_num=1]
Epoch 11: 59%|#####9 | 13/22 [00:03<00:02, 3.52it/s, loss=4.57, v_num=1]
Epoch 11: 59%|#####9 | 13/22 [00:03<00:02, 3.46it/s, loss=4.57, v_num=1]
Epoch 11: 64%|######3 | 14/22 [00:03<00:02, 3.65it/s, loss=4.57, v_num=1]
Epoch 11: 64%|######3 | 14/22 [00:03<00:02, 3.59it/s, loss=4.57, v_num=1]
Epoch 11: 68%|######8 | 15/22 [00:03<00:01, 3.76it/s, loss=4.57, v_num=1]
Epoch 11: 68%|######8 | 15/22 [00:04<00:01, 3.70it/s, loss=4.57, v_num=1]
Epoch 11: 73%|#######2 | 16/22 [00:04<00:01, 3.86it/s, loss=4.57, v_num=1]
Epoch 11: 73%|#######2 | 16/22 [00:04<00:01, 3.81it/s, loss=4.57, v_num=1]
Epoch 11: 77%|#######7 | 17/22 [00:04<00:01, 3.96it/s, loss=4.57, v_num=1]
Epoch 11: 77%|#######7 | 17/22 [00:04<00:01, 3.90it/s, loss=4.57, v_num=1]
Epoch 11: 82%|########1 | 18/22 [00:04<00:00, 4.05it/s, loss=4.57, v_num=1]
Epoch 11: 82%|########1 | 18/22 [00:04<00:01, 3.99it/s, loss=4.57, v_num=1]
Epoch 11: 86%|########6 | 19/22 [00:04<00:00, 4.14it/s, loss=4.57, v_num=1]
Epoch 11: 86%|########6 | 19/22 [00:04<00:00, 4.08it/s, loss=4.57, v_num=1]
Epoch 11: 91%|######### | 20/22 [00:04<00:00, 4.22it/s, loss=4.57, v_num=1]
Epoch 11: 91%|######### | 20/22 [00:04<00:00, 4.16it/s, loss=4.56, v_num=1]
Epoch 11: 95%|#########5| 21/22 [00:04<00:00, 4.30it/s, loss=4.56, v_num=1]
Epoch 11: 95%|#########5| 21/22 [00:04<00:00, 4.24it/s, loss=4.56, v_num=1]
Epoch 11: 100%|##########| 22/22 [00:05<00:00, 4.36it/s, loss=4.56, v_num=1]
Epoch 11: 100%|##########| 22/22 [00:05<00:00, 4.31it/s, loss=4.56, v_num=1]
Epoch 11: 100%|##########| 22/22 [00:05<00:00, 4.31it/s, loss=4.56, v_num=1]
Epoch 11: 0%| | 0/22 [00:00<?, ?it/s, loss=4.56, v_num=1]
Epoch 12: 0%| | 0/22 [00:00<?, ?it/s, loss=4.56, v_num=1]
Epoch 12: 5%|4 | 1/22 [00:01<00:38, 1.85s/it, loss=4.56, v_num=1]
Epoch 12: 5%|4 | 1/22 [00:01<00:40, 1.91s/it, loss=4.56, v_num=1]
Epoch 12: 9%|9 | 2/22 [00:02<00:20, 1.01s/it, loss=4.56, v_num=1]
Epoch 12: 9%|9 | 2/22 [00:02<00:20, 1.04s/it, loss=4.56, v_num=1]
Epoch 12: 14%|#3 | 3/22 [00:02<00:13, 1.38it/s, loss=4.56, v_num=1]
Epoch 12: 14%|#3 | 3/22 [00:02<00:14, 1.34it/s, loss=4.56, v_num=1]
Epoch 12: 18%|#8 | 4/22 [00:02<00:10, 1.72it/s, loss=4.56, v_num=1]
Epoch 12: 18%|#8 | 4/22 [00:02<00:10, 1.68it/s, loss=4.56, v_num=1]
Epoch 12: 23%|##2 | 5/22 [00:02<00:08, 2.02it/s, loss=4.56, v_num=1]
Epoch 12: 23%|##2 | 5/22 [00:02<00:08, 1.97it/s, loss=4.56, v_num=1]
Epoch 12: 27%|##7 | 6/22 [00:02<00:07, 2.27it/s, loss=4.56, v_num=1]
Epoch 12: 27%|##7 | 6/22 [00:02<00:07, 2.22it/s, loss=4.56, v_num=1]
Epoch 12: 32%|###1 | 7/22 [00:02<00:06, 2.49it/s, loss=4.56, v_num=1]
Epoch 12: 32%|###1 | 7/22 [00:02<00:06, 2.44it/s, loss=4.56, v_num=1]
Epoch 12: 36%|###6 | 8/22 [00:02<00:05, 2.70it/s, loss=4.56, v_num=1]
Epoch 12: 36%|###6 | 8/22 [00:03<00:05, 2.65it/s, loss=4.56, v_num=1]
Epoch 12: 41%|#### | 9/22 [00:03<00:04, 2.88it/s, loss=4.56, v_num=1]
Epoch 12: 41%|#### | 9/22 [00:03<00:04, 2.82it/s, loss=4.56, v_num=1]
Epoch 12: 45%|####5 | 10/22 [00:03<00:03, 3.05it/s, loss=4.56, v_num=1]
Epoch 12: 45%|####5 | 10/22 [00:03<00:04, 2.99it/s, loss=4.56, v_num=1]
Epoch 12: 50%|##### | 11/22 [00:03<00:03, 3.20it/s, loss=4.56, v_num=1]
Epoch 12: 50%|##### | 11/22 [00:03<00:03, 3.15it/s, loss=4.56, v_num=1]
Epoch 12: 55%|#####4 | 12/22 [00:03<00:02, 3.34it/s, loss=4.56, v_num=1]
Epoch 12: 55%|#####4 | 12/22 [00:03<00:03, 3.29it/s, loss=4.56, v_num=1]
Epoch 12: 59%|#####9 | 13/22 [00:03<00:02, 3.47it/s, loss=4.56, v_num=1]
Epoch 12: 59%|#####9 | 13/22 [00:03<00:02, 3.42it/s, loss=4.56, v_num=1]
Epoch 12: 64%|######3 | 14/22 [00:03<00:02, 3.59it/s, loss=4.56, v_num=1]
Epoch 12: 64%|######3 | 14/22 [00:03<00:02, 3.54it/s, loss=4.56, v_num=1]
Epoch 12: 68%|######8 | 15/22 [00:04<00:01, 3.70it/s, loss=4.56, v_num=1]
Epoch 12: 68%|######8 | 15/22 [00:04<00:01, 3.65it/s, loss=4.56, v_num=1]
Epoch 12: 73%|#######2 | 16/22 [00:04<00:01, 3.81it/s, loss=4.56, v_num=1]
Epoch 12: 73%|#######2 | 16/22 [00:04<00:01, 3.75it/s, loss=4.56, v_num=1]
Epoch 12: 77%|#######7 | 17/22 [00:04<00:01, 3.90it/s, loss=4.56, v_num=1]
Epoch 12: 77%|#######7 | 17/22 [00:04<00:01, 3.84it/s, loss=4.56, v_num=1]
Epoch 12: 82%|########1 | 18/22 [00:04<00:01, 3.99it/s, loss=4.56, v_num=1]
Epoch 12: 82%|########1 | 18/22 [00:04<00:01, 3.93it/s, loss=4.56, v_num=1]
Epoch 12: 86%|########6 | 19/22 [00:04<00:00, 4.07it/s, loss=4.56, v_num=1]
Epoch 12: 86%|########6 | 19/22 [00:04<00:00, 4.02it/s, loss=4.56, v_num=1]
Epoch 12: 91%|######### | 20/22 [00:04<00:00, 4.16it/s, loss=4.56, v_num=1]
Epoch 12: 91%|######### | 20/22 [00:04<00:00, 4.10it/s, loss=4.56, v_num=1]
Epoch 12: 95%|#########5| 21/22 [00:04<00:00, 4.23it/s, loss=4.56, v_num=1]
Epoch 12: 95%|#########5| 21/22 [00:05<00:00, 4.18it/s, loss=4.56, v_num=1]
Epoch 12: 100%|##########| 22/22 [00:05<00:00, 4.30it/s, loss=4.56, v_num=1]
Epoch 12: 100%|##########| 22/22 [00:05<00:00, 4.25it/s, loss=4.55, v_num=1]
Epoch 12: 100%|##########| 22/22 [00:05<00:00, 4.25it/s, loss=4.55, v_num=1]
Epoch 12: 0%| | 0/22 [00:00<?, ?it/s, loss=4.55, v_num=1]
Epoch 13: 0%| | 0/22 [00:00<?, ?it/s, loss=4.55, v_num=1]
Epoch 13: 5%|4 | 1/22 [00:01<00:38, 1.82s/it, loss=4.55, v_num=1]
Epoch 13: 5%|4 | 1/22 [00:01<00:39, 1.88s/it, loss=4.55, v_num=1]
Epoch 13: 9%|9 | 2/22 [00:01<00:19, 1.00it/s, loss=4.55, v_num=1]
Epoch 13: 9%|9 | 2/22 [00:02<00:20, 1.03s/it, loss=4.55, v_num=1]
Epoch 13: 14%|#3 | 3/22 [00:02<00:13, 1.39it/s, loss=4.55, v_num=1]
Epoch 13: 14%|#3 | 3/22 [00:02<00:14, 1.35it/s, loss=4.55, v_num=1]
Epoch 13: 18%|#8 | 4/22 [00:02<00:10, 1.72it/s, loss=4.55, v_num=1]
Epoch 13: 18%|#8 | 4/22 [00:02<00:10, 1.68it/s, loss=4.55, v_num=1]
Epoch 13: 23%|##2 | 5/22 [00:02<00:08, 2.02it/s, loss=4.55, v_num=1]
Epoch 13: 23%|##2 | 5/22 [00:02<00:08, 1.97it/s, loss=4.55, v_num=1]
Epoch 13: 27%|##7 | 6/22 [00:02<00:07, 2.27it/s, loss=4.55, v_num=1]
Epoch 13: 27%|##7 | 6/22 [00:02<00:07, 2.23it/s, loss=4.55, v_num=1]
Epoch 13: 32%|###1 | 7/22 [00:02<00:05, 2.50it/s, loss=4.55, v_num=1]
Epoch 13: 32%|###1 | 7/22 [00:02<00:06, 2.45it/s, loss=4.55, v_num=1]
Epoch 13: 36%|###6 | 8/22 [00:02<00:05, 2.71it/s, loss=4.55, v_num=1]
Epoch 13: 36%|###6 | 8/22 [00:03<00:05, 2.65it/s, loss=4.55, v_num=1]
Epoch 13: 41%|#### | 9/22 [00:03<00:04, 2.89it/s, loss=4.55, v_num=1]
Epoch 13: 41%|#### | 9/22 [00:03<00:04, 2.84it/s, loss=4.55, v_num=1]
Epoch 13: 45%|####5 | 10/22 [00:03<00:03, 3.05it/s, loss=4.55, v_num=1]
Epoch 13: 45%|####5 | 10/22 [00:03<00:04, 3.00it/s, loss=4.55, v_num=1]
Epoch 13: 50%|##### | 11/22 [00:03<00:03, 3.21it/s, loss=4.55, v_num=1]
Epoch 13: 50%|##### | 11/22 [00:03<00:03, 3.15it/s, loss=4.55, v_num=1]
Epoch 13: 55%|#####4 | 12/22 [00:03<00:02, 3.34it/s, loss=4.55, v_num=1]
Epoch 13: 55%|#####4 | 12/22 [00:03<00:03, 3.29it/s, loss=4.54, v_num=1]
Epoch 13: 59%|#####9 | 13/22 [00:03<00:02, 3.47it/s, loss=4.54, v_num=1]
Epoch 13: 59%|#####9 | 13/22 [00:03<00:02, 3.42it/s, loss=4.54, v_num=1]
Epoch 13: 64%|######3 | 14/22 [00:03<00:02, 3.60it/s, loss=4.54, v_num=1]
Epoch 13: 64%|######3 | 14/22 [00:03<00:02, 3.54it/s, loss=4.54, v_num=1]
Epoch 13: 68%|######8 | 15/22 [00:04<00:01, 3.71it/s, loss=4.54, v_num=1]
Epoch 13: 68%|######8 | 15/22 [00:04<00:01, 3.65it/s, loss=4.54, v_num=1]
Epoch 13: 73%|#######2 | 16/22 [00:04<00:01, 3.82it/s, loss=4.54, v_num=1]
Epoch 13: 73%|#######2 | 16/22 [00:04<00:01, 3.76it/s, loss=4.54, v_num=1]
Epoch 13: 77%|#######7 | 17/22 [00:04<00:01, 3.91it/s, loss=4.54, v_num=1]
Epoch 13: 77%|#######7 | 17/22 [00:04<00:01, 3.85it/s, loss=4.54, v_num=1]
Epoch 13: 82%|########1 | 18/22 [00:04<00:01, 4.00it/s, loss=4.54, v_num=1]
Epoch 13: 82%|########1 | 18/22 [00:04<00:01, 3.94it/s, loss=4.54, v_num=1]
Epoch 13: 86%|########6 | 19/22 [00:04<00:00, 4.08it/s, loss=4.54, v_num=1]
Epoch 13: 86%|########6 | 19/22 [00:04<00:00, 4.02it/s, loss=4.55, v_num=1]
Epoch 13: 91%|######### | 20/22 [00:04<00:00, 4.16it/s, loss=4.55, v_num=1]
Epoch 13: 91%|######### | 20/22 [00:04<00:00, 4.11it/s, loss=4.55, v_num=1]
Epoch 13: 95%|#########5| 21/22 [00:04<00:00, 4.23it/s, loss=4.55, v_num=1]
Epoch 13: 95%|#########5| 21/22 [00:05<00:00, 4.18it/s, loss=4.55, v_num=1]
Epoch 13: 100%|##########| 22/22 [00:05<00:00, 4.30it/s, loss=4.55, v_num=1]
Epoch 13: 100%|##########| 22/22 [00:05<00:00, 4.25it/s, loss=4.54, v_num=1]
Epoch 13: 100%|##########| 22/22 [00:05<00:00, 4.25it/s, loss=4.54, v_num=1]
Epoch 13: 0%| | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 14: 0%| | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 14: 5%|4 | 1/22 [00:01<00:39, 1.88s/it, loss=4.54, v_num=1]
Epoch 14: 5%|4 | 1/22 [00:01<00:40, 1.94s/it, loss=4.54, v_num=1]
Epoch 14: 9%|9 | 2/22 [00:02<00:20, 1.02s/it, loss=4.54, v_num=1]
Epoch 14: 9%|9 | 2/22 [00:02<00:21, 1.05s/it, loss=4.54, v_num=1]
Epoch 14: 14%|#3 | 3/22 [00:02<00:14, 1.35it/s, loss=4.54, v_num=1]
Epoch 14: 14%|#3 | 3/22 [00:02<00:14, 1.32it/s, loss=4.54, v_num=1]
Epoch 14: 18%|#8 | 4/22 [00:02<00:10, 1.68it/s, loss=4.54, v_num=1]
Epoch 14: 18%|#8 | 4/22 [00:02<00:10, 1.64it/s, loss=4.54, v_num=1]
Epoch 14: 23%|##2 | 5/22 [00:02<00:08, 1.97it/s, loss=4.54, v_num=1]
Epoch 14: 23%|##2 | 5/22 [00:02<00:08, 1.92it/s, loss=4.54, v_num=1]
Epoch 14: 27%|##7 | 6/22 [00:02<00:07, 2.22it/s, loss=4.54, v_num=1]
Epoch 14: 27%|##7 | 6/22 [00:02<00:07, 2.17it/s, loss=4.54, v_num=1]
Epoch 14: 32%|###1 | 7/22 [00:02<00:06, 2.40it/s, loss=4.54, v_num=1]
Epoch 14: 32%|###1 | 7/22 [00:02<00:06, 2.35it/s, loss=4.54, v_num=1]
Epoch 14: 36%|###6 | 8/22 [00:03<00:05, 2.59it/s, loss=4.54, v_num=1]
Epoch 14: 36%|###6 | 8/22 [00:03<00:05, 2.54it/s, loss=4.54, v_num=1]
Epoch 14: 41%|#### | 9/22 [00:03<00:04, 2.77it/s, loss=4.54, v_num=1]
Epoch 14: 41%|#### | 9/22 [00:03<00:04, 2.72it/s, loss=4.54, v_num=1]
Epoch 14: 45%|####5 | 10/22 [00:03<00:04, 2.94it/s, loss=4.54, v_num=1]
Epoch 14: 45%|####5 | 10/22 [00:03<00:04, 2.89it/s, loss=4.54, v_num=1]
Epoch 14: 50%|##### | 11/22 [00:03<00:03, 3.09it/s, loss=4.54, v_num=1]
Epoch 14: 50%|##### | 11/22 [00:03<00:03, 3.04it/s, loss=4.54, v_num=1]
Epoch 14: 55%|#####4 | 12/22 [00:03<00:03, 3.22it/s, loss=4.54, v_num=1]
Epoch 14: 55%|#####4 | 12/22 [00:03<00:03, 3.17it/s, loss=4.54, v_num=1]
Epoch 14: 59%|#####9 | 13/22 [00:03<00:02, 3.32it/s, loss=4.54, v_num=1]
Epoch 14: 59%|#####9 | 13/22 [00:03<00:02, 3.27it/s, loss=4.54, v_num=1]
Epoch 14: 64%|######3 | 14/22 [00:04<00:02, 3.44it/s, loss=4.54, v_num=1]
Epoch 14: 64%|######3 | 14/22 [00:04<00:02, 3.39it/s, loss=4.54, v_num=1]
Epoch 14: 68%|######8 | 15/22 [00:04<00:01, 3.56it/s, loss=4.54, v_num=1]
Epoch 14: 68%|######8 | 15/22 [00:04<00:01, 3.50it/s, loss=4.54, v_num=1]
Epoch 14: 73%|#######2 | 16/22 [00:04<00:01, 3.66it/s, loss=4.54, v_num=1]
Epoch 14: 73%|#######2 | 16/22 [00:04<00:01, 3.61it/s, loss=4.54, v_num=1]
Epoch 14: 77%|#######7 | 17/22 [00:04<00:01, 3.75it/s, loss=4.54, v_num=1]
Epoch 14: 77%|#######7 | 17/22 [00:04<00:01, 3.70it/s, loss=4.54, v_num=1]
Epoch 14: 82%|########1 | 18/22 [00:04<00:01, 3.84it/s, loss=4.54, v_num=1]
Epoch 14: 82%|########1 | 18/22 [00:04<00:01, 3.79it/s, loss=4.54, v_num=1]
Epoch 14: 86%|########6 | 19/22 [00:04<00:00, 3.92it/s, loss=4.54, v_num=1]
Epoch 14: 86%|########6 | 19/22 [00:04<00:00, 3.87it/s, loss=4.54, v_num=1]
Epoch 14: 91%|######### | 20/22 [00:04<00:00, 4.00it/s, loss=4.54, v_num=1]
Epoch 14: 91%|######### | 20/22 [00:05<00:00, 3.96it/s, loss=4.54, v_num=1]
Epoch 14: 95%|#########5| 21/22 [00:05<00:00, 4.08it/s, loss=4.54, v_num=1]
Epoch 14: 95%|#########5| 21/22 [00:05<00:00, 4.03it/s, loss=4.54, v_num=1]
Epoch 14: 100%|##########| 22/22 [00:05<00:00, 4.15it/s, loss=4.54, v_num=1]
Epoch 14: 100%|##########| 22/22 [00:05<00:00, 4.10it/s, loss=4.54, v_num=1]
Epoch 14: 100%|##########| 22/22 [00:05<00:00, 4.10it/s, loss=4.54, v_num=1]
Epoch 14: 0%| | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 15: 0%| | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 15: 5%|4 | 1/22 [00:01<00:38, 1.85s/it, loss=4.54, v_num=1]
Epoch 15: 5%|4 | 1/22 [00:01<00:40, 1.91s/it, loss=4.54, v_num=1]
Epoch 15: 9%|9 | 2/22 [00:02<00:20, 1.00s/it, loss=4.54, v_num=1]
Epoch 15: 9%|9 | 2/22 [00:02<00:20, 1.03s/it, loss=4.54, v_num=1]
Epoch 15: 14%|#3 | 3/22 [00:02<00:13, 1.39it/s, loss=4.54, v_num=1]
Epoch 15: 14%|#3 | 3/22 [00:02<00:14, 1.35it/s, loss=4.54, v_num=1]
Epoch 15: 18%|#8 | 4/22 [00:02<00:10, 1.73it/s, loss=4.54, v_num=1]
Epoch 15: 18%|#8 | 4/22 [00:02<00:10, 1.68it/s, loss=4.54, v_num=1]
Epoch 15: 23%|##2 | 5/22 [00:02<00:08, 2.02it/s, loss=4.54, v_num=1]
Epoch 15: 23%|##2 | 5/22 [00:02<00:08, 1.97it/s, loss=4.54, v_num=1]
Epoch 15: 27%|##7 | 6/22 [00:02<00:07, 2.27it/s, loss=4.54, v_num=1]
Epoch 15: 27%|##7 | 6/22 [00:02<00:07, 2.22it/s, loss=4.54, v_num=1]
Epoch 15: 32%|###1 | 7/22 [00:02<00:06, 2.50it/s, loss=4.54, v_num=1]
Epoch 15: 32%|###1 | 7/22 [00:02<00:06, 2.44it/s, loss=4.54, v_num=1]
Epoch 15: 36%|###6 | 8/22 [00:02<00:05, 2.70it/s, loss=4.54, v_num=1]
Epoch 15: 36%|###6 | 8/22 [00:03<00:05, 2.65it/s, loss=4.54, v_num=1]
Epoch 15: 41%|#### | 9/22 [00:03<00:04, 2.88it/s, loss=4.54, v_num=1]
Epoch 15: 41%|#### | 9/22 [00:03<00:04, 2.82it/s, loss=4.54, v_num=1]
Epoch 15: 45%|####5 | 10/22 [00:03<00:03, 3.05it/s, loss=4.54, v_num=1]
Epoch 15: 45%|####5 | 10/22 [00:03<00:04, 2.99it/s, loss=4.54, v_num=1]
Epoch 15: 50%|##### | 11/22 [00:03<00:03, 3.20it/s, loss=4.54, v_num=1]
Epoch 15: 50%|##### | 11/22 [00:03<00:03, 3.15it/s, loss=4.54, v_num=1]
Epoch 15: 55%|#####4 | 12/22 [00:03<00:02, 3.34it/s, loss=4.54, v_num=1]
Epoch 15: 55%|#####4 | 12/22 [00:03<00:03, 3.28it/s, loss=4.54, v_num=1]
Epoch 15: 59%|#####9 | 13/22 [00:03<00:02, 3.47it/s, loss=4.54, v_num=1]
Epoch 15: 59%|#####9 | 13/22 [00:03<00:02, 3.41it/s, loss=4.54, v_num=1]
Epoch 15: 64%|######3 | 14/22 [00:03<00:02, 3.59it/s, loss=4.54, v_num=1]
Epoch 15: 64%|######3 | 14/22 [00:03<00:02, 3.53it/s, loss=4.54, v_num=1]
Epoch 15: 68%|######8 | 15/22 [00:04<00:01, 3.70it/s, loss=4.54, v_num=1]
Epoch 15: 68%|######8 | 15/22 [00:04<00:01, 3.65it/s, loss=4.54, v_num=1]
Epoch 15: 73%|#######2 | 16/22 [00:04<00:01, 3.80it/s, loss=4.54, v_num=1]
Epoch 15: 73%|#######2 | 16/22 [00:04<00:01, 3.75it/s, loss=4.54, v_num=1]
Epoch 15: 77%|#######7 | 17/22 [00:04<00:01, 3.90it/s, loss=4.54, v_num=1]
Epoch 15: 77%|#######7 | 17/22 [00:04<00:01, 3.84it/s, loss=4.54, v_num=1]
Epoch 15: 82%|########1 | 18/22 [00:04<00:01, 3.98it/s, loss=4.54, v_num=1]
Epoch 15: 82%|########1 | 18/22 [00:04<00:01, 3.93it/s, loss=4.54, v_num=1]
Epoch 15: 86%|########6 | 19/22 [00:04<00:00, 4.06it/s, loss=4.54, v_num=1]
Epoch 15: 86%|########6 | 19/22 [00:04<00:00, 4.01it/s, loss=4.54, v_num=1]
Epoch 15: 91%|######### | 20/22 [00:04<00:00, 4.14it/s, loss=4.54, v_num=1]
Epoch 15: 91%|######### | 20/22 [00:04<00:00, 4.08it/s, loss=4.54, v_num=1]
Epoch 15: 95%|#########5| 21/22 [00:04<00:00, 4.20it/s, loss=4.54, v_num=1]
Epoch 15: 95%|#########5| 21/22 [00:05<00:00, 4.16it/s, loss=4.54, v_num=1]
Epoch 15: 100%|##########| 22/22 [00:05<00:00, 4.28it/s, loss=4.54, v_num=1]
Epoch 15: 100%|##########| 22/22 [00:05<00:00, 4.23it/s, loss=4.54, v_num=1]
Epoch 15: 100%|##########| 22/22 [00:05<00:00, 4.23it/s, loss=4.54, v_num=1]
Epoch 15: 0%| | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 16: 0%| | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 16: 5%|4 | 1/22 [00:02<00:43, 2.06s/it, loss=4.54, v_num=1]
Epoch 16: 5%|4 | 1/22 [00:02<00:44, 2.12s/it, loss=4.54, v_num=1]
Epoch 16: 9%|9 | 2/22 [00:02<00:22, 1.12s/it, loss=4.54, v_num=1]
Epoch 16: 9%|9 | 2/22 [00:02<00:22, 1.14s/it, loss=4.54, v_num=1]
Epoch 16: 14%|#3 | 3/22 [00:02<00:15, 1.25it/s, loss=4.54, v_num=1]
Epoch 16: 14%|#3 | 3/22 [00:02<00:15, 1.22it/s, loss=4.54, v_num=1]
Epoch 16: 18%|#8 | 4/22 [00:02<00:11, 1.56it/s, loss=4.54, v_num=1]
Epoch 16: 18%|#8 | 4/22 [00:02<00:11, 1.52it/s, loss=4.54, v_num=1]
Epoch 16: 23%|##2 | 5/22 [00:02<00:09, 1.82it/s, loss=4.54, v_num=1]
Epoch 16: 23%|##2 | 5/22 [00:02<00:09, 1.79it/s, loss=4.54, v_num=1]
Epoch 16: 27%|##7 | 6/22 [00:02<00:07, 2.07it/s, loss=4.54, v_num=1]
Epoch 16: 27%|##7 | 6/22 [00:02<00:07, 2.02it/s, loss=4.54, v_num=1]
Epoch 16: 32%|###1 | 7/22 [00:03<00:06, 2.27it/s, loss=4.54, v_num=1]
Epoch 16: 32%|###1 | 7/22 [00:03<00:06, 2.23it/s, loss=4.54, v_num=1]
Epoch 16: 36%|###6 | 8/22 [00:03<00:05, 2.46it/s, loss=4.54, v_num=1]
Epoch 16: 36%|###6 | 8/22 [00:03<00:05, 2.42it/s, loss=4.54, v_num=1]
Epoch 16: 41%|#### | 9/22 [00:03<00:04, 2.63it/s, loss=4.54, v_num=1]
Epoch 16: 41%|#### | 9/22 [00:03<00:05, 2.59it/s, loss=4.54, v_num=1]
Epoch 16: 45%|####5 | 10/22 [00:03<00:04, 2.79it/s, loss=4.54, v_num=1]
Epoch 16: 45%|####5 | 10/22 [00:03<00:04, 2.75it/s, loss=4.54, v_num=1]
Epoch 16: 50%|##### | 11/22 [00:03<00:03, 2.94it/s, loss=4.54, v_num=1]
Epoch 16: 50%|##### | 11/22 [00:03<00:03, 2.89it/s, loss=4.54, v_num=1]
Epoch 16: 55%|#####4 | 12/22 [00:03<00:03, 3.07it/s, loss=4.54, v_num=1]
Epoch 16: 55%|#####4 | 12/22 [00:03<00:03, 3.03it/s, loss=4.54, v_num=1]
Epoch 16: 59%|#####9 | 13/22 [00:04<00:02, 3.19it/s, loss=4.54, v_num=1]
Epoch 16: 59%|#####9 | 13/22 [00:04<00:02, 3.15it/s, loss=4.54, v_num=1]
Epoch 16: 64%|######3 | 14/22 [00:04<00:02, 3.30it/s, loss=4.54, v_num=1]
Epoch 16: 64%|######3 | 14/22 [00:04<00:02, 3.26it/s, loss=4.54, v_num=1]
Epoch 16: 68%|######8 | 15/22 [00:04<00:02, 3.40it/s, loss=4.54, v_num=1]
Epoch 16: 68%|######8 | 15/22 [00:04<00:02, 3.36it/s, loss=4.54, v_num=1]
Epoch 16: 73%|#######2 | 16/22 [00:04<00:01, 3.51it/s, loss=4.54, v_num=1]
Epoch 16: 73%|#######2 | 16/22 [00:04<00:01, 3.46it/s, loss=4.54, v_num=1]
Epoch 16: 77%|#######7 | 17/22 [00:04<00:01, 3.60it/s, loss=4.54, v_num=1]
Epoch 16: 77%|#######7 | 17/22 [00:04<00:01, 3.56it/s, loss=4.54, v_num=1]
Epoch 16: 82%|########1 | 18/22 [00:04<00:01, 3.69it/s, loss=4.54, v_num=1]
Epoch 16: 82%|########1 | 18/22 [00:04<00:01, 3.64it/s, loss=4.54, v_num=1]
Epoch 16: 86%|########6 | 19/22 [00:05<00:00, 3.76it/s, loss=4.54, v_num=1]
Epoch 16: 86%|########6 | 19/22 [00:05<00:00, 3.72it/s, loss=4.54, v_num=1]
Epoch 16: 91%|######### | 20/22 [00:05<00:00, 3.84it/s, loss=4.54, v_num=1]
Epoch 16: 91%|######### | 20/22 [00:05<00:00, 3.80it/s, loss=4.54, v_num=1]
Epoch 16: 95%|#########5| 21/22 [00:05<00:00, 3.91it/s, loss=4.54, v_num=1]
Epoch 16: 95%|#########5| 21/22 [00:05<00:00, 3.87it/s, loss=4.54, v_num=1]
Epoch 16: 100%|##########| 22/22 [00:05<00:00, 3.98it/s, loss=4.54, v_num=1]
Epoch 16: 100%|##########| 22/22 [00:05<00:00, 3.94it/s, loss=4.54, v_num=1]
Epoch 16: 100%|##########| 22/22 [00:05<00:00, 3.94it/s, loss=4.54, v_num=1]
Epoch 16: 0%| | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 17: 0%| | 0/22 [00:00<?, ?it/s, loss=4.54, v_num=1]
Epoch 17: 5%|4 | 1/22 [00:02<00:47, 2.25s/it, loss=4.54, v_num=1]
Epoch 17: 5%|4 | 1/22 [00:02<00:48, 2.31s/it, loss=4.54, v_num=1]
Epoch 17: 9%|9 | 2/22 [00:02<00:24, 1.22s/it, loss=4.54, v_num=1]
Epoch 17: 9%|9 | 2/22 [00:02<00:24, 1.25s/it, loss=4.54, v_num=1]
Epoch 17: 14%|#3 | 3/22 [00:02<00:16, 1.15it/s, loss=4.54, v_num=1]
Epoch 17: 14%|#3 | 3/22 [00:02<00:16, 1.13it/s, loss=4.54, v_num=1]
Epoch 17: 18%|#8 | 4/22 [00:02<00:12, 1.45it/s, loss=4.54, v_num=1]
Epoch 17: 18%|#8 | 4/22 [00:02<00:12, 1.42it/s, loss=4.54, v_num=1]
Epoch 17: 23%|##2 | 5/22 [00:02<00:09, 1.70it/s, loss=4.54, v_num=1]
Epoch 17: 23%|##2 | 5/22 [00:02<00:10, 1.67it/s, loss=4.54, v_num=1]
Epoch 17: 27%|##7 | 6/22 [00:03<00:08, 1.93it/s, loss=4.54, v_num=1]
Epoch 17: 27%|##7 | 6/22 [00:03<00:08, 1.89it/s, loss=4.54, v_num=1]
Epoch 17: 32%|###1 | 7/22 [00:03<00:07, 2.13it/s, loss=4.54, v_num=1]
Epoch 17: 32%|###1 | 7/22 [00:03<00:07, 2.09it/s, loss=4.54, v_num=1]
Epoch 17: 36%|###6 | 8/22 [00:03<00:06, 2.32it/s, loss=4.54, v_num=1]
Epoch 17: 36%|###6 | 8/22 [00:03<00:06, 2.28it/s, loss=4.54, v_num=1]
Epoch 17: 41%|#### | 9/22 [00:03<00:05, 2.48it/s, loss=4.54, v_num=1]
Epoch 17: 41%|#### | 9/22 [00:03<00:05, 2.44it/s, loss=4.54, v_num=1]
Epoch 17: 45%|####5 | 10/22 [00:03<00:04, 2.64it/s, loss=4.54, v_num=1]
Epoch 17: 45%|####5 | 10/22 [00:03<00:04, 2.60it/s, loss=4.54, v_num=1]
Epoch 17: 50%|##### | 11/22 [00:03<00:03, 2.79it/s, loss=4.54, v_num=1]
Epoch 17: 50%|##### | 11/22 [00:04<00:04, 2.74it/s, loss=4.54, v_num=1]
Epoch 17: 55%|#####4 | 12/22 [00:04<00:03, 2.92it/s, loss=4.54, v_num=1]
Epoch 17: 55%|#####4 | 12/22 [00:04<00:03, 2.88it/s, loss=4.54, v_num=1]
Epoch 17: 59%|#####9 | 13/22 [00:04<00:02, 3.05it/s, loss=4.54, v_num=1]
Epoch 17: 59%|#####9 | 13/22 [00:04<00:02, 3.00it/s, loss=4.54, v_num=1]
Epoch 17: 64%|######3 | 14/22 [00:04<00:02, 3.16it/s, loss=4.54, v_num=1]
Epoch 17: 64%|######3 | 14/22 [00:04<00:02, 3.12it/s, loss=4.54, v_num=1]
Epoch 17: 68%|######8 | 15/22 [00:04<00:02, 3.26it/s, loss=4.54, v_num=1]
Epoch 17: 68%|######8 | 15/22 [00:04<00:02, 3.22it/s, loss=4.54, v_num=1]
Epoch 17: 73%|#######2 | 16/22 [00:04<00:01, 3.36it/s, loss=4.54, v_num=1]
Epoch 17: 73%|#######2 | 16/22 [00:04<00:01, 3.31it/s, loss=4.54, v_num=1]
Epoch 17: 77%|#######7 | 17/22 [00:04<00:01, 3.45it/s, loss=4.54, v_num=1]
Epoch 17: 77%|#######7 | 17/22 [00:04<00:01, 3.41it/s, loss=4.53, v_num=1]
Epoch 17: 82%|########1 | 18/22 [00:05<00:01, 3.54it/s, loss=4.53, v_num=1]
Epoch 17: 82%|########1 | 18/22 [00:05<00:01, 3.50it/s, loss=4.53, v_num=1]
Epoch 17: 86%|########6 | 19/22 [00:05<00:00, 3.62it/s, loss=4.53, v_num=1]
Epoch 17: 86%|########6 | 19/22 [00:05<00:00, 3.58it/s, loss=4.53, v_num=1]
Epoch 17: 91%|######### | 20/22 [00:05<00:00, 3.70it/s, loss=4.53, v_num=1]
Epoch 17: 91%|######### | 20/22 [00:05<00:00, 3.66it/s, loss=4.53, v_num=1]
Epoch 17: 95%|#########5| 21/22 [00:05<00:00, 3.77it/s, loss=4.53, v_num=1]
Epoch 17: 95%|#########5| 21/22 [00:05<00:00, 3.73it/s, loss=4.53, v_num=1]
Epoch 17: 100%|##########| 22/22 [00:05<00:00, 3.84it/s, loss=4.53, v_num=1]
Epoch 17: 100%|##########| 22/22 [00:05<00:00, 3.80it/s, loss=4.53, v_num=1]
Epoch 17: 100%|##########| 22/22 [00:05<00:00, 3.80it/s, loss=4.53, v_num=1]
Epoch 17: 0%| | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 18: 0%| | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 18: 5%|4 | 1/22 [00:02<00:47, 2.28s/it, loss=4.53, v_num=1]
Epoch 18: 5%|4 | 1/22 [00:02<00:49, 2.33s/it, loss=4.53, v_num=1]
Epoch 18: 9%|9 | 2/22 [00:02<00:24, 1.22s/it, loss=4.53, v_num=1]
Epoch 18: 9%|9 | 2/22 [00:02<00:25, 1.25s/it, loss=4.53, v_num=1]
Epoch 18: 14%|#3 | 3/22 [00:02<00:16, 1.15it/s, loss=4.53, v_num=1]
Epoch 18: 14%|#3 | 3/22 [00:02<00:16, 1.12it/s, loss=4.53, v_num=1]
Epoch 18: 18%|#8 | 4/22 [00:02<00:12, 1.44it/s, loss=4.53, v_num=1]
Epoch 18: 18%|#8 | 4/22 [00:02<00:12, 1.41it/s, loss=4.53, v_num=1]
Epoch 18: 23%|##2 | 5/22 [00:02<00:09, 1.70it/s, loss=4.53, v_num=1]
Epoch 18: 23%|##2 | 5/22 [00:02<00:10, 1.67it/s, loss=4.53, v_num=1]
Epoch 18: 27%|##7 | 6/22 [00:03<00:08, 1.94it/s, loss=4.53, v_num=1]
Epoch 18: 27%|##7 | 6/22 [00:03<00:08, 1.90it/s, loss=4.53, v_num=1]
Epoch 18: 32%|###1 | 7/22 [00:03<00:07, 2.14it/s, loss=4.53, v_num=1]
Epoch 18: 32%|###1 | 7/22 [00:03<00:07, 2.10it/s, loss=4.52, v_num=1]
Epoch 18: 36%|###6 | 8/22 [00:03<00:06, 2.32it/s, loss=4.52, v_num=1]
Epoch 18: 36%|###6 | 8/22 [00:03<00:06, 2.28it/s, loss=4.52, v_num=1]
Epoch 18: 41%|#### | 9/22 [00:03<00:05, 2.48it/s, loss=4.52, v_num=1]
Epoch 18: 41%|#### | 9/22 [00:03<00:05, 2.44it/s, loss=4.53, v_num=1]
Epoch 18: 45%|####5 | 10/22 [00:03<00:04, 2.63it/s, loss=4.53, v_num=1]
Epoch 18: 45%|####5 | 10/22 [00:03<00:04, 2.59it/s, loss=4.53, v_num=1]
Epoch 18: 50%|##### | 11/22 [00:03<00:03, 2.78it/s, loss=4.53, v_num=1]
Epoch 18: 50%|##### | 11/22 [00:04<00:04, 2.74it/s, loss=4.53, v_num=1]
Epoch 18: 55%|#####4 | 12/22 [00:04<00:03, 2.91it/s, loss=4.53, v_num=1]
Epoch 18: 55%|#####4 | 12/22 [00:04<00:03, 2.86it/s, loss=4.53, v_num=1]
Epoch 18: 59%|#####9 | 13/22 [00:04<00:02, 3.03it/s, loss=4.53, v_num=1]
Epoch 18: 59%|#####9 | 13/22 [00:04<00:03, 2.98it/s, loss=4.53, v_num=1]
Epoch 18: 64%|######3 | 14/22 [00:04<00:02, 3.14it/s, loss=4.53, v_num=1]
Epoch 18: 64%|######3 | 14/22 [00:04<00:02, 3.10it/s, loss=4.53, v_num=1]
Epoch 18: 68%|######8 | 15/22 [00:04<00:02, 3.25it/s, loss=4.53, v_num=1]
Epoch 18: 68%|######8 | 15/22 [00:04<00:02, 3.20it/s, loss=4.53, v_num=1]
Epoch 18: 73%|#######2 | 16/22 [00:04<00:01, 3.35it/s, loss=4.53, v_num=1]
Epoch 18: 73%|#######2 | 16/22 [00:04<00:01, 3.31it/s, loss=4.53, v_num=1]
Epoch 18: 77%|#######7 | 17/22 [00:04<00:01, 3.45it/s, loss=4.53, v_num=1]
Epoch 18: 77%|#######7 | 17/22 [00:04<00:01, 3.40it/s, loss=4.53, v_num=1]
Epoch 18: 82%|########1 | 18/22 [00:05<00:01, 3.54it/s, loss=4.53, v_num=1]
Epoch 18: 82%|########1 | 18/22 [00:05<00:01, 3.49it/s, loss=4.53, v_num=1]
Epoch 18: 86%|########6 | 19/22 [00:05<00:00, 3.62it/s, loss=4.53, v_num=1]
Epoch 18: 86%|########6 | 19/22 [00:05<00:00, 3.58it/s, loss=4.53, v_num=1]
Epoch 18: 91%|######### | 20/22 [00:05<00:00, 3.70it/s, loss=4.53, v_num=1]
Epoch 18: 91%|######### | 20/22 [00:05<00:00, 3.65it/s, loss=4.53, v_num=1]
Epoch 18: 95%|#########5| 21/22 [00:05<00:00, 3.77it/s, loss=4.53, v_num=1]
Epoch 18: 95%|#########5| 21/22 [00:05<00:00, 3.73it/s, loss=4.53, v_num=1]
Epoch 18: 100%|##########| 22/22 [00:05<00:00, 3.83it/s, loss=4.53, v_num=1]
Epoch 18: 100%|##########| 22/22 [00:05<00:00, 3.80it/s, loss=4.53, v_num=1]
Epoch 18: 100%|##########| 22/22 [00:05<00:00, 3.79it/s, loss=4.53, v_num=1]
Epoch 18: 0%| | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 19: 0%| | 0/22 [00:00<?, ?it/s, loss=4.53, v_num=1]
Epoch 19: 5%|4 | 1/22 [00:02<00:45, 2.18s/it, loss=4.53, v_num=1]
Epoch 19: 5%|4 | 1/22 [00:02<00:46, 2.24s/it, loss=4.53, v_num=1]
Epoch 19: 9%|9 | 2/22 [00:02<00:23, 1.18s/it, loss=4.53, v_num=1]
Epoch 19: 9%|9 | 2/22 [00:02<00:24, 1.21s/it, loss=4.53, v_num=1]
Epoch 19: 14%|#3 | 3/22 [00:02<00:16, 1.19it/s, loss=4.53, v_num=1]
Epoch 19: 14%|#3 | 3/22 [00:02<00:16, 1.16it/s, loss=4.53, v_num=1]
Epoch 19: 18%|#8 | 4/22 [00:02<00:12, 1.48it/s, loss=4.53, v_num=1]
Epoch 19: 18%|#8 | 4/22 [00:02<00:12, 1.45it/s, loss=4.53, v_num=1]
Epoch 19: 23%|##2 | 5/22 [00:02<00:09, 1.75it/s, loss=4.53, v_num=1]
Epoch 19: 23%|##2 | 5/22 [00:02<00:09, 1.71it/s, loss=4.54, v_num=1]
Epoch 19: 27%|##7 | 6/22 [00:03<00:08, 1.98it/s, loss=4.54, v_num=1]
Epoch 19: 27%|##7 | 6/22 [00:03<00:08, 1.94it/s, loss=4.54, v_num=1]
Epoch 19: 32%|###1 | 7/22 [00:03<00:06, 2.19it/s, loss=4.54, v_num=1]
Epoch 19: 32%|###1 | 7/22 [00:03<00:06, 2.15it/s, loss=4.54, v_num=1]
Epoch 19: 36%|###6 | 8/22 [00:03<00:06, 2.31it/s, loss=4.54, v_num=1]
Epoch 19: 36%|###6 | 8/22 [00:03<00:06, 2.27it/s, loss=4.54, v_num=1]
Epoch 19: 41%|#### | 9/22 [00:03<00:05, 2.47it/s, loss=4.54, v_num=1]
Epoch 19: 41%|#### | 9/22 [00:03<00:05, 2.43it/s, loss=4.54, v_num=1]
Epoch 19: 45%|####5 | 10/22 [00:03<00:04, 2.62it/s, loss=4.54, v_num=1]
Epoch 19: 45%|####5 | 10/22 [00:03<00:04, 2.58it/s, loss=4.54, v_num=1]
Epoch 19: 50%|##### | 11/22 [00:03<00:03, 2.77it/s, loss=4.54, v_num=1]
Epoch 19: 50%|##### | 11/22 [00:04<00:04, 2.73it/s, loss=4.53, v_num=1]
Epoch 19: 55%|#####4 | 12/22 [00:04<00:03, 2.90it/s, loss=4.53, v_num=1]
Epoch 19: 55%|#####4 | 12/22 [00:04<00:03, 2.85it/s, loss=4.53, v_num=1]
Epoch 19: 59%|#####9 | 13/22 [00:04<00:02, 3.02it/s, loss=4.53, v_num=1]
Epoch 19: 59%|#####9 | 13/22 [00:04<00:03, 2.97it/s, loss=4.53, v_num=1]
Epoch 19: 64%|######3 | 14/22 [00:04<00:02, 3.13it/s, loss=4.53, v_num=1]
Epoch 19: 64%|######3 | 14/22 [00:04<00:02, 3.08it/s, loss=4.53, v_num=1]
Epoch 19: 68%|######8 | 15/22 [00:04<00:02, 3.23it/s, loss=4.53, v_num=1]
Epoch 19: 68%|######8 | 15/22 [00:04<00:02, 3.19it/s, loss=4.53, v_num=1]
Epoch 19: 73%|#######2 | 16/22 [00:04<00:01, 3.32it/s, loss=4.53, v_num=1]
Epoch 19: 73%|#######2 | 16/22 [00:04<00:01, 3.28it/s, loss=4.53, v_num=1]
Epoch 19: 77%|#######7 | 17/22 [00:04<00:01, 3.42it/s, loss=4.53, v_num=1]
Epoch 19: 77%|#######7 | 17/22 [00:05<00:01, 3.38it/s, loss=4.53, v_num=1]
Epoch 19: 82%|########1 | 18/22 [00:05<00:01, 3.51it/s, loss=4.53, v_num=1]
Epoch 19: 82%|########1 | 18/22 [00:05<00:01, 3.46it/s, loss=4.53, v_num=1]
Epoch 19: 86%|########6 | 19/22 [00:05<00:00, 3.59it/s, loss=4.53, v_num=1]
Epoch 19: 86%|########6 | 19/22 [00:05<00:00, 3.55it/s, loss=4.54, v_num=1]
Epoch 19: 91%|######### | 20/22 [00:05<00:00, 3.67it/s, loss=4.54, v_num=1]
Epoch 19: 91%|######### | 20/22 [00:05<00:00, 3.63it/s, loss=4.54, v_num=1]
Epoch 19: 95%|#########5| 21/22 [00:05<00:00, 3.74it/s, loss=4.54, v_num=1]
Epoch 19: 95%|#########5| 21/22 [00:05<00:00, 3.70it/s, loss=4.54, v_num=1]
Epoch 19: 100%|##########| 22/22 [00:05<00:00, 3.81it/s, loss=4.54, v_num=1]
Epoch 19: 100%|##########| 22/22 [00:05<00:00, 3.77it/s, loss=4.53, v_num=1]
Epoch 19: 100%|##########| 22/22 [00:05<00:00, 3.77it/s, loss=4.53, v_num=1]
Epoch 19: 100%|##########| 22/22 [00:06<00:00, 3.64it/s, loss=4.53, v_num=1]
other example
plot_knn_examples(embeddings, filenames)
What’s next?
# You could use the pre-trained model and train a classifier on top.
pretrained_resnet_backbone = model.backbone
# you can also store the backbone and use it in another code
state_dict = {"resnet18_parameters": pretrained_resnet_backbone.state_dict()}
torch.save(state_dict, "model.pth")
THIS COULD BE IN A NEW FILE (e.g. inference.py)
Make sure you place the model.pth file in the same folder as this code
# load the model in a new file for inference
resnet18_new = torchvision.models.resnet18()
# note that we need to create exactly the same backbone in order to load the weights
backbone_new = nn.Sequential(*list(resnet18_new.children())[:-1])
ckpt = torch.load("model.pth")
backbone_new.load_state_dict(ckpt["resnet18_parameters"])
<All keys matched successfully>
Next Steps
Interested in exploring other self-supervised models? Check out our other tutorials:
Total running time of the script: ( 4 minutes 28.300 seconds)