lightly.utils
.io
I/O operations to save and load embeddings.
- class lightly.utils.io.COCO_ANNOTATION_KEYS
Enum of coco annotation keys complemented with a key for custom metadata.
- lightly.utils.io.check_embeddings(path: str, remove_additional_columns: bool = False)
Raises an error if the embeddings csv file has not the correct format
Use this check whenever you want to upload an embedding to the Lightly Platform. This method only checks whether the header row matches the specs: https://docs.lightly.ai/self-supervised-learning/getting_started/command_line_tool.html#id1
- Parameters
path – Path to the embedding csv file
remove_additional_columns – If True, all additional columns which are not in {filenames, embeddings_x, labels} are removed. If false, they are kept unchanged.
- Raises
RuntimeError –
- lightly.utils.io.check_filenames(filenames: List[str])
Raises an error if one of the filenames is misformatted
- Parameters
filenames – A list of string being filenames
- lightly.utils.io.format_custom_metadata(custom_metadata: List[Tuple[str, Dict]])
Transforms custom metadata into a format which can be handled by Lightly.
- Parameters
custom_metadata – List of tuples (filename, metadata) where metadata is a dictionary.
- Returns
A dictionary of formatted custom metadata.
Examples
>>> custom_metadata = [ >>> ('hello.png', {'number_of_people': 1}), >>> ('world.png', {'number_of_people': 3}), >>> ] >>> >>> format_custom_metadata(custom_metadata) >>> > { >>> > 'images': [{'id': 0, 'file_name': 'hello.png'}, {'id': 1, 'file_name': 'world.png'}], >>> > 'metadata': [{'image_id': 0, 'number_of_people': 1}, {'image_id': 1, 'number_of_people': 3}] >>> > }
- lightly.utils.io.load_embeddings(path: str)
Loads embeddings from a csv file in a Lightly compatible format.
- Parameters
path – Path to the csv file.
- Returns
The embeddings as a numpy array, labels as a list of integers, and filenames as a list of strings in the order they were saved.
The embeddings will always be of the Float32 datatype.
Examples
>>> import lightly.utils.io as io >>> embeddings, labels, filenames = io.load_embeddings( >>> 'path/to/my/embeddings.csv')
- lightly.utils.io.load_embeddings_as_dict(path: str, embedding_name: str = 'default', return_all: bool = False)
Loads embeddings from csv and store it in a dictionary for transfer.
Loads embeddings to a dictionary which can be serialized and sent to the Lightly servers. It is recommended that the embedding_name is always specified because the Lightly web-app does not allow two embeddings with the same name.
- Parameters
path – Path to the csv file.
embedding_name – Name of the embedding for the platform.
return_all – If true, return embeddings, labels, and filenames, too.
- Returns
A dictionary containing the embedding information (see load_embeddings)
Examples
>>> import lightly.utils.io as io >>> embedding_dict = io.load_embeddings_as_dict( >>> 'path/to/my/embeddings.csv', >>> embedding_name='MyEmbeddings') >>> >>> result = io.load_embeddings_as_dict( >>> 'path/to/my/embeddings.csv', >>> embedding_name='MyEmbeddings', >>> return_all=True) >>> embedding_dict, embeddings, labels, filenames = result
- lightly.utils.io.save_custom_metadata(path: str, custom_metadata: List[Tuple[str, Dict]])
Saves custom metadata in a .json.
- Parameters
path – Filename of the .json file where the data should be stored.
custom_metadata – List of tuples (filename, metadata) where metadata is a dictionary.
- lightly.utils.io.save_embeddings(path: str, embeddings: numpy.ndarray, labels: List[int], filenames: List[str])
Saves embeddings in a csv file in a Lightly compatible format.
Creates a csv file at the location specified by path and saves embeddings, labels, and filenames.
- Parameters
path – Path to the csv file.
embeddings – Embeddings of the images as a numpy array (n x d).
labels – List of integer labels.
filenames – List of filenames.
- Raises
ValueError – If embeddings, labels, and filenames have different lengths.
Examples
>>> import lightly.utils.io as io >>> io.save_embeddings( >>> 'path/to/my/embeddings.csv', >>> embeddings, >>> labels, >>> filenames)
- lightly.utils.io.save_schema(path: str, task_type: str, ids: List[int], names: List[str])
Saves a prediction schema in the right format.
- Parameters
path – Where to store the schema.
task_type – Task type (e.g. classification, object-detection).
ids – List of category ids.
names – List of category names.
- lightly.utils.io.save_tasks(path: str, tasks: List[str])
Saves a list of prediction task names in the right format.
- Parameters
path – Where to store the task names.
tasks – List of task names.
.embeddings_2d
Transform embeddings to two-dimensional space for visualization.
- class lightly.utils.embeddings_2d.PCA(n_components: int = 2, eps: float = 1e-10)
Handmade PCA to bypass sklearn dependency.
- n_components
Number of principal components to keep.
- eps
Epsilon for numerical stability.
- fit(X: numpy.ndarray)
Fits PCA to data in X.
- Parameters
X – Datapoints stored in numpy array of size n x d.
- Returns
PCA object to transform datapoints.
- transform(X: numpy.ndarray)
Uses PCA to transform data in X.
- Parameters
X – Datapoints stored in numpy array of size n x d.
- Returns
Numpy array of n x p datapoints where p <= d.
- lightly.utils.embeddings_2d.fit_pca(embeddings: numpy.ndarray, n_components: int = 2, fraction: Optional[float] = None)
Fits PCA to randomly selected subset of embeddings.
For large datasets, it can be unfeasible to perform PCA on the whole data. This method can fit a PCA on a fraction of the embeddings in order to save computational resources.
- Parameters
embeddings – Datapoints stored in numpy array of size n x d.
n_components – Number of principal components to keep.
fraction – Fraction of the dataset to fit PCA on.
- Returns
A transformer which can be used to transform embeddings to lower dimensions.
- Raises
ValueError – If fraction < 0 or fraction > 1.
.benchmarking
Helper modules for benchmarking SSL models
- class lightly.utils.benchmarking.BenchmarkModule(dataloader_kNN: torch.utils.data.dataloader.DataLoader, num_classes: int, knn_k: int = 200, knn_t: float = 0.1)
A PyTorch Lightning Module for automated kNN callback
At the end of every training epoch we create a feature bank by feeding the dataloader_kNN passed to the module through the backbone. At every validation step we predict features on the validation data. After all predictions on validation data (validation_epoch_end) we evaluate the predictions on a kNN classifier on the validation data using the feature_bank features from the train data.
We can access the highest test accuracy during a kNN prediction using the max_accuracy attribute.
- backbone
The backbone model used for kNN validation. Make sure that you set the backbone when inheriting from BenchmarkModule.
- max_accuracy
Floating point number between 0.0 and 1.0 representing the maximum test accuracy the benchmarked model has achieved.
- dataloader_kNN
Dataloader to be used after each training epoch to create feature bank.
- num_classes
Number of classes. E.g. for cifar10 we have 10 classes. (default: 10)
- knn_k
Number of nearest neighbors for kNN
- knn_t
Temperature parameter for kNN
Examples
>>> class SimSiamModel(BenchmarkingModule): >>> def __init__(dataloader_kNN, num_classes): >>> super().__init__(dataloader_kNN, num_classes) >>> resnet = lightly.models.ResNetGenerator('resnet-18') >>> self.backbone = nn.Sequential( >>> *list(resnet.children())[:-1], >>> nn.AdaptiveAvgPool2d(1), >>> ) >>> self.resnet_simsiam = >>> lightly.models.SimSiam(self.backbone, num_ftrs=512) >>> self.criterion = lightly.loss.SymNegCosineSimilarityLoss() >>> >>> def forward(self, x): >>> self.resnet_simsiam(x) >>> >>> def training_step(self, batch, batch_idx): >>> (x0, x1), _, _ = batch >>> x0, x1 = self.resnet_simsiam(x0, x1) >>> loss = self.criterion(x0, x1) >>> return loss >>> def configure_optimizers(self): >>> optim = torch.optim.SGD( >>> self.resnet_simsiam.parameters(), lr=6e-2, momentum=0.9 >>> ) >>> return [optim] >>> >>> model = SimSiamModel(dataloader_train_kNN) >>> trainer = pl.Trainer() >>> trainer.fit( >>> model, >>> train_dataloader=dataloader_train_ssl, >>> val_dataloaders=dataloader_test >>> ) >>> # you can get the peak accuracy using >>> print(model.max_accuracy)
- training_epoch_end(outputs)
Called at the end of the training epoch with the outputs of all training steps. Use this in case you need to do something with all the outputs returned by
training_step()
.# the pseudocode for these calls train_outs = [] for train_batch in train_data: out = training_step(train_batch) train_outs.append(out) training_epoch_end(train_outs)
- Parameters
outputs – List of outputs you defined in
training_step()
. If there are multiple optimizers or when usingtruncated_bptt_steps > 0
, the lists have the dimensions (n_batches, tbptt_steps, n_optimizers). Dimensions of length 1 are squeezed.- Returns
None
Note
If this method is not overridden, this won’t be called.
def training_epoch_end(self, training_step_outputs): # do something with all training_step outputs for out in training_step_outputs: ...
- validation_epoch_end(outputs)
Called at the end of the validation epoch with the outputs of all validation steps.
# the pseudocode for these calls val_outs = [] for val_batch in val_data: out = validation_step(val_batch) val_outs.append(out) validation_epoch_end(val_outs)
- Parameters
outputs – List of outputs you defined in
validation_step()
, or if there are multiple dataloaders, a list containing a list of outputs for each dataloader.- Returns
None
Note
If you didn’t define a
validation_step()
, this won’t be called.Examples
With a single dataloader:
def validation_epoch_end(self, val_step_outputs): for out in val_step_outputs: ...
With multiple dataloaders, outputs will be a list of lists. The outer list contains one entry per dataloader, while the inner list contains the individual outputs of each validation step for that dataloader.
def validation_epoch_end(self, outputs): for dataloader_output_result in outputs: dataloader_outs = dataloader_output_result.dataloader_i_outputs self.log("final_metric", final_value)
- validation_step(batch, batch_idx)
Operates on a single batch of data from the validation set. In this step you’d might generate examples or calculate anything of interest like accuracy.
# the pseudocode for these calls val_outs = [] for val_batch in val_data: out = validation_step(val_batch) val_outs.append(out) validation_epoch_end(val_outs)
- Parameters
batch – The output of your
DataLoader
.batch_idx – The index of this batch.
dataloader_idx – The index of the dataloader that produced this batch. (only if multiple val dataloaders used)
- Returns
Any object or value
None
- Validation will skip to the next batch
# pseudocode of order val_outs = [] for val_batch in val_data: out = validation_step(val_batch) if defined("validation_step_end"): out = validation_step_end(out) val_outs.append(out) val_outs = validation_epoch_end(val_outs)
# if you have one val dataloader: def validation_step(self, batch, batch_idx): ... # if you have multiple val dataloaders: def validation_step(self, batch, batch_idx, dataloader_idx=0): ...
Examples:
# CASE 1: A single validation dataset def validation_step(self, batch, batch_idx): x, y = batch # implement your own out = self(x) loss = self.loss(out, y) # log 6 example images # or generated text... or whatever sample_imgs = x[:6] grid = torchvision.utils.make_grid(sample_imgs) self.logger.experiment.add_image('example_images', grid, 0) # calculate acc labels_hat = torch.argmax(out, dim=1) val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0) # log the outputs! self.log_dict({'val_loss': loss, 'val_acc': val_acc})
If you pass in multiple val dataloaders,
validation_step()
will have an additional argument. We recommend setting the default value of 0 so that you can quickly switch between single and multiple dataloaders.# CASE 2: multiple validation dataloaders def validation_step(self, batch, batch_idx, dataloader_idx=0): # dataloader_idx tells you which dataset this is. ...
Note
If you don’t need to validate you don’t need to implement this method.
Note
When the
validation_step()
is called, the model has been put in eval mode and PyTorch gradients have been disabled. At the end of validation, the model goes back to training mode and gradients are enabled.
- lightly.utils.benchmarking.knn_predict(feature: torch.Tensor, feature_bank: torch.Tensor, feature_labels: torch.Tensor, num_classes: int, knn_k: int = 200, knn_t: float = 0.1) torch.Tensor
Run kNN predictions on features based on a feature bank
This method is commonly used to monitor performance of self-supervised learning methods.
The default parameters are the ones used in https://arxiv.org/pdf/1805.01978v1.pdf.
- Parameters
feature – Tensor of shape [N, D] for which you want predictions
feature_bank – Tensor of a database of features used for kNN
feature_labels – Labels for the features in our feature_bank
num_classes – Number of classes (e.g. 10 for CIFAR-10)
knn_k – Number of k neighbors used for kNN
knn_t – Temperature parameter to reweights similarities for kNN
- Returns
A tensor containing the kNN predictions
Examples
>>> images, targets, _ = batch >>> feature = backbone(images).squeeze() >>> # we recommend to normalize the features >>> feature = F.normalize(feature, dim=1) >>> pred_labels = knn_predict( >>> feature, >>> feature_bank, >>> targets_bank, >>> num_classes=10, >>> )
.debug
- lightly.utils.debug.apply_transform_without_normalize(image: PIL.Image.Image, transform)
Applies the transform to the image but skips ToTensor and Normalize.
- lightly.utils.debug.generate_grid_of_augmented_images(input_images: List[PIL.Image.Image], collate_function: Union[lightly.data.collate.BaseCollateFunction, lightly.data.collate.MultiViewCollateFunction])
Returns a grid of augmented images. Images in a column belong together.
This function ignores the transforms ToTensor and Normalize for visualization purposes.
- Parameters
input_images – List of PIL images for which the augmentations should be plotted.
collate_function – The collate function of the self-supervised learning algorithm. Must be of type BaseCollateFunction or MultiViewCollateFunction.
- Returns
A grid of augmented images. Images in a column belong together.
- lightly.utils.debug.plot_augmented_images(input_images: List[PIL.Image.Image], collate_function: Union[lightly.data.collate.BaseCollateFunction, lightly.data.collate.MultiViewCollateFunction])
Returns a figure showing original images in the left column and augmented images to their right.
This function ignores the transforms ToTensor and Normalize for visualization purposes.
- Parameters
input_images – List of PIL images for which the augmentations should be plotted.
collate_function – The collate function of the self-supervised learning algorithm. Must be of type BaseCollateFunction or MultiViewCollateFunction.
- Returns
A figure showing the original images in the left column and the augmented images to their right. If the collate_function is an instance of the BaseCollateFunction, two example augmentations are shown. For MultiViewCollateFunctions all the generated views are shown.
- lightly.utils.debug.std_of_l2_normalized(z: torch.Tensor) torch.Tensor
Calculates the mean of the standard deviation of z along each dimension.
This measure was used by [0] to determine the level of collapse of the learned representations. If the returned number is 0., the outputs z have collapsed to a constant vector. “If the output z has a zero-mean isotropic Gaussian distribution” [0], the returned number should be close to 1/sqrt(d) where d is the dimensionality of the output.
[0]: https://arxiv.org/abs/2011.10566
- Parameters
z – A torch tensor of shape batch_size x dimension.
- Returns
The mean of the standard deviation of the l2 normalized tensor z along each dimension.
.dist
- class lightly.utils.dist.GatherLayer(*args, **kwargs)
Gather tensors from all processes, supporting backward propagation.
This code was taken and adapted from here: https://github.com/Spijkervet/SimCLR
- static backward(ctx, *grads: torch.Tensor) torch.Tensor
Defines a formula for differentiating the operation with backward mode automatic differentiation (alias to the vjp function).
This function is to be overridden by all subclasses.
It must accept a context
ctx
as the first argument, followed by as many outputs as theforward()
returned (None will be passed in for non tensor outputs of the forward function), and it should return as many tensors, as there were inputs toforward()
. Each argument is the gradient w.r.t the given output, and each returned value should be the gradient w.r.t. the corresponding input. If an input is not a Tensor or is a Tensor not requiring grads, you can just pass None as a gradient for that input.The context can be used to retrieve tensors saved during the forward pass. It also has an attribute
ctx.needs_input_grad
as a tuple of booleans representing whether each input needs gradient. E.g.,backward()
will havectx.needs_input_grad[0] = True
if the first input toforward()
needs gradient computated w.r.t. the output.
- static forward(ctx, input: torch.Tensor) Tuple[torch.Tensor, ...]
Performs the operation.
This function is to be overridden by all subclasses.
It must accept a context ctx as the first argument, followed by any number of arguments (tensors or other types).
The context can be used to store arbitrary data that can be then retrieved during the backward pass. Tensors should not be stored directly on ctx (though this is not currently enforced for backward compatibility). Instead, tensors should be saved either with
ctx.save_for_backward()
if they are intended to be used inbackward
(equivalently,vjp
) orctx.save_for_forward()
if they are intended to be used for injvp
.
- lightly.utils.dist.eye_rank(n: int, device: Optional[torch.device] = None) torch.Tensor
Returns an (n, n * world_size) zero matrix with the diagonal for the rank of this process set to 1.
Example output where n=3, the current process has rank 1, and there are 4 processes in total:
rank0 rank1 rank2 rank3 0 0 0 | 1 0 0 | 0 0 0 | 0 0 0 0 0 0 | 0 1 0 | 0 0 0 | 0 0 0 0 0 0 | 0 0 1 | 0 0 0 | 0 0 0
Equivalent to torch.eye for undistributed settings or if world size == 1.
- Parameters
n – Size of the square matrix on a single process.
device – Device on which the matrix should be created.
- lightly.utils.dist.gather(input: torch.Tensor) Tuple[torch.Tensor]
Gathers this tensor from all processes. Supports backprop.
- lightly.utils.dist.rank() int
Returns the rank of the current process.
- lightly.utils.dist.world_size() int
Returns the current world size (number of distributed processes).
.reordering
- lightly.utils.reordering.sort_items_by_keys(keys: List[any], items: List[any], sorted_keys: List[any])
Sorts the items in the same order as the sorted keys.
- Parameters
keys – Keys by which items can be identified.
items – Items to sort.
sorted_keys – Keys in sorted order.
- Returns
The list of sorted items.
Examples
>>> keys = [3, 2, 1] >>> items = ['!', 'world', 'hello'] >>> sorted_keys = [1, 2, 3] >>> sorted_items = sort_items_by_keys( >>> keys, >>> items, >>> sorted_keys, >>> ) >>> print(sorted_items) >>> > ['hello', 'world', '!']
.version_compare
Utility method for comparing versions of libraries
- lightly.utils.version_compare.version_compare(v0: str, v1: str)
Returns 1 if version of v0 is larger than v1 and -1 otherwise
Use this method to compare Python package versions and see which one is newer.
Examples
>>> # compare two versions >>> version_compare('1.2.0', '1.1.2') >>> 1