Lightly Worker Installation Guide

The Lightly Worker does the heavy work when processing your input videos or images. Make sure you have an appropriate machine ready to install it.

Requirements

These are the requirements for running the Lightly Worker.

	CPU	Operating System	GPU	RAM	Software Requirements
Required	x86_64	Any	None	8GB	Docker
Recommended	x86_64	Linux	T4 or similar	32GB	NVIDIA Container Toolkit

Note that it requires an x86 CPU (Intel or AMD), which excludes the ARM-based M-CPUs in newer Macs.

Pull and Update the Lightly Worker Image

Use the following command to pull the latest Lightly Worker image:

docker pull lightly/worker:latest

In case you experience any issues pulling the docker image check out the section "Install Docker" below.

For updating the Lightly Worker Image, just rerun the same command.

Sanity Check

Next, verify that the Lightly Worker was installed correctly by running the sanity check:

docker run --shm-size="1024m" --rm -it lightly/worker:latest sanity_check=True

You should see an output similar to this one:

[2022-05-02 20:37:27] Lightly Docker Solution v2.10.0
[2022-05-02 20:37:27] Congratulations! It looks like the Lightly container is running!

Install Docker, Optionally With GPU Support

The Lightly Worker is a containerized application available on Docker Hub. If you do not have it already, install Docker on your machine. Below you'll find the instructions to install Docker with GPU support on Ubuntu.

For other operating systems, follow the official Docker installation guide.

Set up the package repository

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  
sudo apt-get update

Install nvidia-docker

sudo apt-get install -y nvidia-docker2

Restart the Docker service

sudo systemctl restart docker

Test the installation

sudo docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   34C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Make sure you can run Docker as non-root (recommended for security)

sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

Test that you can run Docker as non-root with GPU support.

docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi