Lightly Worker Installation Guide

The Lightly Worker does the heavy work when processing your input videos or images. Make sure you have an appropriate machine ready to install it.

Requirements

These are the requirements for running the Lightly Worker.

CPUOperating SystemGPURAMSoftware Requirements
Requiredx86_64, arm64AnyNone8GBDocker
Recommendedx86_64LinuxT4 or similar32GBNVIDIA Container Toolkit

🚧

Mac with Apple Silicon (M1 or newer)

If you use a Mac with an Apple silicon chip, make sure to enable Rosetta emulation in Docker Desktop for fast processing. To enable it go to Docker Desktop > Settings > General > Use Rosetta for x86_64/amd64 emulation on Apple Silicon. This requires Docker Desktop 4.25 or later.

Pull and Update the Lightly Worker Image

Use the following command to pull the latest Lightly Worker image:

docker pull lightly/worker:latest

In case you experience any issues pulling the docker image check out the section "Install Docker" below.

For updating the Lightly Worker Image, just rerun the same command.

Sanity Check

Next, verify that the Lightly Worker was installed correctly by running the sanity check:

docker run --shm-size="1024m" --rm -it lightly/worker:latest sanity_check=True

You should see an output similar to this one:

[2022-05-02 20:37:27] Lightly Docker Solution v2.10.0
[2022-05-02 20:37:27] Congratulations! It looks like the Lightly container is running!

Install Docker, Optionally With GPU Support

Below you'll find the instructions to install Docker with GPU support on Ubuntu.

For other operating systems, follow the official Docker installation guide.

  1. Set up the package repository
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  
sudo apt-get update
  1. Install NVIDIA Container Toolkit
sudo apt-get install -y nvidia-container-toolkit

For other OS than Ubuntu, or if problems occur, follow the official docs.

  1. Restart the Docker service
sudo systemctl restart docker
  1. Test the installation
sudo docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   34C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

  1. Make sure you can run Docker as non-root (recommended for security)
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker
  1. Test that you can run Docker as non-root with GPU support.
docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi