Install Lightly

Register at Lightly

If you haven't done this already, go to the Lightly Platform and sign up.

API Token

To authenticate the Python scripts and Lightly Worker, you need to provide them with your API token. You can retrieve it from the preferences page in the Lightly Platform.

🚧

Treat the API token like a password

Keep the token to yourself, and don’t share it with others. Anyone possessing the API token can access your datasets and all other related secrets!

All-in-One Jupyter Notebook for Running Lightly

There is a self-contained, all-in-one jupyter notebook for setting up and running Lightly and also a quick start guide. This convenient notebook includes all necessary installations and instructions in a single, easy-to-follow format. It's an excellent alternative to navigating through the more extensive guides. Here's what it covers:

1. Installing docker, the Lightly Worker, and the Lightly Python Client.
2. Downloading a dataset.
3. Scheduling a run on the dataset and processing it with the Lightly Worker.

Install the Lightly Worker

Use the following commands to pull the latest Lightly Worker image and check if it works:

docker pull lightly/worker:latest
docker run --shm-size="1024m" --rm -it lightly/worker:latest sanity_check=True

You should see an output similar to this one:

latest: Pulling from lightly/worker
... # The docker container is downloaded
Status: Downloaded newer image for lightly/worker:latest

[2024-04-10 15:11:36] Lightly Worker Solution v2.11.3
[2024-04-10 15:11:36] Congratulations! It looks like the Lightly container is running!

If this fails, see our in-depth installation guide.

Install the Lightly Python Client

You will use the lightly Python package to interact with the Lightly Platform. It offers helper functions to create and delete datasets, schedule runs, and access the results. In contrast to the Lightly Worker, the Python client doesn't require a lot of compute. Run the following commands on any machine from which you want to schedule data processing jobs.

Note that the Lightly Python client supports Python versions from 3.6 to 3.11. Python 3.12 is not yet supported as PyTorch has no official support for it yet.

pip3 install lightly

For updating, use the usual pip command:

pip3 install --upgrade lightly

📘

Install Python Client without PyTorch

To save disk space, it is possible to manually install only those lightly package dependencies which are required for the API client functionality. In particular, this will bypass installation of PyTorch and Torchvision:

Shell

pip install -r https://raw.githubusercontent.com/lightly-ai/lightly/master/requirements/base.txt
pip install lightly --no-deps

Register the Lightly Worker

Next, use the Python client to register a worker. The following script registers a worker and print its worker ID to the console.

# execute the following code once to get a worker_id
from lightly.api import ApiWorkflowClient

client = ApiWorkflowClient(token="MY_LIGHTLY_TOKEN") # replace this with your token

# Create a Lightly Worker. If a worker with this name already exists, the id of the existing
# worker is returned.
worker_id = client.register_compute_worker(name="Default")
print(f"worker_id: {worker_id}")

Store the worker_id as you will use it for scheduling runs. In case you lost/ forgot your worker_id you can find it in the Lightly Platform under My Workers page.

🚧

Running Parallel Lightly Workers

If you plan to use multiple workers in parallel, register them separately and use a unique name for each of them.

Start the Lightly Worker

Finally, on the machine where the Lightly Worker is installed, start it in waiting mode. In this mode, the worker will long-poll the Lightly API for new runs to process. (On CPU-only machines, remove the --gpus all config option.)

Cloud Storage

If you use cloud storage as datasource, start the worker as follows:

docker run --shm-size="1024m" --gpus all --rm -it \
	-e LIGHTLY_TOKEN={MY_LIGHTLY_TOKEN} \
	-e LIGHTLY_WORKER_ID={MY_WORKER_ID} \
	lightly/worker:latest

Local storage

If you want to use local storage as datasource, you need to mount two directories to the Lightly Worker:

docker run --shm-size="1024m" --gpus all --rm -it \
	-v /MY/PATH/TO/INPUT/DIRECTORY:/input_mount:ro \
	-v /MY/PATH/TO/WORKING/DIRECTORY:/lightly_mount \
	-e LIGHTLY_TOKEN={MY_LIGHTLY_TOKEN} \
	-e LIGHTLY_WORKER_ID={MY_WORKER_ID} \
	lightly/worker:latest

The two directories mounted via the -v flag are:

input_mount This is the local directory mounted to the Lightly Worker. It is /MY/PATH/TO/INPUT/DIRECTORY in the example. It contains the input images or videos for processing.

lightly_mount This is the working directory for the Lightly Worker. It is /MY/PATH/TO/WORKING/DIRECTORY in the example.

To learn more, read up on datasources for local storage.

Expected output

With your worker running, you should see an output similar to this one:

[2024-04-10 15:13:03] Lightly Worker Solution v2.11.3
[2024-04-10 15:13:04] You are using docker build: Tue Apr  2 07:46:28 UTC 2024.
[2024-04-10 15:13:04] Starting worker with id '660fe220b66cf09fb01bc2f5'...
[2024-04-10 13:13:04] Worker 2.11.3 can only process jobs scheduled with Lightly Python client 1.5 or higher.
[2024-04-10 15:13:04] Worker with labels '[]' started. Waiting for jobs...

Great! The next step is to configure a dataset and schedule a job the Lightly Worker can pick up.

🚧

Loading Environment Variables from a File

When passing environment variables directly to a docker container using -e they could be exposed to other users that have access to the same machine. Instead you can pass a file with the variables using --env-file. See the official docker docs for more information.