Install Lightly

Register at Lightly

If you haven't done this already, go to the Lightly Platform and sign up.

API Token

To authenticate the Python scripts and Lightly Worker, you need to provide them with your API token. You can retrieve it from the preferences page in the Lightly Platform.

🚧
Treat the API token like a password
Keep the token to yourself, and don’t share it with others. Anyone possessing the API token can access your datasets and all other related secrets!

Install the Lightly Worker

The Lightly Worker is Lightly's muscles, which means it lifts the heavy work when processing your input videos or images. Make sure you have an appropriate machine ready to install the Lightly Worker. If unsure, check out our hardware recommendations.

Docker

The Lightly Worker is a containerized application available on Docker Hub.

If you do not have it already, install Docker on your machine. We also have a more in-depth guide for installing Docker with GPU support

Pull and Update the Lightly Worker Image

Use the following command to pull the latest Lightly Worker image:

docker pull lightly/worker:latest

In case you experience any issues pulling the docker image check out our Known Issues & FAQ.

For updating the Lightly Worker Image, just run the same command again.

Sanity Check

Next, verify that the Lightly Worker was installed correctly by running the sanity check:

docker run --shm-size="1024m" --rm -it lightly/worker:latest sanity_check=True

You should see an output similar to this one:

[2022-05-02 20:37:27] Lightly Docker Solution v2.3.0
[2022-05-02 20:37:27] Congratulations! It looks like the Lightly container is running!

Install the Lightly Python Client

You will use the lightly Python package to interact with the Lightly Platform. It offers helper functions to create and delete datasets, schedule runs, and access the results.

pip3 install lightly

For updating, use the usual pip command:

pip3 install --upgrade lightly

📘
To save disk space, it is possible to manually install only those lightly package dependencies which are required for the API client functionality. In particular, this will bypass installation of PyTorch and Torchvision:
pip install -r https://raw.githubusercontent.com/lightly-ai/lightly/master/requirements/base.txt
pip install lightly --no-deps

Register the Lightly Worker

Finally, start the Lightly Worker in waiting mode. In this mode, the worker will long-poll the Lightly API for new runs to process. To do so, a worker first needs to be registered - run this Python script with your API token from above.

# execute the following code once to get a worker_id
from lightly.api import ApiWorkflowClient

client = ApiWorkflowClient(token="MY_LIGHTLY_TOKEN") # replace this with your token

# Create a Lightly Worker. If a worker with this name already exists, the id of the existing
# worker is returned.
worker_id = client.register_compute_worker(name="Default")
print(worker_id)

Store the worker_id as you will use it for scheduling runs. In case you lost/ forgot your worker_id you can find it in the Lightly Platform under My Worker Runs -> My Workers page.

🚧
Running Parallel Lightly Workers
If you plan on using multiple workers in parallel, you want them to use a unique worker_id. When you use the client.register_compute_worker() using default parameters, you create a worker with the name default. If you then call the same api again, we return the same worker_id as there exists already a worker with name default. To register multiple workers, you need to pass the name argument to the function like this client.register_compute_worker(name='new_worker_name').

Then, on the same machine you downloaded the worker docker image to, start the worker with the following command.

📘
Local Storage
If you want to use local storage, use the command from the second tab and replace the paths on the left side of the colon with the directories you want to mount. You will learn about the significance of them later. For now, consider the input_mount your input directory and the lightly_mount as a working directory. To learn more, read up on datasources for local storage.

docker run --shm-size="1024m" --gpus all --rm -it \
	-e LIGHTLY_TOKEN={MY_LIGHTLY_TOKEN} \
	-e LIGHTLY_WORKER_ID={MY_WORKER_ID} \
	lightly/worker:latest

docker run --shm-size="1024m" --gpus all --rm -it \
  -v /MY/PATH/TO/INPUT/DIRECTORY:/input_mount:ro \
  -v /MY/PATH/TO/WORKING/DIRECTORY:/lightly_mount \
	-e LIGHTLY_TOKEN={MY_LIGHTLY_TOKEN} \
	-e LIGHTLY_WORKER_ID={MY_WORKER_ID} \
	lightly/worker:latest

🚧
Loading Environment Variables from a File
When passing environment variables directly to a docker container using -e they could be exposed to other users that have access to the same machine. Instead you can pass a file with the variables using --env-file. See the official docker docs for more information.

With your worker running, you should see an output similar to this one:

[2022-11-14 11:16:08] Lightly Docker Solution v2.3.9
[2022-11-14 11:16:08] You are using docker build: Tue Nov  8 11:31:05 UTC 2022.
[2022-11-14 11:16:08] Starting worker with id 637221746d1fd6f110889ddd
[2022-11-14 11:16:08] Worker with labels [] started. Waiting for runs...