Install Lightly
Register at Lightly
If you haven't done this already, go to the Lightly Platform and sign up.
API Token
To authenticate the Python scripts and Lightly Worker, you need to provide them with your API token. You can retrieve it from the preferences page in the Lightly Platform.
Treat the API token like a password
Keep the token to yourself, and don’t share it with others. Anyone possessing the API token can access your datasets and all other related secrets!
Install the Lightly Worker
The Lightly Worker is Lightly's muscles, which means it lifts the heavy work when processing your input videos or images. Make sure you have an appropriate machine ready to install the Lightly Worker. If unsure, check out our hardware recommendations.
Docker
The Lightly Worker is a containerized application available on Docker Hub.
If you do not have it already, install Docker on your machine. We also have a more in-depth guide for installing Docker with GPU support
Pull and Update the Lightly Worker Image
Use the following command to pull the latest Lightly Worker image:
docker pull lightly/worker:latest
In case you experience any issues pulling the docker image check out our Known Issues & FAQ.
For updating the Lightly Worker Image, just run the same command again.
Sanity Check
Next, verify that the Lightly Worker was installed correctly by running the sanity check:
docker run --shm-size="1024m" --rm -it lightly/worker:latest sanity_check=True
You should see an output similar to this one:
[2022-05-02 20:37:27] Lightly Docker Solution v2.3.0
[2022-05-02 20:37:27] Congratulations! It looks like the Lightly container is running!
Install the Lightly Python Client
You will use the lightly Python package to interact with the Lightly Platform. It offers helper functions to create and delete datasets, schedule runs, and access the results.
pip3 install lightly
For updating, use the usual pip command:
pip3 install --upgrade lightly
To save disk space, it is possible to manually install only those lightly package dependencies which are required for the API client functionality. In particular, this will bypass installation of PyTorch and Torchvision:
pip install -r https://raw.githubusercontent.com/lightly-ai/lightly/master/requirements/base.txt pip install lightly --no-deps
Register the Lightly Worker
Finally, start the Lightly Worker in waiting mode. In this mode, the worker will long-poll the Lightly API for new runs to process. To do so, a worker first needs to be registered - run this Python script with your API token from above.
# execute the following code once to get a worker_id
from lightly.api import ApiWorkflowClient
client = ApiWorkflowClient(token="MY_LIGHTLY_TOKEN") # replace this with your token
#Â Create a Lightly Worker. If a worker with this name already exists, the id of the existing
#Â worker is returned.
worker_id = client.register_compute_worker(name="Default")
print(worker_id)
Store the worker_id
as you will use it for scheduling runs. In case you lost/ forgot your worker_id
you can find it in the Lightly Platform under My Worker Runs -> My Workers page.
Running Parallel Lightly Workers
If you plan on using multiple workers in parallel, you want them to use a unique
worker_id
. When you use theclient.register_compute_worker()
using default parameters, you create a worker with the namedefault
. If you then call the same api again, we return the sameworker_id
as there exists already a worker with namedefault
. To register multiple workers, you need to pass thename
argument to the function like thisclient.register_compute_worker(name='new_worker_name')
.
Then, on the same machine you downloaded the worker docker image to, start the worker with the following command.
docker run --shm-size="1024m" --gpus all --rm -it \
-e LIGHTLY_TOKEN={MY_LIGHTLY_TOKEN} \
-e LIGHTLY_WORKER_ID={MY_WORKER_ID} \
lightly/worker:latest
Loading Environment Variables from a File
When passing environment variables directly to a docker container using
-e
they could be exposed to other users that have access to the same machine. Instead you can pass a file with the variables using--env-file
. See the official docker docs for more information.
With your worker running, you should see an output similar to this one:
[2022-11-14 11:16:08] Lightly Docker Solution v2.3.9
[2022-11-14 11:16:08] You are using docker build: Tue Nov 8 11:31:05 UTC 2022.
[2022-11-14 11:16:08] Starting worker with id 637221746d1fd6f110889ddd
[2022-11-14 11:16:08] Worker with labels [] started. Waiting for runs...
Updated about 2 months ago