Assign Scheduled Runs to Specific Workers
When using multiple Lightly Workers that should work on multiple scheduled runs, it can be very useful to assign scheduled runs to specific workers. Lightly also offers this feature. It is based on labels:
- Each worker can have a set of labels, e.g.
["gpu-A100", "gpu", "bobs_worker", "team_worker"]
. Multiple workers can have some labels in common (e.g. multiplegpu
workers), but also have labels that only they have. - When scheduling a run, you can specify a set of labels the worker picking up the run must have. E.g. specify
["gpu-A100"]
to let the run be picked up by any worker with the labelgpu-A100
.
Specifying Worker Labels
The labels of a Lightly Worker must be assigned when registering it and cannot be changed later. Follow the usual steps when registering the Lightly Worker, but just add the labels
argument when calling client.register_compute_worker()
:
# execute the following code once to get a worker_id
from lightly.api import ApiWorkflowClient
client = ApiWorkflowClient(token="MY_LIGHTLY_TOKEN") # replace this with your token
# THE FOLLOWING LINE IS DIFFERENT
worker_id = client.register_compute_worker(name="worker-with-labels", labels=["gpu-A100", "gpu", "bobs_worker", "team_worker"])
# THE LAST LINE WAS DIFFERENT
print(worker_id)
Next, start the Lightly Worker on your machine with the just created worker_id
, just like usually.
Lightly Workers do not have any default labels.
Specifying Labels when Scheduling a Run
Follow the the usual steps of scheduling a run with one change: Additionally specify the runs_on
argument when calling client.schedule_compute_worker_run()
:
from lightly.api import ApiWorkflowClient
# Create a client with your token and configure it to use your dataset ID.
client = ApiWorkflowClient(token="MY_LIGHTLY_TOKEN", dataset_id="MY_DATASET_ID")
# Configure and schedule a run.
scheduled_run_id = client.schedule_compute_worker_run(
# THE FOLLOWING LINE IS DIFFERENT
runs_on =["gpu-A100"]
# THE LAST LINE WAS DIFFERENT
worker_config={...},
selection_config={...},
)
print(scheduled_run_id)
This scheduled run is going to picked up only by Workers that have the label "gpu-A100" among their labels.
Label Matching
The Worker only picks up runs whose runs_on
labels are a SUBSET of its own labels.
For legacy reasons, workers without labels will pickup all runs, including those that have a runs_on
specified.
For the same reason, runs without the runs_on
specified will be picked up by all workers, including those that have labels.
Thus we recommend you to fully switch to using labels for both all your workers and all your scheduled runs.
Updated 8 months ago