Debugging

Monitor a run

You can see the state of a run at these places:

Recommended: In the Lightly Platform.
For automation: From python by following the example earlier on this page: Monitoring a run.
In the command line output on the machine where you called the docker runcommand.

At all places, you can see the current state of the run and if it succeeded or failed. If it failed, check the error message, it often tells you directly what to fix and how.

Recover a Run

If you want to download artifacts of the latest Lightly Worker run for a specific dataset, you can also get that run given the dataset_id instead of the scheduled_run_id:

from lightly.api import ApiWorkflowClient

# Create the Lightly client to connect to the API.
client = ApiWorkflowClient(token="MY_LIGHTLY_TOKEN", dataset_id="MY_DATASET_ID")

# get all runs for a given dataset sorted from old to new
runs = client.get_compute_worker_runs(dataset_id=client.dataset_id)
run = runs[-1] # get the latest run
client.download_compute_worker_run_artifacts(run=run, output_dir="my_run/artifacts")

Logs

The log file is one of the Other Outputs created during a Lightly Worker run. It contains valuable log messages for debugging. If your run is not processed correctly or an error occurred, this file contains more detailed information about what went wrong. You can also download the log file of a scheduled run with the API client:

run = client.get_compute_worker_run_from_scheduled_run(scheduled_run_id=scheduled_run_id)
client.download_compute_worker_run_log(run=run, output_path="my_run/artifacts/log.txt")

Memory Logs

The memory log file is particularly useful if the Lightly Worker crashes due to an Out-Of-Memory error. You can also download it with the API client:

# download the log file
run = client.get_compute_worker_run_from_scheduled(scheduled_run_id=scheduled_run_id)
client.download_compute_worker_run_memory_log(run=run, output_path="my_run/artifacts/memlog.txt")

Extracting Logs from an Ongoing Run

Logs are uploaded to Lightly Platform after a run has finished. In rare occasions when logs from an ongoing run are needed the log file can be extracted from the running docker container. The log file is stored in /output_dir/<date>/<time>/log.txt:

# Find your running docker container id
docker ps

# Locate the log file
docker exec <YOUR_CONTAINER_ID> find /output_dir -name log.txt

# Copy the log file from the container to your local filesystem
docker cp <YOUR_CONTAINER_ID>:<OUTPUT_OF_THE_COMMAND_ABOVE> .

For example:

> docker ps
CONTAINER ID   IMAGE                   COMMAND                  CREATED          STATUS          PORTS     NAMES
21fa88e3c4df   lightly/worker:latest   "/bin/bash onprem-do…"   11 minutes ago   Up 11 minutes             interesting_wright

> docker exec 21fa88e3c4df find /output_dir -name log.txt
/output_dir/2023-12-31/23:59:59/log.txt

> docker cp 21fa88e3c4df:/output_dir/2023-12-31/23:59:59/log.txt .
Preparing to copy...
Copying from container - 32.77kB
Copying from container - 45.57kB
Successfully copied 45.57kB to /home/ubuntu/tmp/.