Outputs
Dataset
The datasets created by the LightlyOne Worker can be analyzed interactively with the tools provided in the LightlyOne Platform. These tools include interactive histograms, scatter, and correlation plots.
For more information, head to The LightlyOne Platform.
Artifacts
Each scheduled run creates several artifacts upon execution. These artifacts are uploaded to the LightlyOne Platform and can be accessed with the help of the Python client.
Here’s a list of the artifacts generated by the LightlyOne Worker:
- Checkpoint
- Report
- Corruptness Check Information
- Log File and Memory Log File
- Selected Sequences when using Sequence Selection
Download the Log File After a Crash
In case the run failed you can download the log.txt
file which contains detailed information to debug the error. It's also possible to retrieve the logs using the Python api as outlined here.
Download All Artifacts
The following code shows how to schedule a LightlyOne Worker run, wait until it has finished, and then download all artifacts to a local directory named my_run/artifacts
.
scheduled_run_id = client.schedule_compute_worker_run(
selection_config={
"n_samples": 50,
"strategies": [
{
"input": {
"type": "EMBEDDINGS"
},
"strategy": {
"type": "DIVERSITY"
}
}
]
}
)
# wait until the run is finished
for run_info in client.compute_worker_run_info_generator(scheduled_run_id=scheduled_run_id):
pass
# download all artifacts to "my_run/artifacts"
run = client.get_compute_worker_run_from_scheduled_run(scheduled_run_id=scheduled_run_id)
client.download_compute_worker_run_artifacts(run=run, output_dir="my_run/artifacts")
# you can also only download specific artifacts as outlined in the api docs
# https://docs.lightly.ai/self-supervised-learning/lightly.api.html
# e.g. to only get the report.json file
client.download_compute_worker_run_report_json(run=run, output_path="my_run/artifacts/report.json")
Download Artifacts by Dataset
It’s also possible to get the artifacts by only knowing the dataset id.
from lightly.api import ApiWorkflowClient
# Create the LightlyOne client to connect to the API.
client = ApiWorkflowClient(token="MY_LIGHTLY_TOKEN", dataset_id="MY_DATASET_ID")
# get all runs for a given dataset sorted from old to new
runs = client.get_compute_worker_runs(dataset_id=client.dataset_id)
run = runs[-1] # get the latest run
# download all artifacts to "my_run/artifacts"
client.download_compute_worker_run_artifacts(run=run, output_dir="my_run/artifacts")
The above examples show how to download all artifacts at once. It’s also possible to download each artifact on its own. To see how, please refer to the individual artifacts on the page Report above.
Direct Access Via Cloud Provider
You can also inspect your cloud bucket or local directory manually to fetch some of the artifacts described above. Check out the cloud storage - lightly path structure document to see where this is possible and what path pattern to expect.
Updated 3 months ago