LightlyOne Worker 2.11.0 Release Notes
LightlyOne Worker 2.11.0 requires Lightly Python Client 1.5.0
Upgrade now
# Download and upgrade to the latest Lightly Worker version
docker pull lightly/worker:latest
# Updgrade to the latest Lightly Python Client version
pip install lightly --upgrade
Changes
use_datapool
is disabled by default to safeguard your dataset from accidental datapool usage. To enable datapool, set"worker": { "use_datapool": True}
. It is safe to enable this also for datasets not using datapool. See our docs for more information.- When using S3 datasources that have a retention policy enabled, we error out if it has not been configured how we should handle expiring files. You can either
SKIP
the file orABORT
the run to safeguard from failing runs. The time difference which triggers the aforementioned handling can also be configured. See our docs for more information. - The internal user within the LightlyOne Worker was renamed from
boris
tolightly_worker
to reduce confusion when using local storage and dealing with permissions. In the unlikely case of you specifically giving permission to theboris
user for mounted directories, please update the user tolightly_worker
. - The worker now errors when all predictions are missing or erroneous. If only some are invalid, the worker skips and ignores these and continues the selection.
input_dir
andshared_dir
mount points previously used to pass in checkpoints or relevant filenames have been removed. Please use the workflow outlined in Local Storage.
Improvements
-
When scheduling a run, all output of the LightlyOne Worker is now streamed directly to the LightlyOne Platform and can be monitored by yourself and your team.
-
By updating the pytorch version inside the LightlyOne Worker to 2.1.2 we reduced memory usage by several GB on systems with newer Nvidia Display Drivers (version 535.154.05). However, there is no significant memory change on older drivers (such as 470.223.02). Upgrading to driver 535.154.05 is therefore recommended for better memory efficiency. This update does not affect the CUDA version or LightlyOne Worker speed.
-
Selection configurations with prediction input and
CATEGORY_COUNT
no longer require acategories
property to be set and will default to allcategories
within theschema.json
. -
LightlyOne uses an LRU cache for predictions and metadata which defaults to either 20% of free disk space or 100GB, whichever is smaller. You can explicitly set this value to a different value by specifying
worker.cache_size
config option or disable it. -
It is now possible to use LightlyOne Worker behind a proxy. Configure the
HTTPS_PROXY
andLIGHTLY_CA_CERTS
environment variable when starting the LightlyOne Worker. -
Many error messages have been improved.