Lightly Python ClientReleases
Log In
Back to All

LightlyOne Worker 2.11.0 Release Notes

6 months ago by Jeremy Prescott

LightlyOne Worker 2.11.0 requires Lightly Python Client 1.5.0

Upgrade now

# Download and upgrade to the latest Lightly Worker version
docker pull lightly/worker:latest

# Updgrade to the latest Lightly Python Client version
pip install lightly --upgrade

Changes

  • use_datapool is disabled by default to safeguard your dataset from accidental datapool usage. To enable datapool, set "worker": { "use_datapool": True}. It is safe to enable this also for datasets not using datapool. See our docs for more information.
  • When using S3 datasources that have a retention policy enabled, we error out if it has not been configured how we should handle expiring files. You can either SKIP the file or ABORT the run to safeguard from failing runs. The time difference which triggers the aforementioned handling can also be configured. See our docs for more information.
  • The internal user within the LightlyOne Worker was renamed from boris to lightly_worker to reduce confusion when using local storage and dealing with permissions. In the unlikely case of you specifically giving permission to the boris user for mounted directories, please update the user to lightly_worker.
  • The worker now errors when all predictions are missing or erroneous. If only some are invalid, the worker skips and ignores these and continues the selection.
  • input_dir and shared_dir mount points previously used to pass in checkpoints or relevant filenames have been removed. Please use the workflow outlined in Local Storage.

Improvements

  • When scheduling a run, all output of the LightlyOne Worker is now streamed directly to the LightlyOne Platform and can be monitored by yourself and your team.

  • By updating the pytorch version inside the LightlyOne Worker to 2.1.2 we reduced memory usage by several GB on systems with newer Nvidia Display Drivers (version 535.154.05). However, there is no significant memory change on older drivers (such as 470.223.02). Upgrading to driver 535.154.05 is therefore recommended for better memory efficiency. This update does not affect the CUDA version or LightlyOne Worker speed.

  • Selection configurations with prediction input and CATEGORY_COUNT no longer require a categories property to be set and will default to all categories within the schema.json.

  • LightlyOne uses an LRU cache for predictions and metadata which defaults to either 20% of free disk space or 100GB, whichever is smaller. You can explicitly set this value to a different value by specifying worker.cache_size config option or disable it.

  • It is now possible to use LightlyOne Worker behind a proxy. Configure the HTTPS_PROXY and LIGHTLY_CA_CERTS environment variable when starting the LightlyOne Worker.

  • Many error messages have been improved.