Python PackageReleases
Log In
Back to All

Lightly Worker 2.11.0 Release Notes

3 months ago by Jeremy Prescott

Lightly Worker 2.11.0 requires Lightly-SSL 1.5.0

Upgrade now

# Download and upgrade to the latest Lightly Worker version
docker pull lightly/worker:latest

# Updgrade to the latest lightly-ssl version
pip install lightly --upgrade

Changes

  • use_datapool is disabled by default to safeguard your dataset from accidental datapool usage. To enable datapool, set "worker": { "use_datapool": True}. It is safe to enable this also for datasets not using datapool. See our docs for more information.
  • When using S3 datasources that have a retention policy enabled, we error out if it has not been configured how we should handle expiring files. You can either SKIP the file or ABORT the run to safeguard from failing runs. The time difference which triggers the aforementioned handling can also be configured. See our docs for more information.
  • The internal user within the Lightly Worker was renamed from boris to lightly_worker to reduce confusion when using local storage and dealing with permissions. In the unlikely case of you specifically giving permission to the boris user for mounted directories, please update the user to lightly_worker.
  • The worker now errors when all predictions are missing or erroneous. If only some are invalid, the worker skips and ignores these and continues the selection.
  • input_dir and shared_dir mount points previously used to pass in checkpoints or relevant filenames have been removed. Please use the workflow outlined in Local Storage.

Improvements

  • When scheduling a run, all output of the Lightly Worker is now streamed directly to the Lightly Platform and can be monitored by yourself and your team.

  • By updating the pytorch version inside the Lightly Worker to 2.1.2 we reduced memory usage by several GB on systems with newer Nvidia Display Drivers (version 535.154.05). However, there is no significant memory change on older drivers (such as 470.223.02). Upgrading to driver 535.154.05 is therefore recommended for better memory efficiency. This update does not affect the CUDA version or Lightly Worker speed.

  • Selection configurations with prediction input and CATEGORY_COUNT no longer require a categories property to be set and will default to all categories within the schema.json.

  • Lightly uses an LRU cache for predictions and metadata which defaults to either 20% of free disk space or 100GB, whichever is smaller. You can explicitly set this value to a different value by specifying worker.cache_size config option or disable it.

  • It is now possible to use Lightly Worker behind a proxy. Configure the HTTPS_PROXY and LIGHTLY_CA_CERTS environment variable when starting the Lightly Worker.

  • Many error messages have been improved.