isolateserver.py: trim old items |
||||
Issue descriptionRight now items in the bot local isolated caches are removed when: - Maximum number of files in the cache (self.policies.max_items) - Maximum total bytes of the cache (self.policies.max_cache_size) - Minimum free disk space (self.policies.min_free_space) Proposal: Also trim items not touched since N days. The hypothesis is that keeping objects more than 30 days old is dead weight not worth keeping. The number of days to be fine tuned, 9 days is likely a good number. The change is fairly simple and harmless. Benefit: Will significantly reduce the size of the cache for bots that are not continuously recycled. AIs: - Add flag to run_isolated.py/isolateserver.py - Implement trimming of old items in isolateserver.DiskCache._trim() - Expose flag in bot_config.get_settings()
,
Apr 18 2018
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/a13cead8267a44095f7851bed2b50f0edf7babdc commit a13cead8267a44095f7851bed2b50f0edf7babdc Author: Marc-Antoine Ruel <maruel@chromium.org> Date: Wed Apr 18 22:07:21 2018 [client] Trim items from the local cache older than 3 weeks This will help performance by removing dead luggage. I suspect this is a significant issue on non-recycled bots. This shouldn't have performance cost on recycled bots. Tweak LRUDict loading to be faster by loading all at once instead of doing an iteration. Update run_isolated_test that was using really old time, so adding time based eviction broke the test. Bug: 801870 Change-Id: I930bbb83d619fc49c1b51f25d9d884b40c8f3b1d Reviewed-on: https://chromium-review.googlesource.com/1017603 Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org> [modify] https://crrev.com/a13cead8267a44095f7851bed2b50f0edf7babdc/client/isolateserver.py [modify] https://crrev.com/a13cead8267a44095f7851bed2b50f0edf7babdc/client/tests/isolateserver_test.py [modify] https://crrev.com/a13cead8267a44095f7851bed2b50f0edf7babdc/client/tests/run_isolated_test.py [modify] https://crrev.com/a13cead8267a44095f7851bed2b50f0edf7babdc/client/utils/lru.py
,
May 15 2018
,
May 17 2018
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/c12f6ca8939f882d5d3373b29297468037b269b3 commit c12f6ca8939f882d5d3373b29297468037b269b3 Author: Marc-Antoine Ruel <maruel@chromium.org> Date: Thu May 17 19:07:51 2018 [client] trim old named caches and CIPD. - CIPD cache is now 512Gb, 300 items and 3 weeks old. - CIPD client cache is now 1Gb, 10 items and 3 weeks old - Named cache is now 3 weeks old, 50 items. - Maximum size is not implemented yet but planned to be 1Tb. - We can make this configurable later if needed. Refactor towards a more coherent object eviction policy. The end goal is to move DiskCache and named caches into local_caching.py. Bug: 801870 Change-Id: I6a2d19ea8ddc1e6e812e85a75599e8bf688e90ef Reviewed-on: https://chromium-review.googlesource.com/1060296 Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org> Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> Reviewed-by: Nodir Turakulov <nodir@chromium.org> [modify] https://crrev.com/c12f6ca8939f882d5d3373b29297468037b269b3/appengine/swarming/server/bot_archive.py [modify] https://crrev.com/c12f6ca8939f882d5d3373b29297468037b269b3/appengine/swarming/swarming_bot/bot_code/task_runner_test.py [modify] https://crrev.com/c12f6ca8939f882d5d3373b29297468037b269b3/client/cipd.py [modify] https://crrev.com/c12f6ca8939f882d5d3373b29297468037b269b3/client/isolateserver.py [add] https://crrev.com/c12f6ca8939f882d5d3373b29297468037b269b3/client/local_caching.py [modify] https://crrev.com/c12f6ca8939f882d5d3373b29297468037b269b3/client/named_cache.py [modify] https://crrev.com/c12f6ca8939f882d5d3373b29297468037b269b3/client/run_isolated.py [modify] https://crrev.com/c12f6ca8939f882d5d3373b29297468037b269b3/client/tests/isolateserver_test.py [modify] https://crrev.com/c12f6ca8939f882d5d3373b29297468037b269b3/client/tests/named_cache_test.py [modify] https://crrev.com/c12f6ca8939f882d5d3373b29297468037b269b3/client/tests/run_isolated_test.py
,
May 17 2018
,
May 22 2018
There's still some refactoring to do but the crux is done. Rest is tracked as issue 844151 . |
||||
►
Sign in to add a comment |
||||
Comment 1 by mar...@chromium.org
, Feb 18 2018