move named cache directories instead of doing symlinks |
|||
Issue descriptioninstead of doing symlinks for named caches, move name cache dirs to the rundir before running a task, and then move them back. If bot dies and we don't move the dir back, that's ok because the probability of cache getting corrupted is higher when bot dies, than when it does not. for the reasons why, see bug 706224, #66
,
May 9 2017
,
May 9 2017
,
May 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/d1ab255eac4e9ef88d437d476a586d22cda15952 commit d1ab255eac4e9ef88d437d476a586d22cda15952 Author: nodir <nodir@chromium.org> Date: Wed May 10 19:18:44 2017 named caches: move instead of symlinking Moving named caches is better than symlinking because - C++ compiler used in swarming tasks reads symlinks => compiler output contains absolute paths to named caches => compiler cache key contains CWD => compiler cache is useless - if a bot dies, there is a higher chance that a cache is corrupted => better to discard the cache by not moving it back - simpler Move named cache directories to the task rundir before running a task, and move them back when task completes (and bot did not die). Revisit NamedCacheManager design, simplify its API. install() leaves the cache manager in consistent state: the entries are removed from the cache. uninstall() puts them back. R=maruel@chromium.org BUG= 715326 Review-Url: https://codereview.chromium.org/2866283002 [modify] https://crrev.com/d1ab255eac4e9ef88d437d476a586d22cda15952/client/named_cache.py [modify] https://crrev.com/d1ab255eac4e9ef88d437d476a586d22cda15952/client/run_isolated.py [modify] https://crrev.com/d1ab255eac4e9ef88d437d476a586d22cda15952/client/tests/named_cache_test.py [modify] https://crrev.com/d1ab255eac4e9ef88d437d476a586d22cda15952/client/tests/run_isolated_test.py
,
May 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/62d13e1c00497d006173c4d9cb496098284caf9a commit 62d13e1c00497d006173c4d9cb496098284caf9a Author: maruel <maruel@chromium.org> Date: Thu May 11 17:28:22 2017 Revert "named caches: move instead of symlinking" This reverts commit d1ab255eac4e9ef88d437d476a586d22cda15952. Caused failures in task_runner_test.py and local_smoke_test.py. This passed the presubmit because the CL only touches client/, so the Swarming server side tests were bypassed. TBR=nodir@chromium.org BUG= 715326 Review-Url: https://codereview.chromium.org/2875113002 [modify] https://crrev.com/62d13e1c00497d006173c4d9cb496098284caf9a/client/named_cache.py [modify] https://crrev.com/62d13e1c00497d006173c4d9cb496098284caf9a/client/run_isolated.py [modify] https://crrev.com/62d13e1c00497d006173c4d9cb496098284caf9a/client/tests/named_cache_test.py [modify] https://crrev.com/62d13e1c00497d006173c4d9cb496098284caf9a/client/tests/run_isolated_test.py
,
May 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/b551d1cc05919318aa1d643cfa174a2e2c451ae8 commit b551d1cc05919318aa1d643cfa174a2e2c451ae8 Author: nodir <nodir@chromium.org> Date: Thu May 11 20:21:53 2017 Reland "named caches: move instead of symlinking" This relands d1ab255eac4e9ef88d437d476a586d22cda15952 with fixed tests. BUG= 715326 Review-Url: https://codereview.chromium.org/2877483004 [modify] https://crrev.com/b551d1cc05919318aa1d643cfa174a2e2c451ae8/appengine/swarming/local_smoke_test.py [modify] https://crrev.com/b551d1cc05919318aa1d643cfa174a2e2c451ae8/appengine/swarming/swarming_bot/bot_code/task_runner_test.py [modify] https://crrev.com/b551d1cc05919318aa1d643cfa174a2e2c451ae8/client/named_cache.py [modify] https://crrev.com/b551d1cc05919318aa1d643cfa174a2e2c451ae8/client/run_isolated.py [modify] https://crrev.com/b551d1cc05919318aa1d643cfa174a2e2c451ae8/client/tests/named_cache_test.py [modify] https://crrev.com/b551d1cc05919318aa1d643cfa174a2e2c451ae8/client/tests/run_isolated_test.py
,
May 12 2017
phosek@, fyi this was deployed. cache directories are now real dirs, not symlinks |
|||
►
Sign in to add a comment |
|||
Comment 1 by mar...@chromium.org
, Apr 26 2017Owner: mar...@chromium.org
Status: Assigned (was: Untriaged)