New issue
Advanced search Search tips

Issue 715326 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: May 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Feature

Blocking:
issue 706224



Sign in to add a comment

move named cache directories instead of doing symlinks

Project Member Reported by no...@chromium.org, Apr 25 2017

Issue description

instead of doing symlinks for named caches, move name cache dirs to the rundir before running a task, and then move them back. If bot dies and we don't move the dir back, that's ok because the probability of cache getting corrupted is higher when bot dies, than when it does not.

for the reasons why, see bug 706224,  #66 
 

Comment 1 by mar...@chromium.org, Apr 26 2017

Cc: -mar...@chromium.org
Owner: mar...@chromium.org
Status: Assigned (was: Untriaged)
I'll give it a try and we'll see if it's problematic.

Comment 2 by no...@chromium.org, May 9 2017

Cc: mar...@chromium.org
Owner: no...@chromium.org
Status: Started (was: Assigned)
Project Member

Comment 4 by bugdroid1@chromium.org, May 10 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/d1ab255eac4e9ef88d437d476a586d22cda15952

commit d1ab255eac4e9ef88d437d476a586d22cda15952
Author: nodir <nodir@chromium.org>
Date: Wed May 10 19:18:44 2017

named caches: move instead of symlinking

Moving named caches is better than symlinking because
- C++ compiler used in swarming tasks reads symlinks =>
  compiler output contains absolute paths to named caches =>
  compiler cache key contains CWD =>
  compiler cache is useless
- if a bot dies, there is a higher chance that a cache is corrupted =>
  better to discard the cache by not moving it back
- simpler

Move named cache directories to the task rundir before running a task,
and move them back when task completes (and bot did not die).

Revisit NamedCacheManager design, simplify its API.
install() leaves the cache manager in consistent state: the entries are removed
from the cache. uninstall() puts them back.

R=maruel@chromium.org
BUG= 715326 

Review-Url: https://codereview.chromium.org/2866283002

[modify] https://crrev.com/d1ab255eac4e9ef88d437d476a586d22cda15952/client/named_cache.py
[modify] https://crrev.com/d1ab255eac4e9ef88d437d476a586d22cda15952/client/run_isolated.py
[modify] https://crrev.com/d1ab255eac4e9ef88d437d476a586d22cda15952/client/tests/named_cache_test.py
[modify] https://crrev.com/d1ab255eac4e9ef88d437d476a586d22cda15952/client/tests/run_isolated_test.py

Project Member

Comment 5 by bugdroid1@chromium.org, May 11 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/62d13e1c00497d006173c4d9cb496098284caf9a

commit 62d13e1c00497d006173c4d9cb496098284caf9a
Author: maruel <maruel@chromium.org>
Date: Thu May 11 17:28:22 2017

Revert "named caches: move instead of symlinking"

This reverts commit d1ab255eac4e9ef88d437d476a586d22cda15952.

Caused failures in task_runner_test.py and local_smoke_test.py. This passed the
presubmit because the CL only touches client/, so the Swarming server side tests
were bypassed.

TBR=nodir@chromium.org
BUG= 715326 

Review-Url: https://codereview.chromium.org/2875113002

[modify] https://crrev.com/62d13e1c00497d006173c4d9cb496098284caf9a/client/named_cache.py
[modify] https://crrev.com/62d13e1c00497d006173c4d9cb496098284caf9a/client/run_isolated.py
[modify] https://crrev.com/62d13e1c00497d006173c4d9cb496098284caf9a/client/tests/named_cache_test.py
[modify] https://crrev.com/62d13e1c00497d006173c4d9cb496098284caf9a/client/tests/run_isolated_test.py

Comment 7 by no...@chromium.org, May 12 2017

Cc: phosek@chromium.org
Status: Verified (was: Started)
phosek@, fyi this was deployed. cache directories are now real dirs, not symlinks

Sign in to add a comment