recipe-related bots can fail because of lock issues |
||||||||||||
Issue descriptionSee e.g. https://build.chromium.org/p/tryserver.infra/builders/Build%20Presubmit/builds/3599 : ** Presubmit ERRORS ** scripts/slave/unittests/recipe_lint_test.py (2.73s) failed fatal: Unable to create '/b/build/slave/presubmit_build/build/build/scripts/slave/.recipe_deps/depot_tools/.git/index.lock': File exists. https://luci-logdog.appspot.com/v/?s=bb%2Ftryserver.infra%2FBuild_Presubmit%2F3599%2F%2B%2Frecipes%2Fsteps%2Fpresubmit%2F0%2Fstdout This contributes to the roller getting stuck, see issue 600945 .
,
Apr 13 2016
,
Apr 13 2016
This slave seems to be completely broken because of this: https://build.chromium.org/p/tryserver.v8/buildslaves/slave412-c4
,
Apr 13 2016
Please respawn slave412-c4 to fix that slave for now, but the root cause seems to be deeper here...
,
Apr 13 2016
FTR, this has been happening for >2 months now for various slaves of mine, and even on my workstation it happens regularly (perhaps because I am a fan of Ctrl+C). I thought I had filed a bug about this already, but I can't find it aom. Anyway, this is very similar to bot_update experience with locks, so perhaps the same solution can be adopted? martiniss@ iannucci@ please fix this. Once the problem happens, the whole slave is effectively eternally broken. Even worse, it's RED, not PURPLE (bug: http://crbug.com/603001 )
,
Apr 13 2016
Just refreshed slave412-c4.
,
Apr 13 2016
Thanks!
,
Apr 13 2016
This ticket is about fixing the root cause now, I'm removing it from the trooper queue. In the meantime please file separate tickets went bots are borked. I'll fix them manually. Assigning to martiniss@, please take a look.
,
Apr 13 2016
,
Apr 13 2016
>:( i'll look at this
,
Apr 13 2016
(mad that it's happening, not that I have to look at it)
,
Apr 15 2016
I think this problem got fixed... recipe_lint_test needed to use a deps path, which I did in https://codereview.chromium.org/1889743002 If it comes up again, feel free to re-open.
,
Apr 16 2016
Hmm, thanks, but I don't see how a normal bot running recipes would have been affected by this fix... I'll see if it happens again...
,
Apr 18 2016
Ah ok, I haven't fixed everything; I read this bug too fast. I've fixed the build presubmit AFAIK, but other stuff appears to have broken, and I'm not exactly sure why. Re-opening
,
Apr 26 2016
,
Apr 27 2016
,
Jun 27 2016
,
Jun 29 2016
I believe I ended up fixing this, but didn't post an update to the bug :/ |
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by machenb...@chromium.org
, Apr 13 2016Summary: recipe-related bots can fail because of lock issues (was: recipe-related presubmit can fail because of lock issues)