New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 600947 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocking:
issue 584077
issue 595787



Sign in to add a comment

recipe-related bots can fail because of lock issues

Project Member Reported by phajdan.jr@chromium.org, Apr 6 2016

Issue description

See e.g. https://build.chromium.org/p/tryserver.infra/builders/Build%20Presubmit/builds/3599 :

** Presubmit ERRORS **
scripts/slave/unittests/recipe_lint_test.py (2.73s) failed
fatal: Unable to create '/b/build/slave/presubmit_build/build/build/scripts/slave/.recipe_deps/depot_tools/.git/index.lock': File exists.

https://luci-logdog.appspot.com/v/?s=bb%2Ftryserver.infra%2FBuild_Presubmit%2F3599%2F%2B%2Frecipes%2Fsteps%2Fpresubmit%2F0%2Fstdout

This contributes to the roller getting stuck, see  issue 600945  .
 
Labels: Infra Pri-1 Type-Bug
Summary: recipe-related bots can fail because of lock issues (was: recipe-related presubmit can fail because of lock issues)
This also affects other bots running recipes. We're seeing this more regularly on all kinds of bots in v8, e.g.:
https://build.chromium.org/p/tryserver.v8/builders/v8_linux_arm_rel_ng/builds/38/steps/steps/logs/stdio

Saved log: https://paste.googleplex.com/6139566461288448
Labels: Infra-Troopers
This slave seems to be completely broken because of this:
https://build.chromium.org/p/tryserver.v8/buildslaves/slave412-c4
Please respawn slave412-c4 to fix that slave for now, but the root cause seems to be deeper here...
Cc: iannucci@chromium.org martiniss@chromium.org
FTR, this has been happening for >2 months now for various slaves of mine, and even on my workstation it happens regularly (perhaps because I am a fan of Ctrl+C). I thought I had filed a bug about this already, but I can't find it aom.

Anyway, this is very similar to bot_update experience with locks, so perhaps the same solution can be adopted?

martiniss@ iannucci@ please fix this. Once the problem happens, the whole slave is effectively eternally broken. Even worse, it's RED, not PURPLE (bug:  http://crbug.com/603001 )

Comment 6 by pschm...@google.com, Apr 13 2016

Just refreshed slave412-c4.
Thanks!
Blocking: 595787
Owner: martiniss@chromium.org
Status: Assigned (was: Untriaged)
This ticket is about fixing the root cause now, I'm removing it from the trooper queue.

In the meantime please file separate tickets went bots are borked. I'll fix them manually.

Assigning to martiniss@, please take a look.
Labels: -Infra-Troopers
Status: Started (was: Assigned)
>:( i'll look at this
(mad that it's happening, not that I have to look at it)
Status: Fixed (was: Started)
I think this problem got fixed... recipe_lint_test needed to use a deps path, which I did in https://codereview.chromium.org/1889743002

If it comes up again, feel free to re-open.
Hmm, thanks, but I don't see how a normal bot running recipes would have been affected by this fix... I'll see if it happens again...
Status: Started (was: Fixed)
Ah ok, I haven't fixed everything; I read this bug too fast.

I've fixed the build presubmit AFAIK, but other stuff appears to have broken, and I'm not exactly sure why. Re-opening
Components: Infra>Platform>Recipes
Labels: -Infra-Recipes
Labels: -Infra
Status: Assigned (was: Started)
Status: Fixed (was: Assigned)
I believe I ended up fixing this, but didn't post an update to the bug :/ 

Sign in to add a comment