All waterfall bots get an exception in the middle of their run |
|||
Issue descriptionRepro: - open https://build.chromium.org/p/chromium.gpu.fyi/waterfall?reload=120 - Notice all the purple A large number of bots from all waterfalls get an exception 5-25 minutes in the test run and restart.
,
Sep 23 2016
Yes, this is related to the earlier outage today which required a reboot of Windows bots. Some of those bots where interrupted mid-run. The interruption may have cause a corrupted cache on some bots.
,
Sep 23 2016
A fix which will cleanup the cache via puppet is on the way.
,
Sep 23 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/infra/puppet/+/ec12dfff06569ca80821d218c979ca5507449e91 commit ec12dfff06569ca80821d218c979ca5507449e91 Author: Scott Lee <ddoman@chromium.org> Date: Fri Sep 23 13:16:23 2016
,
Sep 23 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/infra/puppet/+/83413ccece3b5394b8afdf2f75524b2a45aed7b9 commit 83413ccece3b5394b8afdf2f75524b2a45aed7b9 Author: Scott Lee <ddoman@chromium.org> Date: Fri Sep 23 13:21:37 2016
,
Sep 23 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/infra/puppet/+/1658f236570db75a67456579c6a9aacc190ae5f3 commit 1658f236570db75a67456579c6a9aacc190ae5f3 Author: Scott Lee <ddoman@chromium.org> Date: Fri Sep 23 13:52:04 2016
,
Sep 23 2016
Issue 649675 has been merged into this issue.
,
Sep 23 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/infra/puppet/+/f83b578740ccf790c99fc88d8745a6ce057d0390 commit f83b578740ccf790c99fc88d8745a6ce057d0390 Author: Scott Lee <ddoman@chromium.org> Date: Fri Sep 23 15:33:50 2016
,
Sep 23 2016
We think this is resolved.
,
Sep 23 2016
I see that https://build.chromium.org/p/chromium.gpu.fyi/builders/Win7%20Release%20%28New%20Intel%29 and https://build.chromium.org/p/chromium.gpu.fyi/builders/Win7%20Debug%20%28New%20Intel%29 are not alive yet. Is it the same issue?
,
Sep 23 2016
The first one says ===backing off, sleeping for 273 secs=== ===Running git.bat checkout --force 3fd37d926efd712c690acee0bafbf07fd27d519e (attempt #5)=== In directory: C:\b\c\b\Win7_Release__New_Intel_\src Traceback (most recent call last): File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 1099, in <module> sys.exit(main()) File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 1083, in main checkout(options, git_slns, specs, revisions, step_text, shallow) File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 991, in checkout gclient_output = ensure_checkout(**checkout_parameters) File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 728, in ensure_checkout git_ref = git_checkout(solutions, revisions, shallow, refs, git_cache_dir) File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 520, in git_checkout force_revision(sln_dir, revision) File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 455, in force_revision git('checkout', '--force', revision, cwd=folder_name) File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 226, in git return call(*cmd, **kwargs) File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 211, in call code, outval) __main__.SubprocessFailed: git.bat checkout --force 3fd37d926efd712c690acee0bafbf07fd27d519e failed with code 128 in C:\b\c\b\Win7_Release__New_Intel_\src after 5 attempts. fatal: Unable to create 'C:/b/c/b/Win7_Release__New_Intel_/src/.git/index.lock': File exists. So a trooper needs to ssh in and remove that lock file.
,
Sep 23 2016
The other one says something similar: ===backing off, sleeping for 285 secs=== ===Running git.bat checkout --force ec9c7f30eb10ab3fc9ab6707f7b736446ab89aa1 (attempt #5)=== In directory: C:\b\c\b\Win7_Debug__New_Intel_\src Traceback (most recent call last): File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 1099, in <module> sys.exit(main()) File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 1083, in main checkout(options, git_slns, specs, revisions, step_text, shallow) File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 991, in checkout fatal: Unable to create 'C:/b/c/b/Win7_Debug__New_Intel_/src/.git/index.lock': File exists. Another git process seems to be running in this repository, e.g. an editor opened by 'git commit'. Please make sure all processes are terminated then try again. If it still fails, a git process may have crashed in this repository earlier: remove the file manually to continue. ===Failed in 0.0 mins=== Something failed: git.bat checkout --force ec9c7f30eb10ab3fc9ab6707f7b736446ab89aa1 failed with code 128 in C:\b\c\b\Win7_Debug__New_Intel_\src after 5 attempts.. Ran 419.332000017 seconds past deadline. Aborting. gclient_output = ensure_checkout(**checkout_parameters) File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 728, in ensure_checkout git_ref = git_checkout(solutions, revisions, shallow, refs, git_cache_dir) File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 520, in git_checkout force_revision(sln_dir, revision) File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 455, in force_revision git('checkout', '--force', revision, cwd=folder_name) File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 226, in git return call(*cmd, **kwargs) File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 211, in call code, outval) __main__.SubprocessFailed: git.bat checkout --force ec9c7f30eb10ab3fc9ab6707f7b736446ab89aa1 failed with code 128 in C:\b\c\b\Win7_Debug__New_Intel_\src after 5 attempts. So file a ticket to remove the lockfile for each (at go/heytrooper)
,
Sep 26 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/infra/puppet/+/295b025b84135029f8df05bf253f80ad76812168 commit 295b025b84135029f8df05bf253f80ad76812168 Author: Scott Lee <ddoman@chromium.org> Date: Mon Sep 26 03:25:46 2016
,
Sep 27 2016
|
|||
►
Sign in to add a comment |
|||
Comment 1 by cwallez@chromium.org
, Sep 23 2016