New issue
Advanced search Search tips

Issue 649661 link

Starred by 4 users

Issue metadata

Status: Fixed
Owner: ----
Closed: Sep 2016
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 0
Type: Bug



Sign in to add a comment

All waterfall bots get an exception in the middle of their run

Project Member Reported by cwallez@chromium.org, Sep 23 2016

Issue description

Repro:
 - open https://build.chromium.org/p/chromium.gpu.fyi/waterfall?reload=120
 - Notice all the purple

A large number of bots from all waterfalls get an exception 5-25 minutes in the test run and restart.
 
Components: Infra>Labs
Seems it could be related to https://bugs.chromium.org/p/chromium/issues/detail?id=649603 as the purpleness started shortly after the tree was reopened.

Comment 2 by emso@chromium.org, Sep 23 2016

Yes, this is related to the earlier outage today which required a reboot of Windows bots. Some of those bots where interrupted mid-run. The interruption may have cause a corrupted cache on some bots.

Comment 3 by emso@chromium.org, Sep 23 2016

A fix which will cleanup the cache via puppet is on the way.
Project Member

Comment 4 by bugdroid1@chromium.org, Sep 23 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/ec12dfff06569ca80821d218c979ca5507449e91

commit ec12dfff06569ca80821d218c979ca5507449e91
Author: Scott Lee <ddoman@chromium.org>
Date: Fri Sep 23 13:16:23 2016

Project Member

Comment 5 by bugdroid1@chromium.org, Sep 23 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/83413ccece3b5394b8afdf2f75524b2a45aed7b9

commit 83413ccece3b5394b8afdf2f75524b2a45aed7b9
Author: Scott Lee <ddoman@chromium.org>
Date: Fri Sep 23 13:21:37 2016

Project Member

Comment 6 by bugdroid1@chromium.org, Sep 23 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/1658f236570db75a67456579c6a9aacc190ae5f3

commit 1658f236570db75a67456579c6a9aacc190ae5f3
Author: Scott Lee <ddoman@chromium.org>
Date: Fri Sep 23 13:52:04 2016

Comment 7 by thakis@chromium.org, Sep 23 2016

Issue 649675 has been merged into this issue.
Project Member

Comment 8 by bugdroid1@chromium.org, Sep 23 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/f83b578740ccf790c99fc88d8745a6ce057d0390

commit f83b578740ccf790c99fc88d8745a6ce057d0390
Author: Scott Lee <ddoman@chromium.org>
Date: Fri Sep 23 15:33:50 2016

Comment 9 by thakis@chromium.org, Sep 23 2016

Status: Fixed (was: Available)
We think this is resolved.
The first one says

===backing off, sleeping for 273 secs===
===Running git.bat checkout --force 3fd37d926efd712c690acee0bafbf07fd27d519e (attempt #5)===
In directory: C:\b\c\b\Win7_Release__New_Intel_\src
Traceback (most recent call last):
  File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 1099, in <module>
    sys.exit(main())
  File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 1083, in main
    checkout(options, git_slns, specs, revisions, step_text, shallow)
  File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 991, in checkout
    gclient_output = ensure_checkout(**checkout_parameters)
  File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 728, in ensure_checkout
    git_ref = git_checkout(solutions, revisions, shallow, refs, git_cache_dir)
  File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 520, in git_checkout
    force_revision(sln_dir, revision)
  File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 455, in force_revision
    git('checkout', '--force', revision, cwd=folder_name)
  File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 226, in git
    return call(*cmd, **kwargs)
  File "C:\b\rr\tmpg61y9d\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 211, in call
    code, outval)
__main__.SubprocessFailed: git.bat checkout --force 3fd37d926efd712c690acee0bafbf07fd27d519e failed with code 128 in C:\b\c\b\Win7_Release__New_Intel_\src after 5 attempts.
fatal: Unable to create 'C:/b/c/b/Win7_Release__New_Intel_/src/.git/index.lock': File exists.


So a trooper needs to ssh in and remove that lock file.
The other one says something similar:

===backing off, sleeping for 285 secs===
===Running git.bat checkout --force ec9c7f30eb10ab3fc9ab6707f7b736446ab89aa1 (attempt #5)===
In directory: C:\b\c\b\Win7_Debug__New_Intel_\src
Traceback (most recent call last):
  File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 1099, in <module>
    sys.exit(main())
  File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 1083, in main
    checkout(options, git_slns, specs, revisions, step_text, shallow)
  File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 991, in checkout
fatal: Unable to create 'C:/b/c/b/Win7_Debug__New_Intel_/src/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
===Failed in 0.0 mins===

Something failed: git.bat checkout --force ec9c7f30eb10ab3fc9ab6707f7b736446ab89aa1 failed with code 128 in C:\b\c\b\Win7_Debug__New_Intel_\src after 5 attempts..
Ran 419.332000017 seconds past deadline. Aborting.
    gclient_output = ensure_checkout(**checkout_parameters)
  File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 728, in ensure_checkout
    git_ref = git_checkout(solutions, revisions, shallow, refs, git_cache_dir)
  File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 520, in git_checkout
    force_revision(sln_dir, revision)
  File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 455, in force_revision
    git('checkout', '--force', revision, cwd=folder_name)
  File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 226, in git
    return call(*cmd, **kwargs)
  File "C:\b\rr\tmp81cp3g\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 211, in call
    code, outval)
__main__.SubprocessFailed: git.bat checkout --force ec9c7f30eb10ab3fc9ab6707f7b736446ab89aa1 failed with code 128 in C:\b\c\b\Win7_Debug__New_Intel_\src after 5 attempts.


So file a ticket to remove the lockfile for each (at go/heytrooper)
Project Member

Comment 13 by bugdroid1@chromium.org, Sep 26 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/295b025b84135029f8df05bf253f80ad76812168

commit 295b025b84135029f8df05bf253f80ad76812168
Author: Scott Lee <ddoman@chromium.org>
Date: Mon Sep 26 03:25:46 2016

Components: -Infra
Labels: cit-pm

Sign in to add a comment