New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 922722 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Jan 18
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug-Regression



Sign in to add a comment

Any build on win-46-m0 on Win64_trunk official.desktop.continuous waterfall fails due to "Exception steps exception bot_update exception uncaught exception failed failure reason"

Project Member Reported by pbommana@google.com, Jan 16 (6 days ago)

Issue description

Any triggered build on build slave "win-46-m0" is failing due to "Exception steps exception bot_update exception uncaught exception failed failure reason".


 

Comment 1 by pbommana@google.com, Jan 16 (6 days ago)

Summary: Any build on win-46-m0 on Win64_trunk official.desktop.continuous waterfall fails due to "Exception steps exception bot_update exception uncaught exception failed failure reason" (was: Exception steps exception bot_update exception uncaught exception failed failure reason on Win64_trunk official.desktop.continuous waterfall.)

Comment 2 by pbommana@google.com, Jan 16 (6 days ago)

Labels: -Pri-2 ReleaseBlock-Dev Pri-1
Note : reason to set the priority-1 and dev blocker, since any build on this slave so far is failing and we have a planned dev release tomorrow.

Comment 3 by pbommana@google.com, Jan 16 (6 days ago)

Cc: sergeybe...@chromium.org

Comment 4 by sergeybe...@chromium.org, Jan 16 (6 days ago)

Components: -Infra>Labs Infra
Labels: Infra-Troopers
Please file these with go/bug-a-trooper to get faster triage.

Comment 5 by sergeybe...@chromium.org, Jan 16 (6 days ago)

Will look soon (another fire right now)

Comment 6 by sergeybe...@chromium.org, Jan 16 (6 days ago)

Could you provide a direct link to the build?

Comment 8 by tandrii@chromium.org, Jan 17 (6 days ago)

Error: 

[0:00:28] Retrying rename [C:\b\c\git_cache\_cache_tmpdlrmdgchromium.googlesource.com-angle-angle] => [C:\b\c\git_cache\chromium.googlesource.com-angle-angle] in 0.25 second(s) (1 / 10 attempts): [Error 5] Access is denied
[0:00:28] Retrying rename [C:\b\c\git_cache\_cache_tmpdlrmdgchromium.googlesource.com-angle-angle] => [C:\b\c\git_cache\chromium.googlesource.com-angle-angle] in 0.50 second(s) (2 / 10 attempts): [Error 5] Access is denied
[0:00:28] Retrying rename [C:\b\c\git_cache\_cache_tmpdlrmdgchromium.googlesource.com-angle-angle] => [C:\b\c\git_cache\chromium.googlesource.com-angle-angle] in 1.00 second(s) (3 / 10 attempts): [Error 5] Access is denied
[0:00:29] Retrying rename [C:\b\c\git_cache\_cache_tmpdlrmdgchromium.googlesource.com-angle-angle] => [C:\b\c\git_cache\chromium.googlesource.com-angle-angle] in 2.00 second(s) (4 / 10 attempts): [Error 5] Access is denied
[0:00:31] Retrying rename [C:\b\c\git_cache\_cache_tmpdlrmdgchromium.googlesource.com-angle-angle] => [C:\b\c\git_cache\chromium.googlesource.com-angle-angle] in 4.00 second(s) (5 / 10 attempts): [Error 5] Access is denied
[0:00:36] Retrying rename [C:\b\c\git_cache\_cache_tmpdlrmdgchromium.googlesource.com-angle-angle] => [C:\b\c\git_cache\chromium.googlesource.com-angle-angle] in 8.00 second(s) (6 / 10 attempts): [Error 5] Access is denied
[0:00:44] Retrying rename [C:\b\c\git_cache\_cache_tmpdlrmdgchromium.googlesource.com-angle-angle] => [C:\b\c\git_cache\chromium.googlesource.com-angle-angle] in 16.00 second(s) (7 / 10 attempts): [Error 5] Access is denied
[0:01:00] Retrying rename [C:\b\c\git_cache\_cache_tmpdlrmdgchromium.googlesource.com-angle-angle] => [C:\b\c\git_cache\chromium.googlesource.com-angle-angle] in 32.00 second(s) (8 / 10 attempts): [Error 183] Cannot create a file when that file already exists
[0:01:32] Retrying rename [C:\b\c\git_cache\_cache_tmpdlrmdgchromium.googlesource.com-angle-angle] => [C:\b\c\git_cache\chromium.googlesource.com-angle-angle] in 64.00 second(s) (9 / 10 attempts): [Error 183] Cannot create a file when that file already exists
----------------------------------------
Traceback (most recent call last):
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\metrics.py", line 262, in print_notice_and_exit
    yield
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\gclient.py", line 3107, in <module>
    sys.exit(main(sys.argv[1:]))
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\gclient.py", line 3093, in main
    return dispatcher.execute(OptionParser(), argv)
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\subcommand.py", line 252, in execute
    return command(parser, args[1:])
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\gclient.py", line 2672, in CMDsync
    ret = client.RunOnDeps('update', args)
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\gclient.py", line 1705, in RunOnDeps
    patch_refs=patch_refs, target_branches=target_branches)
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\gclient_utils.py", line 1075, in run
    self.item.run(*self.args, **self.kwargs)
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\gclient.py", line 897, in run
    file_list)
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\gclient_scm.py", line 129, in RunCommand
    return getattr(self, command)(options, args, file_list)
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\gclient_scm.py", line 545, in update
    self._UpdateMirrorIfNotContains(mirror, options, rev_type, revision)
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\gclient_scm.py", line 979, in _UpdateMirrorIfNotContains
    lock_timeout=getattr(options, 'lock_timeout', 0))
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\git_cache.py", line 567, in populate
    self.Rename(tempdir, self.mirror_path)
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\git_cache.py", line 301, in Rename
    printerr=self.print)
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\git_cache.py", line 70, in exponential_backoff_retry
    return fn()
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\git_cache.py", line 298, in <lambda>
    lambda: os.rename(src, dst),
WindowsError: [Error 183] Cannot create a file when that file already exists
===Failed in 2.6 mins of C:\b\depot_tools\win_tools-2_7_6_bin\python\bin\python.exe -u C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\gclient.py sync --verbose --reset --force --ignore_locks --output-json c:\users\chrome~1\appdata\local\temp\tmph1y1cf.json --nohooks --noprehooks --delete_unversioned_trees --break_repo_locks --revision src@unmanaged ===

Traceback (most recent call last):
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\recipes\recipe_modules\bot_update\resources\bot_update.py", line 1181, in <module>
    sys.exit(main())
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\recipes\recipe_modules\bot_update\resources\bot_update.py", line 1165, in main
    checkout(options, git_slns, specs, revisions, step_text)
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\recipes\recipe_modules\bot_update\resources\bot_update.py", line 1075, in checkout
    gclient_output = ensure_checkout(**checkout_parameters)
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\recipes\recipe_modules\bot_update\resources\bot_update.py", line 860, in ensure_checkout
    gerrit_rebase_patch_ref)
  File "C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\recipes\recipe_modules\bot_update\resources\bot_update.py", line 385, in gclient_sync
    raise GclientSyncFailed(e.message, e.code, e.output)
__main__.GclientSyncFailed: C:\b\depot_tools\win_tools-2_7_6_bin\python\bin\python.exe -u C:\b\rr\tmpckh7gp\rw\checkout\recipes\.recipe_deps\depot_tools\gclient.py sync --verbose --reset --force --ignore_locks --output-json c:\users\chrome~1\appdata\local\temp\tmph1y1cf.json --nohooks --noprehooks --delete_unversioned_trees --break_repo_locks --revision src@unmanaged failed with code 1 in C:\b\c\b\win64_trunk.
step returned non-zero exit code: 1


Looks like Windows bot's filesystem got into weird state.

Comment 9 by sergeybe...@chromium.org, Jan 17 (6 days ago)

Owner: sergeybe...@chromium.org
Status: Assigned (was: Untriaged)
Indeed. And it's just a single machine, so I'll try to clobber it, maybe it'll be enough.

Comment 10 by sergeybe...@chromium.org, Jan 17 (6 days ago)

Clobbering now. Any new build should already use a fresh directory.

Comment 11 by sergeybe...@chromium.org, Jan 17 (6 days ago)

Done clobbering. Let's see if it works when the bot picks up another build.

Comment 12 by ajha@chromium.org, Jan 17 (5 days ago)

Just to update, still seeing consistent Win64_trunk failure on official.desktop.continuous builder.

Link to the latest builder where failure is seen:

https://uberchromegw.corp.google.com/i/official.desktop.continuous/builders/win64%20trunk/builds/50156

Link to the log:
================
https://logs.chromium.org/logs/chrome/bb/official.desktop.continuous/win64_trunk/50156/+/recipes/steps/bot_update/0/stdout.

Comment 13 by sergeybe...@chromium.org, Jan 17 (5 days ago)

Owner: jbudorick@chromium.org
Handing off to the next trooper (I won't realistically be able to handle this in time).

Comment 14 by d...@chromium.org, Jan 17 (5 days ago)

Cc: -sergeybe...@chromium.org jbudorick@chromium.org
Owner: d...@chromium.org
Something screwy the filesystem in general on this machine. I'll give it a redploy from scratch.

Comment 16 by d...@chromium.org, Jan 18 (4 days ago)

Status: Fixed (was: Assigned)
Bot is now green after a reinstall and redeploy.

Sign in to add a comment