New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 855366 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: ----
Type: ----



Sign in to add a comment

The SyncChrome stage failed: : return code: 1; command: /b/c/cbuild/repository/chromite/bin/sync_chrome --reset --ignore_locks --internal --tag 69.0.3469.0 /b/c/cbuild/repository/.cache/distfiles/target-master/chrome

Project Member Reported by mojahsu@chromium.org, Jun 22 2018

Issue description

Many Chome PFQs failed with the error.

Link to build or pfq page.
https://luci-milo.appspot.com/buildbot/chromeos/peach_pit-chrome-pfq/5814
https://luci-milo.appspot.com/buildbot/chromeos/caroline-arcnext-chrome-pfq/601
https://luci-milo.appspot.com/buildbot/chromeos/kevin-arcnext-chrome-pfq/602

----------------------------------------
src/third_party/skia (ERROR)
----------------------------------------
[0:02:27] Started.
_____ src/third_party/skia at 61582510eeb8484ad3e12df97bf781a3129f60d0
[0:02:27] running "git cat-file -e 61582510eeb8484ad3e12df97bf781a3129f60d0^{commit}" in "/b/git-cache/skia.googlesource.com-skia"
skipping mirror update, it has rev=61582510eeb8484ad3e12df97bf781a3129f60d0 already
________ running 'git -c core.deltaBaseCacheLimit=2g clone --no-checkout --progress --shared --verbose /b/git-cache/skia.googlesource.com-skia /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/third_party/_gclient_skia_gSBcpz' in '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal'
[0:02:27] Cloning into '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/third_party/_gclient_skia_gSBcpz'...
[0:02:27] done.
________ running 'git -c core.deltaBaseCacheLimit=2g fetch origin --prune --verbose' in '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/third_party/skia'
[0:02:30] From /b/git-cache/skia.googlesource.com-skia
[0:02:30]  = [up to date]            master                      -> origin/master
[0:02:30]  = [up to date]            android/next-release        -> origin/android/next-release
[0:02:30]  = [up to date]            android/o-dr1-release       -> origin/android/o-dr1-release
[0:02:30]  = [up to date]            android/o-mr1-release       -> origin/android/o-mr1-release
[0:02:30]  = [up to date]            android/oreo                -> origin/android/oreo
[0:02:30]  = [up to date]            chrome/3468                 -> origin/chrome/3468
[0:02:30]  = [up to date]            chrome/3469                 -> origin/chrome/3469
[0:02:30]  = [up to date]            chrome/m39                  -> origin/chrome/m39
[0:02:30]  = [up to date]            chrome/m40                  -> origin/chrome/m40
[0:02:30]  = [up to date]            chrome/m41                  -> origin/chrome/m41
[0:02:30]  = [up to date]            chrome/m42                  -> origin/chrome/m42
[0:02:30]  = [up to date]            chrome/m43                  -> origin/chrome/m43
[0:02:30]  = [up to date]            chrome/m44                  -> origin/chrome/m44
[0:02:30]  = [up to date]            chrome/m45                  -> origin/chrome/m45
[0:02:30]  = [up to date]            chrome/m46                  -> origin/chrome/m46
[0:02:30]  = [up to date]            chrome/m47                  -> origin/chrome/m47
[0:02:30]  = [up to date]            chrome/m48                  -> origin/chrome/m48
[0:02:30]  = [up to date]            chrome/m49                  -> origin/chrome/m49
[0:02:30]  = [up to date]            chrome/m50                  -> origin/chrome/m50
[0:02:30]  = [up to date]            chrome/m51                  -> origin/chrome/m51
[0:02:30]  = [up to date]            chrome/m52                  -> origin/chrome/m52
[0:02:30]  = [up to date]            chrome/m53                  -> origin/chrome/m53
[0:02:30]  = [up to date]            chrome/m54                  -> origin/chrome/m54
[0:02:30]  = [up to date]            chrome/m55                  -> origin/chrome/m55
[0:02:30]  = [up to date]            chrome/m56                  -> origin/chrome/m56
[0:02:30]  = [up to date]            chrome/m57                  -> origin/chrome/m57
[0:02:30]  = [up to date]            chrome/m58                  -> origin/chrome/m58
[0:02:30]  = [up to date]            chrome/m59                  -> origin/chrome/m59
[0:02:30]  = [up to date]            chrome/m60                  -> origin/chrome/m60
[0:02:31]  = [up to date]            chrome/m61                  -> origin/chrome/m61
[0:02:31]  = [up to date]            chrome/m62                  -> origin/chrome/m62
[0:02:31]  = [up to date]            chrome/m63                  -> origin/chrome/m63
[0:02:31]  = [up to date]            chrome/m64                  -> origin/chrome/m64
[0:02:31]  = [up to date]            chrome/m65                  -> origin/chrome/m65
[0:02:31]  = [up to date]            chrome/m66                  -> origin/chrome/m66
[0:02:31]  = [up to date]            chrome/m67                  -> origin/chrome/m67
[0:02:31]  = [up to date]            chrome/m68                  -> origin/chrome/m68
[0:02:31]  = [up to date]            infra/config                -> origin/infra/config
[0:02:31]  = [up to date]            skqp/dev                    -> origin/skqp/dev
[0:02:31]  = [up to date]            skqp/release                -> origin/skqp/release
[0:02:31]  * [new ref]               refs/branch-heads/m20_1132  -> branch-heads/m20_1132
[0:02:31]  * [new ref]               refs/branch-heads/m21_1180  -> branch-heads/m21_1180
[0:02:31]  * [new ref]               refs/branch-heads/m22_1229  -> branch-heads/m22_1229
[0:02:31]  * [new ref]               refs/branch-heads/m23_1271  -> branch-heads/m23_1271
[0:02:31]  * [new ref]               refs/branch-heads/m24_1312  -> branch-heads/m24_1312
[0:02:31]  * [new ref]               refs/branch-heads/m25_1364  -> branch-heads/m25_1364
[0:02:31]  * [new ref]               refs/branch-heads/m26_1410  -> branch-heads/m26_1410
[0:02:31]  * [new ref]               refs/branch-heads/m26_1411  -> branch-heads/m26_1411
[0:02:31]  * [new ref]               refs/branch-heads/m27_1453  -> branch-heads/m27_1453
[0:02:31]  * [new ref]               refs/branch-heads/m28_1500  -> branch-heads/m28_1500
[0:02:31]  * [new ref]               refs/branch-heads/m29_1547  -> branch-heads/m29_1547
[0:02:31]  * [new ref]               refs/branch-heads/m30_1599  -> branch-heads/m30_1599
[0:02:31]  * [new ref]               refs/branch-heads/m31_1650  -> branch-heads/m31_1650
[0:02:31]  * [new ref]               refs/branch-heads/m32_1700  -> branch-heads/m32_1700
[0:02:31]  * [new ref]               refs/branch-heads/m33_1750  -> branch-heads/m33_1750
[0:02:31]  * [new ref]               refs/branch-heads/m35_13966 -> branch-heads/m35_13966
[0:02:31]  * [new ref]               refs/branch-heads/master    -> branch-heads/master
[0:02:31]  * [new ref]               refs/branch-heads/trunk     -> branch-heads/trunk
[0:02:31]  = [up to date]            chrome/m37_2062             -> chrome/m37_2062
[0:02:31]  = [up to date]            chrome/m37_2062_94          -> chrome/m37_2062_94
[0:02:31]  = [up to date]            chrome/m38_2125             -> chrome/m38_2125
[0:02:31]  = [up to date]            chrome/m39_2138             -> chrome/m39_2138
[0:02:32] Checked out 61582510eeb8484ad3e12df97bf781a3129f60d0 to a detached HEAD. Before making any commits
in this repo, you should use 'git checkout <branch>' to switch to
an existing branch or use 'git checkout origin -b <branch>' to
create a new branch for your work.
----------------------------------------
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromium/tools/depot_tools/gclient.py", line 3184, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/b/c/cbuild/repository/chromium/tools/depot_tools/gclient.py", line 3170, in main
    return dispatcher.execute(OptionParser(), argv)
  File "/b/c/cbuild/repository/chromium/tools/depot_tools/subcommand.py", line 252, in execute
    return command(parser, args[1:])
  File "/b/c/cbuild/repository/chromium/tools/depot_tools/gclient.py", line 2812, in CMDsync
    ret = client.RunOnDeps('update', args)
  File "/b/c/cbuild/repository/chromium/tools/depot_tools/gclient.py", line 1701, in RunOnDeps
    patch_refs=patch_refs)
  File "/b/c/cbuild/repository/chromium/tools/depot_tools/gclient_utils.py", line 1075, in run
    self.item.run(*self.args, **self.kwargs)
  File "/b/c/cbuild/repository/chromium/tools/depot_tools/gclient.py", line 1018, in run
    self.ParseDepsFile(expand_vars=(command != 'flatten'))
  File "/b/c/cbuild/repository/chromium/tools/depot_tools/gclient.py", line 772, in ParseDepsFile
    filepath, self.get_vars())
  File "/b/c/cbuild/repository/chromium/tools/depot_tools/gclient_eval.py", line 366, in Parse
    return Exec(content, expand_vars, filename, vars_override)
  File "/b/c/cbuild/repository/chromium/tools/depot_tools/gclient_eval.py", line 340, in Exec
    return _GCLIENT_SCHEMA.validate(local_scope)
  File "/b/c/cbuild/repository/chromium/tools/depot_tools/third_party/schema/schema.py", line 304, in validate
    raise SchemaError([None] + x.autos, [e] + x.errors)
third_party.schema.schema.SchemaError: Wrong keys 'gclient_gn_args_from' in {'gclient_gn_args_from': 'src', 'recursedeps': ['common', '../src'], 'use_relative_paths': True, 'deps': <gclient_eval._NodeDict object at 0x7fdea3ab16d0>, 'vars': <gclient_eval._NodeDict object at 0x7fdea3ab18d0>}
sync_chrome: Unhandled exception:
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/bin/sync_chrome", line 169, in <module>
    DoMain()
  File "/b/c/cbuild/repository/chromite/bin/sync_chrome", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/c/cbuild/repository/chromite/lib/commandline.py", line 911, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/c/cbuild/repository/chromite/scripts/sync_chrome.py", line 76, in main
    SyncChrome(gclient_path, options)
  File "/b/c/cbuild/repository/chromite/scripts/sync_chrome.py", line 52, in SyncChrome
    ignore_locks=options.ignore_locks)
  File "/b/c/cbuild/repository/chromite/lib/gclient.py", line 264, in Sync
    return cros_build_lib.RunCommand(cmd, cwd=cwd, **run_args)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 665, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: /b/c/cbuild/repository/chromium/tools/depot_tools/gclient sync --with_branch_heads --with_tags --reset --force --delete_unversioned_trees --nohooks --verbose --ignore_locks
cmd=['/b/c/cbuild/repository/chromium/tools/depot_tools/gclient', 'sync', '--with_branch_heads', '--with_tags', '--reset', '--force', '--delete_unversioned_trees', '--nohooks', '--verbose', '--ignore_locks'], cwd=/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal
23:33:50: WARNING: Command failed with retriable error.
return code: 1; command: /b/c/cbuild/repository/chromite/bin/sync_chrome --reset --ignore_locks --internal --tag 69.0.3469.0 /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal
cmd=['/b/c/cbuild/repository/chromite/bin/sync_chrome', '--reset', '--ignore_locks', '--internal', '--tag', u'69.0.3469.0', '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal'], cwd=/b/c/cbuild/repository
23:33:50: ERROR: <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: 1; command: /b/c/cbuild/repository/chromite/bin/sync_chrome --reset --ignore_locks --internal --tag 69.0.3469.0 /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal
cmd=['/b/c/cbuild/repository/chromite/bin/sync_chrome', '--reset', '--ignore_locks', '--internal', '--tag', u'69.0.3469.0', '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal'], cwd=/b/c/cbuild/repository
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/lib/failures_lib.py", line 230, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/chrome_stages.py", line 113, in PerformStage
    useflags, **kwargs)
  File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 2882, in SyncChrome
    retry_util.RunCommandWithRetries(constants.SYNC_RETRIES, cmd, cwd=build_root)
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 358, in RunCommandWithRetries
    return RetryCommand(cros_build_lib.RunCommand, max_retry, *args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 330, in RetryCommand
    return GenericRetry(ShouldRetry, max_retry, functor, *args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 249, in GenericRetry
    return _run()
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 182, in _Wrapper
    ret = func(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 248, in _run
    return functor(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 665, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: /b/c/cbuild/repository/chromite/bin/sync_chrome --reset --ignore_locks --internal --tag 69.0.3469.0 /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal
cmd=['/b/c/cbuild/repository/chromite/bin/sync_chrome', '--reset', '--ignore_locks', '--internal', '--tag', u'69.0.3469.0', '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal'], cwd=/b/c/cbuild/repository

23:33:50: INFO: Translating result <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: 1; command: /b/c/cbuild/repository/chromite/bin/sync_chrome --reset --ignore_locks --internal --tag 69.0.3469.0 /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal
cmd=['/b/c/cbuild/repository/chromite/bin/sync_chrome', '--reset', '--ignore_locks', '--internal', '--tag', u'69.0.3469.0', '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal'], cwd=/b/c/cbuild/repository
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/lib/failures_lib.py", line 230, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/chrome_stages.py", line 113, in PerformStage
    useflags, **kwargs)
  File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 2882, in SyncChrome
    retry_util.RunCommandWithRetries(constants.SYNC_RETRIES, cmd, cwd=build_root)
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 358, in RunCommandWithRetries
    return RetryCommand(cros_build_lib.RunCommand, max_retry, *args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 330, in RetryCommand
    return GenericRetry(ShouldRetry, max_retry, functor, *args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 249, in GenericRetry
    return _run()
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 182, in _Wrapper
    ret = func(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 248, in _run
    return functor(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 665, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: /b/c/cbuild/repository/chromite/bin/sync_chrome --reset --ignore_locks --internal --tag 69.0.3469.0 /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal
cmd=['/b/c/cbuild/repository/chromite/bin/sync_chrome', '--reset', '--ignore_locks', '--internal', '--tag', u'69.0.3469.0', '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal'], cwd=/b/c/cbuild/repository
 to fail.
 
Owner: manucornet@chromium.org
Routing to the Chrome gardener
It looks like the pfq-informational builders have recovered, so, assuming the failure was the same (both the pfq and pfq-informational builders were failing in SyncChrome), the pfq should pick up the fix tonight.

(The SyncChrome output is enormous, so LogDog is still spooling output to my browser, and it will be challenging to compare the results at best, but I'll try to take a look when they finish).

I confirmed that the errors look the same, so hopefully this will be fixed tonight.

Comment 4 by x...@chromium.org, Jun 25 2018

Status: Fixed (was: Untriaged)
The error has gone.

Sign in to add a comment