Passing this to troopers, as this seems to be a continuation of issue 621917.

Hm, these are mostly chromeos (+dnj) and after looking at a few of them it's always runhooks which failed, which isn't really gclient.

Marking as available. I don't think has happened recently....

Sorry to keep bouncing this, I chatted with my team and we're not familiar with runhooks either -dnj, any ideas here? 

There are two failures in the recent set:

== GOMA ==
10:15:26: WARNING: Clearing the SDK cache.
10:15:39: ERROR: cros chrome-sdk failed before completing.
10:15:39: ERROR: return code: 1; command: python2 goma_ctl.py ensure_start
creating crash dump dir (/tmp/goma_crash.chrome-bot).
creating cache dir (/tmp/goma_cache.chrome-bot).
Traceback (most recent call last):
  File "goma_ctl.py", line 2442, in <module>
    sys.exit(main())
  File "goma_ctl.py", line 2437, in main
    goma.Dispatch(sys.argv[1:])
  File "goma_ctl.py", line 1082, in Dispatch
    self._action_mappings.get(args[0], self._DefaultAction)()
  File "goma_ctl.py", line 677, in _EnsureStartCompilerProxy
    self._GenericStartCompilerProxy(ensure=True)
  File "goma_ctl.py", line 671, in _GenericStartCompilerProxy
    raise Error('Failed to start compiler_proxy successfully.')
__main__.Error: Failed to start compiler_proxy successfully.

The CrOS part doesn't look too interesting - it's bootstrapping a "goma_ctl" command. This isn't fundamentally broken, as other builds on the same builder work just fine.

It's failing on "goma_ctl.py ensure start". I don't see any information that suggests this is CrOS-specific, nor do I see any information indicating why the Goma proxy didn't start. I'll add +ukai@ for more insight here, since I don't know how to dig into this.

== src/tools/clang/scripts/update.py ==
Downloading prebuilt clang
Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Mac/clang-277962-1.tgz Traceback (most recent call last):
  File "src/tools/clang/scripts/update.py", line 907, in <module>
    sys.exit(main())
  File "src/tools/clang/scripts/update.py", line 903, in main
    return UpdateClang(args)
  File "src/tools/clang/scripts/update.py", line 425, in UpdateClang
    DownloadAndUnpack(cds_full_url, LLVM_BUILD_DIR)
  File "src/tools/clang/scripts/update.py", line 142, in DownloadAndUnpack
    DownloadUrl(url, f)
  File "src/tools/clang/scripts/update.py", line 109, in DownloadUrl
    chunk = response.read(CHUNK_SIZE)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 567, in read
    s = self.fp.read(amt)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
socket.error: [Errno 54] Connection reset by peer
Error: Command '/usr/bin/python src/tools/clang/scripts/update.py --if-needed' returned non-zero exit status 1 in /b/c/b/mac

This looks like network flake. Solution would be to wrap this in a retry loop so we're more resilient to network flake.

not sure, but guess similar issue with https://bugs.chromium.org/p/chromium/issues/detail?id=642250?
(system clock drift to fail oauth2)

anyway, if update.py also fails, it might be network issue, rather than goma issue.

This might be known issue, which has been filed several times before.  I believe goma team can do nothing on this issue.
https://bugs.chromium.org/p/chromium/issues/list?can=1&q=%22gclient+%28with+patch%29%22+is+flaky&colspec=ID+Pri+M+Stars+ReleaseBlock+Component+Status+Owner+Summary+OS+Modified&x=m&y=releaseblock&cells=ids

Detected 4 new flakes for test/step "gclient (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRnY2xpZW50ICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Trooper Bug Queue (unless already there).

Do this step need to execute compiler_proxy by itself?
compile step seems to restart compiler_proxy, again.
I believe we should avoid executing goma_ctl.py in gclient runhook.  (Let me try to disable it like not running gyp for gclient runhook.)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/697e77eade8bf2165e6757b81d571a1b9ef93e1f

commit 697e77eade8bf2165e6757b81d571a1b9ef93e1f
Author: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Date: Tue Sep 06 07:10:07 2016

cros_chrome_sdk: a flag to skip starting goma in cros_chrome_sdk.

Currently, gclient runhook step for chromeos execute goma_ctl.py
ensure_start. e.g. builder chromeos_daisy_chromium_compile_only_ng
It make it impossible for us to investigate the failure reason of
goma_ctl.py ensure_start there, and also it is redundant since
goma_ctl.py restart is called compile.py later.
Let me add a flag to skip starting goma.

TEST=cros chrome-sdk --board=amd64-generic --nostart-goma
BUG= chromium:639432 

Change-Id: Ibd533d1e3dd1f77fa22a4401406c7337ef868c80
Reviewed-on: https://chromium-review.googlesource.com/381090
Commit-Ready: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Tested-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/697e77eade8bf2165e6757b81d571a1b9ef93e1f/cli/cros/cros_chrome_sdk.py

Detected 4 new flakes for test/step "gclient (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRnY2xpZW50ICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Note: #15 is not caused by goma but bad network?  curl seems to fail.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/91b87a011268d84d3d9ebb4e6bc016d769871425

commit 91b87a011268d84d3d9ebb4e6bc016d769871425
Author: yyanagisawa <yyanagisawa@chromium.org>
Date: Thu Sep 08 02:33:38 2016

Roll src/third_party/chromite/ 7381c2dc..697e77ea (2 commits).

https://chromium.googlesource.com/chromiumos/chromite.git/+log/7381c2dc..697e77ea

$ git log 7381c2dc..697e77ea --date=short --no-merges --format='%ad %ae %s'
2016-09-06 yyanagisawa@chromium.org cros_chrome_sdk: a flag to skip starting goma in cros_chrome_sdk.
2016-08-31 yunlian@chromium.org chromite: add peppy to the chromiumos-sdk bot

BUG= 639432 

Review-Url: https://codereview.chromium.org/2319513002
Cr-Commit-Position: refs/heads/master@{#417172}

[modify] https://crrev.com/91b87a011268d84d3d9ebb4e6bc016d769871425/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/91b87a011268d84d3d9ebb4e6bc016d769871425

commit 91b87a011268d84d3d9ebb4e6bc016d769871425
Author: yyanagisawa <yyanagisawa@chromium.org>
Date: Thu Sep 08 02:33:38 2016

Roll src/third_party/chromite/ 7381c2dc..697e77ea (2 commits).

https://chromium.googlesource.com/chromiumos/chromite.git/+log/7381c2dc..697e77ea

$ git log 7381c2dc..697e77ea --date=short --no-merges --format='%ad %ae %s'
2016-09-06 yyanagisawa@chromium.org cros_chrome_sdk: a flag to skip starting goma in cros_chrome_sdk.
2016-08-31 yunlian@chromium.org chromite: add peppy to the chromiumos-sdk bot

BUG= 639432 

Review-Url: https://codereview.chromium.org/2319513002
Cr-Commit-Position: refs/heads/master@{#417172}

[modify] https://crrev.com/91b87a011268d84d3d9ebb4e6bc016d769871425/DEPS

Detected 4 new flakes for test/step "gclient (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRnY2xpZW50ICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592

commit 60507ea6d87dfbee2b248bcdf3a90d4fa84c7592
Author: yyanagisawa <yyanagisawa@chromium.org>
Date: Tue Sep 13 02:51:17 2016

Do not start goma in ChromeOS gclient runhook if gomadir is given.

If gomadir is given, we are very sure we will start goma compiler_proxy
in compile step.  Let me avoid starting goma compiler_proxy in
runhook step, which is redundant and make it difficult for us to
investigate the issue.

BUG= 639432 

Review-Url: https://codereview.chromium.org/2315653002

[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipe_modules/chromium/api.py
[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipes/chromium.expected/full_chromium_chromiumos_ChromiumOS_amd64_generic_Compile.json
[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipes/chromium.expected/full_chromium_chromiumos_ChromiumOS_daisy_Compile.json
[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipes/chromium.expected/full_chromium_chromiumos_ChromiumOS_x86_generic_Compile.json
[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipes/chromium_trybot.expected/chromeos_analyze.json
[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipes/chromium_trybot.expected/full_tryserver_chromium_linux_chromeos_amd64_generic_chromium_compile_only_ng.json
[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipes/chromium_trybot.expected/full_tryserver_chromium_linux_chromeos_daisy_chromium_compile_only_ng.json
[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipes/chromium_trybot.expected/full_tryserver_chromium_linux_chromeos_x86_generic_chromium_compile_only_ng.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592

commit 60507ea6d87dfbee2b248bcdf3a90d4fa84c7592
Author: yyanagisawa <yyanagisawa@chromium.org>
Date: Tue Sep 13 02:51:17 2016

Do not start goma in ChromeOS gclient runhook if gomadir is given.

If gomadir is given, we are very sure we will start goma compiler_proxy
in compile step.  Let me avoid starting goma compiler_proxy in
runhook step, which is redundant and make it difficult for us to
investigate the issue.

BUG= 639432 

Review-Url: https://codereview.chromium.org/2315653002

[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipe_modules/chromium/api.py
[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipes/chromium.expected/full_chromium_chromiumos_ChromiumOS_amd64_generic_Compile.json
[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipes/chromium.expected/full_chromium_chromiumos_ChromiumOS_daisy_Compile.json
[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipes/chromium.expected/full_chromium_chromiumos_ChromiumOS_x86_generic_Compile.json
[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipes/chromium_trybot.expected/chromeos_analyze.json
[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipes/chromium_trybot.expected/full_tryserver_chromium_linux_chromeos_amd64_generic_chromium_compile_only_ng.json
[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipes/chromium_trybot.expected/full_tryserver_chromium_linux_chromeos_daisy_chromium_compile_only_ng.json
[modify] https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592/scripts/slave/recipes/chromium_trybot.expected/full_tryserver_chromium_linux_chromeos_x86_generic_chromium_compile_only_ng.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/691f8ea26cb52afcee385fc4a4bbb54c17bafb1b

commit 691f8ea26cb52afcee385fc4a4bbb54c17bafb1b
Author: recipe-roller <recipe-roller@chromium.org>
Date: Tue Sep 13 03:00:15 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/4e9b84ac0426e09c5d4e612f88c0f659f4fe198c Use [CACHE]/cipd/goma for goma_dir in recipe_modules/filter/example.py (tikuta@chromium.org)
  https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592 Do not start goma in ChromeOS gclient runhook if gomadir is given. (yyanagisawa@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG= 639432 ,621828

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://codereview.chromium.org/2334093002
Cr-Commit-Position: refs/heads/master@{#418151}

[modify] https://crrev.com/691f8ea26cb52afcee385fc4a4bbb54c17bafb1b/infra/config/recipes.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/40ff2dabddc63302cd802685c3d37b4fcb01848f

commit 40ff2dabddc63302cd802685c3d37b4fcb01848f
Author: recipe-roller <recipe-roller@chromium.org>
Date: Tue Sep 13 03:15:10 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/4e9b84ac0426e09c5d4e612f88c0f659f4fe198c Use [CACHE]/cipd/goma for goma_dir in recipe_modules/filter/example.py (tikuta@chromium.org)
  https://crrev.com/60507ea6d87dfbee2b248bcdf3a90d4fa84c7592 Do not start goma in ChromeOS gclient runhook if gomadir is given. (yyanagisawa@chromium.org)
  https://crrev.com/a050d1bb1282da2e4fb25d322fc1fb657a1bda73 Use [CACHE]/cipd/goma for goma_dir in recipes/chromium_libfuzzer.py (tikuta@chromium.org)
  https://crrev.com/d6ecda1568798df2f2ec5e17d8f6571e38c6e8dd Use [CACHE]/cipd/goma for goma_dir in recipes/ozone_ecs.py (tikuta@chromium.org)
  https://crrev.com/bcdf12617db069e17f963d54e50d1afe44508911 Use [CACHE]/cipd/goma for goma_dir in recipes/chromium_codesearch.py (tikuta@chromium.org)
  https://crrev.com/21f9a00d15f981eced788c91ebc47b5371020dfc Use [CACHE]/cipd/goma for goma_dir in recipes/bisection/android_bisect.py (tikuta@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG= 639432 ,621828

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://codereview.chromium.org/2330353002

[modify] https://crrev.com/40ff2dabddc63302cd802685c3d37b4fcb01848f/infra/config/recipes.cfg

"gclient runhooks" step now won't start goma, and you won't see flakes caused by goma there.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/de5c811c6b00831eda151106cb9ca376b41c2f60

commit de5c811c6b00831eda151106cb9ca376b41c2f60
Author: recipe-roller <recipe-roller@chromium.org>
Date: Tue Sep 13 14:01:16 2016