New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 597878 link

Starred by 0 users

Issue metadata

Status: Fixed
Owner: ----
Closed: Mar 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

"ensure_goma" is flaky

Project Member Reported by chromium...@appspot.gserviceaccount.com, Mar 25 2016

Issue description

"ensure_goma" is flaky.

This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is infrastructure-related, please add Infra-Troopers label and change issue status to Untriaged. When done, please remove the issue from Sheriff Bug Queue by removing the Sheriff-Chromium label.

We have detected 3 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyFgsSBUZsYWtlIgtlbnN1cmVfZ29tYQw.

Flaky tests should be disabled within 30 minutes unless culprit CL is found and reverted. Please see more details here: https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/sheriffing-bug-queues#triaging-auto-filed-flakiness-bugs

This flaky test/step was previously tracked in  issue 595922 .
 
Project Member

Comment 1 by chromium...@appspot.gserviceaccount.com, Mar 26 2016

Detected 3 new flakes for test/step "ensure_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyFgsSBUZsYWtlIgtlbnN1cmVfZ29tYQw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Comment 2 by sorin@chromium.org, Mar 28 2016

Labels: -Sheriff-Chromium Infra-Troopers
This appears to be an infra issue.

Comment 3 by d...@chromium.org, Mar 28 2016

Cc: phajdan.jr@chromium.org
+phajdan.jr@, author of "ensure_goma".

The same slave has run the same builder later and succeeded: https://build.chromium.org/p/tryserver.chromium.android/builders/android_chromium_gn_compile_rel/builds/41684

Perhaps this was either OAuth flake or something that needs to be put in a retry loop?

Comment 4 by ukai@chromium.org, Mar 29 2016

Cc: ukai@chromium.org
should we use git retry?

Comment 5 by d...@chromium.org, Mar 29 2016

That seems reasonable in general, but IIRC the "bad credentials" error would have not been retried by "git retry", as it's not known to be transient, nor should it be.
Project Member

Comment 6 by bugdroid1@chromium.org, Mar 29 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/995e5cb0f629c0c1072428ef978078a676a3047a

commit 995e5cb0f629c0c1072428ef978078a676a3047a
Author: ukai@chromium.org <ukai@chromium.org>
Date: Tue Mar 29 01:49:11 2016

ensure_goma: use git retry for clone and fetch

sometimes, git clone failed like
Cloning into '/b/build/slave/linux_chromium_gn_chromeos/build/src/build/goma/client'...
error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
Traceback (most recent call last):
  File "/b/build/scripts/slave/recipe_modules/goma/resources/ensure_goma.py", line 70, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/b/build/scripts/slave/recipe_modules/goma/resources/ensure_goma.py", line 47, in main
    subprocess.check_call(['git', 'clone', config['repo'], client_dir])
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'clone', 'https://chrome-internal.googlesource.com/chrome/tools/goma/linux.git', '/b/build/slave/linux_chromium_gn_chromeos/build/src/build/goma/client']' returned non-zero exit status 128
step returned non-zero exit code: 1

use git-retry in depot_tools to retry for such failures.

R=phajdan.jr@chromium.org, dnj@chromium.org
BUG= 597878 

Review URL: https://codereview.chromium.org/1835943004

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/build@299523 0039d316-1c4b-4281-b951-d872f2087c98

[modify] https://crrev.com/995e5cb0f629c0c1072428ef978078a676a3047a/scripts/slave/recipe_modules/goma/resources/ensure_goma.py

Comment 7 by d...@chromium.org, Mar 29 2016

Status: Fixed (was: Untriaged)
I'm going to mark this optimistically fixed thanks to "git retry".

Sign in to add a comment