New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 801062 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug



Sign in to add a comment

"Builder: coral factory factory-coral-10122.B" Builds Failed in the InitialCheckout Stage

Project Member Reported by marcochen@chromium.org, Jan 11 2018

Issue description

Build fail link - https://uberchromegw.corp.google.com/i/chromeos.branch/builders/coral%20factory%20factory-coral-10122.B/builds/38

Partial logs for error -

fatal: Couldn't find remote ref refs/heads/factory-coral-10122.B-next/chromeos-experimental
[W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref refs/heads/factory-coral-10122.B-next/chromeos-experimental\n"
[W git.go:294] Retrying after 3s (rc=128): transient error string encountered

...

fatal: Couldn't find remote ref refs/heads/factory-coral-10122.B-next/chromeos-experimental
[W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref refs/heads/factory-coral-10122.B-next/chromeos-experimental\n"
[W git.go:317] Command completed with rc 128 after 10 transient failure(s).
error: Cannot fetch chromiumos/third_party/kernel
error: Exited sync due to fetch errors
 
Other info:

  1. I can really see there is a branch [1] in the third_party/kernel.
  2. Sheriff and I can "repo sync" coral factory branch successfully and locally.

[1] https://chromium.googlesource.com/chromiumos/third_party/kernel/+/factory-coral-10122.B-next/chromeos-experimental
Owner: jrbarnette@chromium.org
Assign owner to infra deputy first for visibility.
Cc: stevenh@chromium.org
Labels: -Pri-3 Pri-1
more and more factory / firmware branches met the same error including fizz, glados and others. And would expect to get support for upcoming factory build of partners. Thanks.

fizz: https://uberchromegw.corp.google.com/i/chromeos.branch/builders/fizz%20pre-flight%20firmware-fizz-10139.B/builds/51
master-release release-R64-10176.B is also in the same issue.

https://uberchromegw.corp.google.com/i/chromeos_release/builders/master-release%20release-R64-10176.B


Similar to  bug #801211 
Project Member

Comment 6 by bugdroid1@chromium.org, Jan 11 2018

Labels: merge-merged-release-R64-10176.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/manifest/+/9269bd1ac03fef1c5b622e5aa2b1d283c4245f27

commit 9269bd1ac03fef1c5b622e5aa2b1d283c4245f27
Author: Bernie Thompson <bhthompson@google.com>
Date: Thu Jan 11 17:47:10 2018

Remove kernel-experimental branch from R64 manifest

BUG= chromium:801062 
TEST=None

Change-Id: I473611f74240a6f48b6e76d7a641d250698129c4
Reviewed-on: https://chromium-review.googlesource.com/862102
Reviewed-by: Douglas Anderson <dianders@chromium.org>
Reviewed-by: YH Lin <yueherngl@chromium.org>
Commit-Queue: Bernie Thompson <bhthompson@chromium.org>
Tested-by: Bernie Thompson <bhthompson@chromium.org>

[modify] https://crrev.com/9269bd1ac03fef1c5b622e5aa2b1d283c4245f27/full.xml

Comment 7 by groeck@chromium.org, Jan 11 2018

Wonder if some entity or person tried to create a branch named "factory-coral-10122.B-next" (and release-R64-10176.B-next and so on). That would mess up git quite nicely.

Comment 8 by groeck@chromium.org, Jan 11 2018

Also, there might be a problem if there was an infrastructure change which uses '/' as delimiter to parse branch names.

Project Member

Comment 9 by bugdroid1@chromium.org, Jan 11 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/manifest-internal/+/5a30975300b1584044b144bf7be1f327e887d245

commit 5a30975300b1584044b144bf7be1f327e887d245
Author: Douglas Anderson <dianders@chromium.org>
Date: Thu Jan 11 18:29:08 2018

Cc: jimhoff@google.com
Labels: -Pri-1 OS-Chrome Pri-0
+jimhoff@ GoB git oncall

So removing the kernel-experimental branch did not help, it just fails on the next one.

I can check this out locally and it works ok, though I do start seeing it trying to use git gc on a cleanly wiped checkout which makes me nervous that our kernel repo has grown beyond some boundary that may be making us more likely to see corruption?

I think we are nearing the end of our expertise here, adding GoB git oncall for thoughts on why we might be failing checkouts intermittently like this. How would one normally debug these sort of 'Transient error string identified in STDERR: "fatal: Couldn't find remote ref refs/heads/factory-coral-10122.B-next/chromeos-experimental'?
Cc: pbe...@chromium.org

Comment 12 by jimhoff@google.com, Jan 11 2018

This doesn't sound like a production emergency affecting many customers
(what oncall typically handles), but I am having trouble understanding the
details of the bug.
If it a production emergency, let me know. If it's not, go/gob-ticket is a
good way to get support. Also see our docs
<https://g3doc.corp.google.com/company/teams/gerritcodereview/support.md?cl=head>
on support in general.
Cc: iannucci@chromium.org
Adding the chrome trooper; "[W git.go:283]" messages come from a
chrome infra tool.
Owner: iannucci@chromium.org
Status: Assigned (was: Untriaged)
The stuff that's failing is mediated by code in Chrome Infra, not
Cros Infra.

So, roping in a trooper.
Blockedon: 801322
Blockedon: -801322
I don't think 801322 is related.
After much tribulation, we've found and filed a GoB bug:  b/71872654

Status: Fixed (was: Assigned)
It ain't happenin' no more, and GoB team claims its fixed, so, let's
declare victory.

Sign in to add a comment