Almost all the PFQ informational builders failed at syncchrome stage |
||||||
Issue descriptionFor example, https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/x86-alex-tot-chrome-pfq-informational/builds/23591 https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/cyan-tot-chrome-pfq-informational/builds/2414 The error message: 23:48:21: INFO: RunCommand: cros_sdk -- emerge-x86-alex -p --quiet '=chromeos-base/chromeos-chrome-60.0.3075.0_alpha-r1' 23:48:24: ERROR: Cannot emerge-x86-alex =chromeos-base/chromeos-chrome-60.0.3075.0_alpha-r1 Is Chrome pinned to an older version? @@@STEP_FAILURE@@@ 23:48:24: ERROR: Chrome is pinned. Attempting to continue build for chrome atom chromeos-base/chromeos-chrome-60.0.3075.0_alpha-r1 anyway but build will ultimately fail. 23:48:24: INFO: Deleting pin file at /b/cbuild/repository/src/third_party/chromiumos-overlay/profiles/default/linux/package.mask/chromepin and proceeding. It seems to me that Chrome was pinned to an older version but I asked around and no one knows about it.
,
Apr 18 2017
,
Apr 18 2017
+yunlian@ The failure began between https://crosland.corp.google.com/log/9472.0.0..9473.0.0 it looks like. The first suspect would be https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/d7a94f52e35abd17a571f074ccdb0105d3c35b71 given it is the only change directly touching the chromeos-chrome ebuild, and what we are seeing is essentially a failure to emerge chromeos-chrome (though we don't see exactly why due to the --quiet passed in by the builder script). Is there any chance this CL could cause these builders to fail to emerge?
,
Apr 18 2017
Hmm, it looks like we actually started failing while still at 9472.0.0: https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-tot-chromium-pfq-informational/builds/11670
,
Apr 18 2017
Right, I think something landed during the 9472 window. For Tricky there is a 9472 build that was green https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/tricky-tot-chrome-pfq-informational/builds/4211 and one that was red https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/tricky-tot-chrome-pfq-informational/builds/4212 So it is really probably something that landed later in the 9472 range, but it is at least in 9472+.
,
Apr 18 2017
I'm trying to see if I can repro this failure locally.
,
Apr 18 2017
Ah, I see, right, we build with ToT. Didn't these builders used to include the cros changes? Anyway, I'm looking at the entire builder output and there were definitely chromite changes, 'c3c3338f..8dbd2c59'. No smoking gun, but plenty of changesI don't understand.
,
Apr 18 2017
I can't repro the failure locally. The failure happened in SyncChrome stage and before BuildPackages stage. However, the BuildPackages step can be passed successfully locally. From the log, it failed when running command in SyncChrome stage: cros_sdk -- emerge-peach_pit -p --quiet '=chromeos-base/chromeos-chrome-60.0.3075.0_alpha-r1' However, I could get this command running locally. It failed with error message "emerge: there are no ebuilds to satisfy "=chromeos-base/chromeos-chrome-60.0.3075.0_alpha-r1" for /build/peach_pit/." Does anyone know how to get it run locally?
,
Apr 18 2017
Sorry, a typo in my last comment #8. I could not get this command running locally.
,
Apr 18 2017
I manually repro'ed it locally. Confirmed it was caused by the CL https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/d7a94f52e35abd17a571f074ccdb0105d3c35b71. Here is the error message I got locally: - chromeos-base/chromeos-chrome-60.0.3075.0_rc-r1::chromiumos (masked by: missing keyword, invalid: REQUIRED_USE: USE flag '(gold' is not in IUSE) I'm going to revert the CL.
,
Apr 18 2017
Reproducing the builder steps can be tricky. I ran all of the cros_sdk commands from the 'steps' output but got a different but similar result: !!! All ebuilds that could satisfy "=chromeos-base/chromeos-chrome-60.0.3075.0_alpha-r1" for /build/x86-generic/ have been masked. I'll look through the output some more to see if I can figure out what I am missing. I'm using: https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-tot-chromium-pfq-informational/builds/11670/steps/steps/logs/stdio
,
Apr 18 2017
Coping the chromeos-chrome-9999.ebuild to a different file is a workaround the mask problem. (That's why I have chromeos-chrome-60.0.3075.0_rc-r1 in my error message)
,
Apr 18 2017
Huh. I wonder how the builder is doing that? I couldn't find anything in the output. Anyway, good catch! Please do the revert and assign this to the author. How spammy was the output without --quiet? (~how many lines)? I'm thinking we may not want --quiet there if it masks errors like that....
,
Apr 18 2017
The CL was reverted in https://chromium-review.googlesource.com/c/481007/. Assigned to the CL's for further investigation.
,
Jun 21 2017
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by x...@chromium.org
, Apr 18 2017