New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 712757 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Last visit > 30 days ago
Closed: Jun 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Almost all the PFQ informational builders failed at syncchrome stage

Project Member Reported by x...@chromium.org, Apr 18 2017

Issue description

For example, https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/x86-alex-tot-chrome-pfq-informational/builds/23591 
https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/cyan-tot-chrome-pfq-informational/builds/2414

The error message:
23:48:21: INFO: RunCommand: cros_sdk -- emerge-x86-alex -p --quiet '=chromeos-base/chromeos-chrome-60.0.3075.0_alpha-r1'
23:48:24: ERROR: Cannot emerge-x86-alex =chromeos-base/chromeos-chrome-60.0.3075.0_alpha-r1
Is Chrome pinned to an older version?

@@@STEP_FAILURE@@@
23:48:24: ERROR: Chrome is pinned. Attempting to continue build for chrome atom chromeos-base/chromeos-chrome-60.0.3075.0_alpha-r1 anyway but build will ultimately fail.
23:48:24: INFO: Deleting pin file at /b/cbuild/repository/src/third_party/chromiumos-overlay/profiles/default/linux/package.mask/chromepin and proceeding.

It seems to me that Chrome was pinned to an older version but I asked around and no one knows about it. 
 

Comment 1 by x...@chromium.org, Apr 18 2017

According to bhthompson@, if Chrome is pinned there should be a /src/third_party/chromiumos-overlay/profiles/default/linux/package.mask/chromepin file in https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/profiles/default/linux/package.mask/. However, there is no such file there which means Chrome is not pinned to an older version.

nxia@ suggested that the error log (from this file: https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/commands.py?q=%22Is+Chrome+pinned+to+an+older%22&sq=package:%5Echromeos_(internal%7Cpublic)$&l=1561) might be wrong. Need to look into it to see why the emerge failed.

Comment 2 by x...@chromium.org, Apr 18 2017

Cc: bhthompson@chromium.org
Cc: yunlian@chromium.org
+yunlian@

The failure began between https://crosland.corp.google.com/log/9472.0.0..9473.0.0 it looks like.

The first suspect would be https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/d7a94f52e35abd17a571f074ccdb0105d3c35b71 given it is the only change directly touching the chromeos-chrome ebuild, and what we are seeing is essentially a failure to emerge chromeos-chrome (though we don't see exactly why due to the --quiet passed in by the builder script). Is there any chance this CL could cause these builders to fail to emerge?

Hmm, it looks like we actually started failing while still at 9472.0.0:

https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-tot-chromium-pfq-informational/builds/11670

Right, I think something landed during the 9472 window. 

For Tricky there is a 9472 build that was green https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/tricky-tot-chrome-pfq-informational/builds/4211 and one that was red https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/tricky-tot-chrome-pfq-informational/builds/4212

So it is really probably something that landed later in the 9472 range, but it is at least in 9472+. 

Comment 6 by x...@chromium.org, Apr 18 2017

I'm trying to see if I can repro this failure locally.
Cc: dgarr...@chromium.org
Ah, I see, right, we build with ToT. Didn't these builders used to include the cros changes?

Anyway, I'm looking at the entire builder output and there were definitely chromite changes, 'c3c3338f..8dbd2c59'. No smoking gun, but plenty of changesI don't understand.

Comment 8 by x...@chromium.org, Apr 18 2017

I can't repro the failure locally. The failure happened in SyncChrome stage and before BuildPackages stage. However, the BuildPackages step can be passed successfully locally. 

From the log, it failed when running command in SyncChrome stage: cros_sdk -- emerge-peach_pit -p --quiet '=chromeos-base/chromeos-chrome-60.0.3075.0_alpha-r1'
However, I could get this command running locally. It failed with error message "emerge: there are no ebuilds to satisfy "=chromeos-base/chromeos-chrome-60.0.3075.0_alpha-r1" for /build/peach_pit/."

Does anyone know how to get it run locally?

Comment 9 by x...@chromium.org, Apr 18 2017

Sorry, a typo in my last comment #8. I could not get this command running locally. 

Comment 10 by x...@chromium.org, Apr 18 2017

I manually repro'ed it locally. Confirmed it was caused by the CL https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/d7a94f52e35abd17a571f074ccdb0105d3c35b71. 

Here is the error message I got locally:
- chromeos-base/chromeos-chrome-60.0.3075.0_rc-r1::chromiumos (masked by: missing keyword, invalid: REQUIRED_USE: USE flag '(gold' is not in IUSE)

I'm going to revert the CL.
Reproducing the builder steps can be tricky.

I ran all of the cros_sdk commands from the 'steps' output but got a different but similar result:

!!! All ebuilds that could satisfy "=chromeos-base/chromeos-chrome-60.0.3075.0_alpha-r1" for /build/x86-generic/ have been masked.

I'll look through the output some more to see if I can figure out what I am missing. I'm using:
https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-tot-chromium-pfq-informational/builds/11670/steps/steps/logs/stdio


Comment 12 by x...@chromium.org, Apr 18 2017

Coping the chromeos-chrome-9999.ebuild to a different file is a workaround the mask problem. (That's why I have chromeos-chrome-60.0.3075.0_rc-r1 in my error message)
Huh. I wonder how the builder is doing that? I couldn't find anything in the output.

Anyway, good catch! Please do the revert and assign this to the author.

How spammy was the output without --quiet? (~how many lines)? I'm thinking we may not want --quiet there if it masks errors like that....

Comment 14 by x...@chromium.org, Apr 18 2017

Cc: -yunlian@chromium.org x...@chromium.org
Owner: yunlian@chromium.org
Status: Assigned (was: Untriaged)
The CL was reverted in https://chromium-review.googlesource.com/c/481007/. 
Assigned to the CL's for further investigation.
Status: Verified (was: Assigned)

Sign in to add a comment