yes https://bugs.chromium.org/p/chromium/issues/detail?id=826575

Taking a look.

So the eve-arcnext chrome pfq is green including the build that was manually upreved. 
https://uberchromegw.corp.google.com/i/chromeos/builders/eve-arcnext-chrome-pfq

I advocate that we pin chrome back to undo the uprev so we can reopen the tree, then investigate why prebuilts aren't working properly.

The UploadPrebuilts stage looks odd:
https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Feve-arcnext-chrome-pfq%2F25%2F%2B%2Frecipes%2Fsteps%2FUploadPrebuilts%2F0%2Fstdout

...
16:20:52: WARNING: unable to match packages: set(['chromeos-chrome', 'chromeos-base/chromium-source'])
...

Any chance this warning is the root cause?

Should we pin chrome back to version 67.0.3376.0?

Possible. I am running binhost_test right now, taking a long time.

eve-arcnext-paladin cannot find Chrome prebuilts -- 
_CompatId(arch='amd64', useflags=('accessibility', 'authpolicy', 'autotest', 'build_native_assistant', 'build_tests', 'buildcheck', 'chrome_debug', 'chrome_internal', 'chrome_remoting', 'clang', 'cups', 'debug_fission', 'evdev_gestures', 'fonts', 'gold', 'highdpi', 'libcxx', 'nacl', 'opengles', 'ozone_platform_default_gbm', 'ozone_platform_gbm', 'runhooks', 'smbprovider', 'v4l2_codec', 'vaapi', 'xkbcommon'), cflags=('-O2', '-pipe', '-O2', '-pipe', '-march=corei7', '-g', '-fno-exceptions', '-fno-unwind-tables', '-fno-asynchronous-unwind-tables'))

When we take out config-requested useflags ([u'chrome_internal']), eve-arcnext-paladin cannot find Chrome prebuilts -- 
_CompatId(arch='amd64', useflags=('accessibility', 'authpolicy', 'autotest', 'build_native_assistant', 'build_tests', 'buildcheck', 'chrome_debug', 'chrome_remoting', 'clang', 'cups', 'debug_fission', 'evdev_gestures', 'fonts', 'gold', 'highdpi', 'libcxx', 'nacl', 'opengles', 'ozone_platform_default_gbm', 'ozone_platform_gbm', 'runhooks', 'smbprovider', 'v4l2_codec', 'vaapi', 'xkbcommon'), cflags=('-O2', '-pipe', '-O2', '-pipe', '-march=corei7', '-g', '-fno-exceptions', '-fno-unwind-tables', '-fno-asynchronous-unwind-tables'))

As a side, build_native_assistant is new, it was added here for builds >=67.0.3378.0_rc-r1
https://chrome-internal-review.googlesource.com/#/c/chromeos/overlays/overlay-eve-arcnext-private/+/591624/4/profiles/base/package.use

But Chrome PFQ at 3382 is green, so this is not the immediate culprit.

From the PFQ

[ebuild  N     ] chromeos-base/chromeos-chrome-67.0.3382.0_rc-r1::chromiumos to /build/eve-arcnext/ USE="accessibility authpolicy autotest build_native_assistant build_tests buildcheck chrome_debug chrome_internal chrome_remoting clang cups debug_fission evdev_gestures fonts gold highdpi libcxx nacl opengles runhooks smbprovider v4l2_codec vaapi xkbcommon -afdo_chrome_exp1 -afdo_chrome_exp2 -afdo_use -app_shell -asan -cfi -chrome_debug_tests -chrome_media -clang_tidy -component_build -goma -hardfp -internal_gles_conform -jumbo -lld -mojo (-neon) -opengl -thinlto -v4lplugin -verbose -vtable_verify" OZONE_PLATFORM="gbm -caca -cast -egltest {-test}" OZONE_PLATFORM_DEFAULT="gbm -caca -cast -egltest {-test}" 0 KiB

Again, build_native_assistant is used in the PFQ.

Mhh, we do have uploaded prebuilds
gsutil ls -ah gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-1052*/packages/chromeos-base/chromeos-chrome-67.0.338*.tbz2
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10520.0.0-rc4/packages/chromeos-base/chromeos-chrome-67.0.3381.0_rc-r1.debug.tbz2#1522125108393868
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10520.0.0-rc4/packages/chromeos-base/chromeos-chrome-67.0.3381.0_rc-r1.tbz2#1522125095591889
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10520.0.0-rc5/packages/chromeos-base/chromeos-chrome-67.0.3381.0_rc-r1.debug.tbz2#1522138452738847
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10520.0.0-rc5/packages/chromeos-base/chromeos-chrome-67.0.3381.0_rc-r1.tbz2#1522138449189832
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10521.0.0-rc1/packages/chromeos-base/chromeos-chrome-67.0.3381.0_rc-r1.debug.tbz2#1522157543501021
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10521.0.0-rc1/packages/chromeos-base/chromeos-chrome-67.0.3381.0_rc-r1.tbz2#1522157531271511
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10522.0.0-rc1/packages/chromeos-base/chromeos-chrome-67.0.3381.0_rc-r1.debug.tbz2#1522180085457480
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10522.0.0-rc1/packages/chromeos-base/chromeos-chrome-67.0.3381.0_rc-r1.tbz2#1522180075788437
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10522.0.0-rc2/packages/chromeos-base/chromeos-chrome-67.0.3381.0_rc-r1.debug.tbz2#1522192867431171
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10522.0.0-rc2/packages/chromeos-base/chromeos-chrome-67.0.3381.0_rc-r1.tbz2#1522192862497732
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10522.0.0-rc3/packages/chromeos-base/chromeos-chrome-67.0.3381.0_rc-r1.debug.tbz2#1522203650532187
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10522.0.0-rc3/packages/chromeos-base/chromeos-chrome-67.0.3381.0_rc-r1.tbz2#1522203631875238
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10523.0.0-rc1/packages/chromeos-base/chromeos-chrome-67.0.3382.0_rc-r1.debug.tbz2#1522214950556481
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10523.0.0-rc1/packages/chromeos-base/chromeos-chrome-67.0.3382.0_rc-r1.tbz2#1522214940144946
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10523.0.0-rc2/packages/chromeos-base/chromeos-chrome-67.0.3382.0_rc-r1.debug.tbz2#1522228692211575
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10523.0.0-rc2/packages/chromeos-base/chromeos-chrome-67.0.3382.0_rc-r1.tbz2#1522228675562312
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10523.0.0-rc3/packages/chromeos-base/chromeos-chrome-67.0.3382.0_rc-r1.debug.tbz2#1522238769824627
gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10523.0.0-rc3/packages/chromeos-base/chromeos-chrome-67.0.3382.0_rc-r1.tbz2#1522238746209531

Summary:
1. eve-arcnext pfq is green
2. builds exist on GS
3. prebuilds are not found by build_packages locally or on CQ
4. binhost_test can't find prebuilds
5. visual inspection of flags seems same, but some are "fuzzy" but probably have not changed in a while

Where are the flags tracked? They are not in the prebuild tarball. So there must be a database?

The desired flags are determined by some logic in the setup_board script, at setup_board time, based on overlays and .conf files.

I'm not certain the mechanism of how the "available" use flags are determined. +davidjames perhaps to comment?

Also +jclinton fyi

The upload warning comes from here
https://cs.corp.google.com/chromeos_public/chromite/scripts/upload_prebuilts.py?type=cs&q=%22unable+to+match+packages%22&l=439

Trying to understand what it is trying to tell us. Probably internal/public Chrome source mismatch?

-> today's gardener will pin back to previous chrome, then we can hopefully reopne tree

we can dig into understanding after

Probably easiest for now to pin back.

No, actually it cannot be pinned back. I think it is better to mark the eve-arcnext-paladin and eve-arcnext-chrome-pfq builders experimental.

Ok, I uploaded the pin chrome CLs. I will hold off if that's better.

Or before I do that, how about reverting the native_assistant flag change?

Random revert time. 

I find it hard to predict which revert will fix the system as we changed so many things.

I've marked them as experimental in the tree status and reopen the tree

Ok.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-eve-arcnext-private/+/759f4233c3f3519072f25df3acaaaf4fab7a2a6e

commit 759f4233c3f3519072f25df3acaaaf4fab7a2a6e
Author: Ilja H. Friedel <ihf@google.com>
Date: Wed Mar 28 17:46:13 2018

Marking experimental in the tree status is insufficient. binhost_test does not look at the tree status in order to determine which boards to check, it only respects source. If prebuilts remain broken for that board, the CQ and pre-cq will keep failing.

You need to mark experimental in source.

I will prepare  a CL to mark it in the source

Here is the CL: https://chromium-review.googlesource.com/c/chromiumos/chromite/+/984400

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/9a9e4a1b1ca22d13d2ba115eada1b12d95410393

commit 9a9e4a1b1ca22d13d2ba115eada1b12d95410393
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Wed Mar 28 18:14:06 2018

chromeos_config: mark eve-arcnext experimental in both cq and pfq

BUG= chromium:826755 
TEST=unittest

Change-Id: Id3bccbec90f5880ab76e68086c8c0b6fc6539a3e
Reviewed-on: https://chromium-review.googlesource.com/984400
Commit-Queue: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Trybot-Ready: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/9a9e4a1b1ca22d13d2ba115eada1b12d95410393/cbuildbot/config_dump.json
[modify] https://crrev.com/9a9e4a1b1ca22d13d2ba115eada1b12d95410393/cbuildbot/chromeos_config.py

Looks like the revert in #28 was sufficient to cycle the builder green again:
https://uberchromegw.corp.google.com/i/chromeos/builders/eve-arcnext-paladin/builds/366

I don't understand why. Clearly the PFQ in #13 was using build_native_assistant.

We need to unmark experimental then. shuqianz@ can you do the revert for that?

Sure.

I examined one of the package in #15 by downloading it and unzip.  The package appears to be built with the right flags and contains the specific assistant binary "libassistant.so". 

Can anyone shed some light about comment #16 and #18 on why it cannot find the prebuilt binary which does exist?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1e610ad2338d2ed98bb16be1639ed531f39f421f

commit 1e610ad2338d2ed98bb16be1639ed531f39f421f
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Wed Mar 28 21:24:09 2018

Roll src/third_party/chromite/ 9912024a6..9a9e4a1b1 (1 commit)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/9912024a6e90..9a9e4a1b1ca2

$ git log 9912024a6..9a9e4a1b1 --date=short --no-merges --format='%ad %ae %s'
2018-03-28 shuqianz chromeos_config: mark eve-arcnext experimental in both cq and pfq

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:826755 


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I3027899a5a167981423ec8b5d169f267b9a598c7
Reviewed-on: https://chromium-review.googlesource.com/984403
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#546602}
[modify] https://crrev.com/1e610ad2338d2ed98bb16be1639ed531f39f421f/DEPS

Plan of action now is to mark kevin-arcnext/caroline-arcnext important and keep eve-arcnext experimental:
https://chromium-review.googlesource.com/#/c/chromiumos/chromite/+/984869/

This will give Xiaohui a chance to experiment with the new flag on eve-arcnext without blowing up the tree (only that build).

You should be able to reproduce the failures in the binhost_test by running it locally. binhost_test requires that all paladin builders have working prebuilts and that those prebuilts are recorded in the JSON summary files exported by the last passing Chrome PFQ run.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/85a5ac7c6177ca59c1c906e9787360525dd4ffd8

commit 85a5ac7c6177ca59c1c906e9787360525dd4ffd8
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Thu Mar 29 02:42:48 2018

cbuildbot: mark caroline/kevin-arcnext important.

eve-arcnext was marked experimental due to  crbug.com/826755 .
But we need arcnext coverage. And we want to keep experimenting
with eve-arcnext. So chose other boards.

BUG= chromium:826755 , b:74343188, b:74871043
TEST=./chromeos_config_unittest
Green CQ builders:
https://uberchromegw.corp.google.com/i/chromeos/builders/kevin-arcnext-paladin
https://uberchromegw.corp.google.com/i/chromeos/builders/caroline-arcnext-paladin
Green Chrome PFQ builders:
https://uberchromegw.corp.google.com/i/chromeos/builders/kevin-arcnext-chrome-pfq
https://uberchromegw.corp.google.com/i/chromeos/builders/caroline-arcnext-chrome-pfq

Change-Id: I6d0eda830ef0b0966dd4beb50027ac7cb75b3344
Reviewed-on: https://chromium-review.googlesource.com/984869
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Trybot-Ready: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ahmed Fakhry <afakhry@chromium.org>

[modify] https://crrev.com/85a5ac7c6177ca59c1c906e9787360525dd4ffd8/cbuildbot/config_dump.json
[modify] https://crrev.com/85a5ac7c6177ca59c1c906e9787360525dd4ffd8/cbuildbot/chromeos_config.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5145a016df3650855d958bf5b659ff0d84a8cf16

commit 5145a016df3650855d958bf5b659ff0d84a8cf16
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Thu Mar 29 06:03:45 2018

Roll src/third_party/chromite/ 4e367c2f4..85a5ac7c6 (2 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/4e367c2f4075..85a5ac7c6177

$ git log 4e367c2f4..85a5ac7c6 --date=short --no-merges --format='%ad %ae %s'
2018-03-28 ihf cbuildbot: mark caroline/kevin-arcnext important.
2018-03-26 jkop chromite: Increase VM Test timeout to 90 minutes.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:826755 , chromium:818859 


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I8c9699eac4f21cb7c857f820d12011111218406d
Reviewed-on: https://chromium-review.googlesource.com/985650
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#546742}
[modify] https://crrev.com/5145a016df3650855d958bf5b659ff0d84a8cf16/DEPS

I think I found some clue on why eve-arcnext build would fail to find the chrome prebuilt. 

It seems eve board does not have its own chrome binhost configured.  

eve-arcnext is directly sourcing caroline binhost config.
chroot/build/eve-arcnext/etc/make.conf.board:source /mnt/host/source/src/private-overlays/chromeos-partner-overlay/chromeos/binhost/
target/caroline-LATEST_RELEASE_CHROME_BINHOST.conf

Ideally I would assume eve-arcnext should source its own binhost config, but it seems currently the eve-arcnext binhost has an empty configure file. 

src/private-overlays/chromeos-partner-overlay/chromeos/binhost/target/eve-arcnext-LATEST_RELEASE_CHROME_BINHOST.conf
1:LATEST_RELEASE_CHROME_BINHOST=""

The way it seem to find the binhost configure file to use seem to be looking at src/private-overlays/chromeos-partner-overlay/chromeos/binhost/chrome.json to find the compatible binhost.  This json file does not have an entry for eve nor eve-arcnext.  It does have caroline and caroline-arcnext. 

Maybe once we have eve-arcnext's binhost conf file filled and the chrome.json added the eve-arcnext entry, things could work out.

I think this issue is resolved.

This bug is not "fixed", it is "worked around" for now by reverting the cl that added new chromeos-chrome gn build flags to eve-arcnext board.  We have not found a path forward to enabled the gn flags yet. 

To me it feels like the chromeos build infrastructure makes it very difficult to enable chromeos-chrome build flags on a per board basis because of the prebuilt dance.  I really appreciate it if anyone has any suggestion on how to do this.

Here is the findings for why the previous manual uprev broke:
The uprev Cl https://chrome-internal.googlesource.com/chromeos/overlays/chromeos-partner-overlay/+/309254066f5d799ef7eb7ac27628bdb77130975d

Showed changes to chromeos/overlays/chromeos-partner-overlay/chromeos/binhost/chrome.json that included eve-arcnext-chrome-pfq but without the new use flag "build_native_assistant".  Examining the binary package it points to gs://chromeos-prebuilt/board/eve-arcnext/chrome-R67-10522.0.0-rc2/packages/ also confirmed the build does not have the new flags.  This seems to a race.

To make the binhost available, the pfq has to be marked as imported, otherwise the script won't look at it, nor make it available as binhost.
https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/binhost.py?rcl=b90af5c203f786b1b0e161e81875da0046c4ea14&l=76

We will first mark the eve-arcnext-chrome-pfq as important, but keep eve-arcnext-palaldin as experimental.  This should make eve-arcnext as a potential binhost but won't risk breaking CQ.  Once chrome uprevs, if it hits a good build with the new build flag, which we already have a few now, everything should be green. 

^ that sounds like the correct flow, and is in line with my suspicions when I investigated.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/5d0fcc3fa3ad3e4c200b0fbcb0f9c080e88c0117

commit 5d0fcc3fa3ad3e4c200b0fbcb0f9c080e88c0117
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Tue Apr 03 03:08:00 2018

chromeos_config: mark eve-arcnext-chrome-pfq important.

BUG= chromium:826755 
TEST=All green
https://uberchromegw.corp.google.com/i/chromeos/builders/eve-arcnext-chrome-pfq

Change-Id: I6387e7a9ad5a09cbbd96396d29299cd1584c0c06
Reviewed-on: https://chromium-review.googlesource.com/990808
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Trybot-Ready: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Xiaohui Chen <xiaohuic@chromium.org>
Reviewed-by: Po-Hsien Wang <pwang@chromium.org>

[modify] https://crrev.com/5d0fcc3fa3ad3e4c200b0fbcb0f9c080e88c0117/cbuildbot/config_dump.json
[modify] https://crrev.com/5d0fcc3fa3ad3e4c200b0fbcb0f9c080e88c0117/cbuildbot/chromeos_config.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bb4b07b5df673fcb8f756cbfa6ab227d58b0d963

commit bb4b07b5df673fcb8f756cbfa6ab227d58b0d963
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Tue Apr 03 04:15:47 2018

Roll src/third_party/chromite/ 0f51564af..5d0fcc3fa (1 commit)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/0f51564af246..5d0fcc3fa3ad

$ git log 0f51564af..5d0fcc3fa --date=short --no-merges --format='%ad %ae %s'
2018-04-02 ihf chromeos_config: mark eve-arcnext-chrome-pfq important.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:826755 


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I6d5ed64763d7d87533683ad14220a98e82d1b422
Reviewed-on: https://chromium-review.googlesource.com/991600
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#547623}
[modify] https://crrev.com/bb4b07b5df673fcb8f756cbfa6ab227d58b0d963/DEPS

Could someone on this bug ping me to chat about BinhostTest? We'd like to do a manual PFQ: uprev but still see BinhostTest failures, and the logs are inscrutable (hundreds of lines enumerating differing cflags), e.g. https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fmaster-chromium-pfq%2F6145%2F%2B%2Frecipes%2Fsteps%2FBinhostTest%2F0%2Fstdout

Michael, I think you will need a new issue for the new failure. But basically BinhostTest checks that all prebuilds that are needed were indeed created.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/188669bb75264ec853ca7f7020be14e2244607ee

commit 188669bb75264ec853ca7f7020be14e2244607ee
Author: Aviv Keshet <akeshet@chromium.org>
Date: Sat Apr 07 01:08:15 2018

chromeos_config: make eve-arcnext-pi-android-pfq not important
    
We suspect the Chrome PFQ is unable to uprev while this builder is
important, even though all use flags should match. This change needs to
be reverted after successful uprev.

BUG= chromium:826755 
TEST=./chromeos_config_unittest


Change-Id: If9b792d3a7e63cca9e20dd06ab7f22e5e0897d8b
Reviewed-on: https://chromium-review.googlesource.com/1000838
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Trybot-Ready: Aviv Keshet <akeshet@chromium.org>
Trybot-Ready: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>

[modify] https://crrev.com/188669bb75264ec853ca7f7020be14e2244607ee/cbuildbot/config_dump.json
[modify] https://crrev.com/188669bb75264ec853ca7f7020be14e2244607ee/cbuildbot/chromeos_config.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9315e7bf7856a2c8006f981162d6ba505122b800

commit 9315e7bf7856a2c8006f981162d6ba505122b800
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Sat Apr 07 03:25:19 2018

Roll src/third_party/chromite/ 0f0cab3ea..188669bb7 (1 commit)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/0f0cab3eae12..188669bb7526

$ git log 0f0cab3ea..188669bb7 --date=short --no-merges --format='%ad %ae %s'
2018-04-06 akeshet chromeos_config: make eve-arcnext-pi-android-pfq not important

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:826755 


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I685b258b6e51b38a796fb885eb1e9011dfb57bbc
Reviewed-on: https://chromium-review.googlesource.com/1000418
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#549023}
[modify] https://crrev.com/9315e7bf7856a2c8006f981162d6ba505122b800/DEPS

[corrected] Looks like we finally got an uprev with the new chrome use flag. But to get there we had to tear down both CQ and Android PFQ (in this case only for eve, but still). I don't like this. There must be a better way. I think it is worth writing a post mortem, because this will happen over and over.

After the uprev I prepared to mark the CQ + Android PFQ important again and binhost test shows the same error once more. I think it is time to debug binhost test or add more failure information to the logs.

https://chromium-review.googlesource.com/#/c/1002192/

AssertionError: When we take out config-requested useflags ([u'chrome_internal']), eve-arcnext-paladin cannot find Chrome prebuilts -- _CompatId(arch='amd64', useflags=('accessibility', 'authpolicy', 'autotest', 'build_native_assistant', 'build_tests', 'buildcheck', 'chrome_debug', 'chrome_remoting', 'clang', 'cups', 'debug_fission', 'evdev_gestures', 'fonts', 'gold', 'highdpi', 'libcxx', 'nacl', 'opengles', 'ozone_platform_default_gbm', 'ozone_platform_gbm', 'runhooks', 'smbprovider', 'v4l2_codec', 'vaapi', 'xkbcommon'), cflags=('-O2', '-pipe', '-O2', '-pipe', '-march=corei7', '-g', '-fno-exceptions', '-fno-unwind-tables', '-fno-asynchronous-unwind-tables'))

AssertionError: When we take out config-requested useflags ([u'chrome_internal']), eve-arcnext-paladin cannot find Chrome prebuilts -- _CompatId(arch='amd64', useflags=('accessibility', 'authpolicy', 'autotest', 'build_native_assistant', 'build_tests', 'buildcheck', 'chrome_debug', 'chrome_remoting', 'clang', 'cups', 'debug_fission', 'evdev_gestures', 'fonts', 'gold', 'highdpi', 'libcxx', 'nacl', 'opengles', 'ozone_platform_default_gbm', 'ozone_platform_gbm', 'runhooks', 'smbprovider', 'v4l2_codec', 'vaapi', 'xkbcommon'), cflags=('-O2', '-pipe', '-O2', '-pipe', '-march=corei7', '-g', '-fno-exceptions', '-fno-unwind-tables', '-fno-asynchronous-unwind-tables'))

The error there is telling you that you don't have an eve-arcnext-chromium-pfq. Without such a PFQ, rolling forward Chrome would break eve-arcnext for end-developers (who build Chromium, not Chrome)

Hi, is there any next action we can take here?

One question is that if it is easily possible to mark Android PFQ as important at least (leaving paladin experimental).
Context: a bug b/111330620 for ARC constable to make PFQ important is filed today and I'm the constable this week.

No update in more than 1 month. Closing since the original issue was fixed and any follow-up is not a P1.