we decided not to go forward with the suggestion in here.
That would have broken the CQ for a while until the master-pfq was successful.

We would like to know if there is a better way to do what we want. Ie: change a chrome USE flag for a board that is in the PFQ and the CQ.

you must never change the chrome build such that the paladin has to rebuild it.  chumping a CL to bypass checks doesn't change that.  the Chrome PFQ must pass and produce the binpkgs that paladins would utilize.

the error is in the *chromium* master pfq.  it's telling you that there are no compatible chromium builds available for peach pit.  we have fewer chromium builds than chrome builds.

you can probably add the package.use entry to the private overlay's profile.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/b3dcbb513b33f9f35e71b650b156e6b8ba399fa2

commit b3dcbb513b33f9f35e71b650b156e6b8ba399fa2
Author: Manoj Gupta <manojgupta@google.com>
Date: Sun Feb 12 19:00:14 2017

Revert peach_pit chrome build with clang.

Revert a4caf0eaca9c0262ba737e441425ab0e358edded since master chrome
pfq is failing.
parent file is not removed as suggested  by vapier@ .

BUG= chromium:691295 
TEST: No clang in chromeos-chrome in build packages.
Change-Id: I927423cba7fed18735861900d1f2c2207ff5302c
Reviewed-on: https://chromium-review.googlesource.com/441251
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Luis Lozano <llozano@chromium.org>
Commit-Queue: Manoj Gupta <manojgupta@chromium.org>
Tested-by: Manoj Gupta <manojgupta@chromium.org>
Trybot-Ready: Manoj Gupta <manojgupta@chromium.org>

[delete] https://crrev.com/0b48be67c440831d8c602f344fc91ea46e11e166/overlay-variant-peach-pit/profiles/base/package.use

regarding #2

vapier@ 
We did add package.use entry for both private and public:
https://chrome-internal-review.googlesource.com/c/327007/
https://chromium-review.googlesource.com/434498

but it did not work. Do you understand why?
the peach-pit builder is chrome_pfq (not chromium)
I assume the CQ builders use chrome (not chromium), so why this is not matching?

We want to learn how to do this so that we can make a single ARM64 board to use "clang:.
But this is turning out to be more difficult that we thought.

i'm suggesting adding it *only* to the internal overlay so it'd impact the Chrome build and not the Chromium build.  i'm basing that theory off the error in the Binhost test:

AssertionError: When we take out config-requested useflags (['chrome_internal']), peach_pit-paladin cannot find Chrome prebuilts -- _CompatId(arch='arm', useflags=('accessibility', 'autotest', 'build_tests', 'buildcheck', 'chrome_debug', 'chrome_remoting', 'clang', 'cups', 'evdev_gestures', 'fonts', 'gold', 'hardfp', 'highdpi', 'nacl', 'neon', 'opengles', 'ozone_platform_default_gbm', 'ozone_platform_gbm', 'runhooks', 'v4l2_codec', 'xkbcommon'), cflags=('-O2', '-O2', '-pipe', '-march=armv7-a', '-mtune=cortex-a15', '-mfpu=neon', '-mfloat-abi=hard', '-g', '-fno-exceptions', '-fno-unwind-tables', '-fno-asynchronous-unwind-tables'))

thanks we will give this a try.

lowering severity to this bug since the master PFQ is not failing anymore due to this.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-variant-peach-pit-private/+/671ba2b346b2ba70d3ecb24897de3aa9a745ee06

commit 671ba2b346b2ba70d3ecb24897de3aa9a745ee06
Author: Yunlian Jiang <yunlian@google.com>
Date: Tue Feb 14 08:06:58 2017

we are having some more issues:

To avoid problems, we chose to migrate Chrome build to LLVM in some boards that are not part of the PFQ or CQ. This seemed to work (after some help from vapier).
We know have squawks, terra, veyron_jaq and peach_pi building chrome with LLVM.

However, we have now found that, if you want to change the overlay for any of those boards, you can't because the preCQ fails saying there are no chrome prebuilds for that board.
See: https://chromium-review.googlesource.com/442647

It seems to me that the only solution is to add some more boards to the PFQ that will generate prebuilds for the boards that we are migrating.

I have a preliminary change for this at:

https://chromium-review.googlesource.com/c/441993/

however, we are adding 5 boards to the ChromePFQ. We are trying to see if we can reduce the set (if there are some that are reduntant).
I marked those as non-important and the would not be doing any testing. Just generating prebuilds.

Is this the correct direction?

I haven't been following closely, but I assume there is good reason to switch some boards and not others (within an architecture type, i.e. arm/x86/amd64)? 

That sounds... difficult to maintain.

The basic rules are:
* All non-release or non-PFQ builds require Chrome prebuilts that match the requirements of their overlays.
* The PFQ builders are the only things that publish Chrome prebuilts.

binhost test ensures that we meet those requirements.

So... one solution is to add more builders to the PFQ.

Another is to make the overlay changes in such a way that the PFQ builders are the first builder to apply them (for example, check the Chrome version, and make it one higher than the current version).

I have no core objection to adding more PFQ builders. Though:
* Get gardener sign off
* Ensure we have lab capacity
* Have a plan to shut them down when the transition is over.
* Understand the delays and waterfall restarts involved in bringing up/shutting down PFQ builds.

If this is a short term solution, i.e. we add them for a week or two then
remove them, I am OK with that, just notify chrome-os-gardeners@google.com.

about #9 (stevenjb): we are just migrating Chrome to build with LLVM for a few boards as a means of testing (dogfooding). We use those to check for increased crashes/bugs before we migrate ALL the boards. 
We did this in the past to migrate non-chrome packages and it worked fine. Just that it is more difficult to deploy when doing it for Chrome.
about #11: we are moving these boards to build chrome with LLVM before the branch point. These boards will be built this way for 58. After branch point for 59, we will enable for ALL boards and we will remove these temporary boards from the PFQ.

about #10 (dgarrett@)

ok, we have figured out the minimal set of PFQ boards to add (terra and veyron_jaq).

Is it ok to add them as "build" only and "not important".
Will add you to reviewer in a few minutes.

SGTM, thanks!

The latest master pfq still fails for peach_pit.
We landed a patch to use llvm to build chrome for peach_pit.

After the peach_pit-pfq finished with my change, (it uses llvm to build chrome)
the latest master pfq started to build and it still fails. It complains it
could not find the prebuilts. 
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/host/source/chromite/lib/cros_test_lib.py", line 283, in _stacked_tearDown
    target(obj)
  File "/mnt/host/source/chromite/cbuildbot/binhost_test.py", line 65, in tearDown
    self.assertFalse(self.fatal_complaints, '\n'.join(self.fatal_complaints))
AssertionError: When we take out config-requested useflags (['chrome_internal']), peach_pit-paladin cannot find Chrome prebuilts -- _CompatId(arch='arm', useflags=('accessibility', 'autotest', 'build_tests', 'buildcheck', 'chrome_debug', 'chrome_remoting', 'clang', 'cups', 'evdev_gestures', 'fonts', 'gold', 'hardfp', 'highdpi', 'nacl', 'neon', 'opengles', 'ozone_platform_default_gbm', 'ozone_platform_gbm', 'runhooks', 'v4l2_codec', 'xkbcommon'), cflags=('-O2', '-O2', '-pipe', '-march=armv7-a', '-mtune=cortex-a15', '-mfpu=neon', '-mfloat-abi=hard', '-g', '-fno-exceptions', '-fno-unwind-tables', '-fno-asynchronous-unwind-tables'))

ok, according to what dgarrett@ mentioned to me yesterday, the prebuilts for a PFQ board will not become available until the master PFQ passes. 

So, we wanted to migrated peach_pit to LLVM/clang but peach_pit was already in the PFQ. So it seems we are in a chicken and egg problem since Master PFQ will not pass because there are no prebuilts with clang for peach_pit. 

two possible solutions:
1) instead of migrating peach_pit, migrate peach_pi which is not in the list of boards in the PFQ. Disadvantage: we would need to add peach_pi as experimental and then restart the waterfall. 
2). make peach_pit experimental for a few hours until the master PFQ passes once? would this work?

dgarrett@?  please advise. 

Obsolete