We have https://uberchromegw.corp.google.com/i/chromeos/builders/terra-chrome-pfq, it that good enough?

No, not at all.

The paladins test that new CLs won't break the board in question, and should never be experimental, except during bring up or temporary failures (which are very high priority to fix). If it's always read today, something is wrong with that board.

There should never be a release builder for an experimental paladin.

> The paladins test that new CLs won't break the board in question

Reinforcing this answer:  The terra-chrome-pfq builder is also
experimental.  Because of that, when it broke a couple of weeks
ago, that builder went red, but _Chrome still uprevved_.  That
caused the terra release builder to go red with the failure, and
a DOA build went out to some dev-channel users.  THAT WAS BAD,
and a key contributor is that both the PFQ and Paladin were
experimental.

It's possible that the lack of pre-builts was a consequence of the
red builds that were present at the time this bug was filed; dgarrett@
will know more.

Both the paladin and the PFQ need to be non-experimental.
Escalating, now that I've realized just how bad this problem really is.

Yes, red builds that are experimental can result in missing prebuilts.

Marking the PFQ as non-experimental, which may help the paladin.

Oh... it's being used for thinlto testing. Assigning to yunlian@ to sort out what should change here to address this.

Who else owns this board?

I uploaded https://crrev.com/c/1000018 as a strawman for discussion.

My revised CL marks the chrome pfq builder as green, which should fix the paladin. If that works, we can mark the paladin as important.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/f21981e56d8bec7b5fa273dbffcad3c0ee232595

commit f21981e56d8bec7b5fa273dbffcad3c0ee232595
Author: Don Garrett <dgarrett@google.com>
Date: Sat Apr 07 05:00:17 2018

chromeos_config: Mark terra-chrome-pfq important.

Terra is a publically released board, without paladin or chrome_pfq
testing. Enable chrome-pfq testing (it's green today). After this is
in place, and we get a chrome uprev, we will have a chrome prebuilt
for terra USE=thinlto.

In theory, the paladin will go green, and we should then mark it as
important.

BUG= chromium:824482 
TEST=chromeos_config_unittest

Change-Id: I07f7e11d59e8f34426c18508d46a8a49a1b9cf06
Reviewed-on: https://chromium-review.googlesource.com/1000018
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>

[modify] https://crrev.com/f21981e56d8bec7b5fa273dbffcad3c0ee232595/cbuildbot/config_dump.json
[modify] https://crrev.com/f21981e56d8bec7b5fa273dbffcad3c0ee232595/cbuildbot/chromeos_config.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/65fdfe646c39ff6f2ec27c76e31abc923a8130a3

commit 65fdfe646c39ff6f2ec27c76e31abc923a8130a3
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Sat Apr 07 07:59:29 2018

Roll src/third_party/chromite/ 188669bb7..f10b48279 (4 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/188669bb7526..f10b482799b3

$ git log 188669bb7..f10b48279 --date=short --no-merges --format='%ad %ae %s'
2018-04-04 djkurtz grunt: mark grunt-paladin important
2018-04-06 dgarrett chromeos_config: Mark terra-chrome-pfq important.
2018-04-06 dgarrett chromeos_config: Move an eve to swarming.
2018-04-06 dgarrett chromeos_config: Helper for custom board_configs.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:824482 , chromium:829898 ,chromium:None


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: Ia44e90f0126d2b0fbf2aeb7b5c7ea80d48502ae2
Reviewed-on: https://chromium-review.googlesource.com/1000955
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#549051}
[modify] https://crrev.com/65fdfe646c39ff6f2ec27c76e31abc923a8130a3/DEPS

I'm still waiting on a green PFQ run to see if this will fix the paladin. The problems on the PFQ appear unrelated.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/0e6f95c50047972223589157e570cae436338777

commit 0e6f95c50047972223589157e570cae436338777
Author: Don Garrett <dgarrett@google.com>
Date: Wed Apr 18 21:02:17 2018

chromeos_config: Mark terra-paladin important.

Terra paladin is green, and it's needed to protect this board against
bad changes. Turn it on!

BUG= chromium:824482 
TEST=chromeos_config_unittest

Change-Id: I5234d1d4ac3113d7f99745abc8f1e93abf2e18b6
Reviewed-on: https://chromium-review.googlesource.com/1014656
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@google.com>

[modify] https://crrev.com/0e6f95c50047972223589157e570cae436338777/cbuildbot/config_dump.json
[modify] https://crrev.com/0e6f95c50047972223589157e570cae436338777/cbuildbot/chromeos_config.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4415c060668e8dcf9a2eef7b0c5da72043dfef52

commit 4415c060668e8dcf9a2eef7b0c5da72043dfef52
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Wed Apr 18 22:56:06 2018

Roll src/third_party/chromite/ 4e29b6223..0e6f95c50 (1 commit)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/4e29b62239da..0e6f95c50047

$ git log 4e29b6223..0e6f95c50 --date=short --no-merges --format='%ad %ae %s'
2018-04-16 dgarrett chromeos_config: Mark terra-paladin important.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:824482 


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I3ac131eea17885da2142ab25d7a5b7221c6702de
Reviewed-on: https://chromium-review.googlesource.com/1017880
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#551870}
[modify] https://crrev.com/4415c060668e8dcf9a2eef7b0c5da72043dfef52/DEPS