Starting with this build:
https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/11946

CommitQueueCompletion CommitQueueCompletion ( 39 mins, 12 secs )
stdio
veyron_pinky-paladin: The BuildPackages stage failed: Cannot find prebuilts for chromeos-base/chromeos-chrome on veyron_pinky
veyron_mighty-paladin: The BuildPackages stage failed: Cannot find prebuilts for chromeos-base/chromeos-chrome on veyron_mighty
veyron_minnie-paladin: The BuildPackages stage failed: Cannot find prebuilts for chromeos-base/chromeos-chrome on veyron_minnie
veyron_speedy-paladin: The BuildPackages stage failed: Cannot find prebuilts for chromeos-base/chromeos-chrome on veyron_speedy

Note that veyron_rialto variant succeeded. Rialto does NOT use arc++ and thus this looks related to the *-cheets turn down for products which are already including ARC++ (and thus might be getting their prebuilts from -cheets builders which were just removed).

10:37 grundler  ok. So all the veyron_*-paladins failed (except -rialto) due to missing chromeos-chrome prebuilts. Who owns this?
10:37 grundler  stevenjb: is this a chrome builder problem or a problem related to -cheets turn down?
10:39 grundler  I'm asking because master-paladin has been failing since Aug 03 23:34 (I'm assuming due to this)
10:40 *  grundler  confirms - master-paladin failures are due to missing prebuilts in veyron variants
10:40 grundler  https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/11946
10:40 stevenjb  grundler: I don't really know, I will defer to @bhthompson
10:40 stevenjb  But seems likely.
10:41 grundler  I wonder if "sloppy" use of wildcards accidently removed those
10:42 grundler  bhthompson: can you double check please?
10:51 bhthompson  hmm if it is a normal paladin, it should not matter
10:51 bhthompson  there may be something wrong with veyron in general
10:52 bhthompson  i don't think they are related
10:53 bhthompson  does the chrome PFQ generate the prebuilts?
10:54 bhthompson  in any case we need to finish the restart, which will involve killing the current chrome PFQ run, and the toolchain run, if there are no objections I will do that so we can get the canary build started
10:57 stevenjb  bhthompson: Yes, the PFQ generates the prebuilts.
10:58 stevenjb  bhthompson: That sounds fine. We just had a good PFQ run this morning, that should keep the TPMs happy for a bit :)
10:58 bhthompson  so if the paladins failed finding a prebuilt it would mean that something likely broke in the PFQ ?
11:00 bhthompson  ill start axing the builds then so we restart
11:01 dgarrett  That veyron_mighty failure is interesting.
11:02 dgarrett  grundler: I can help with that, but we are supposed to have safeguards to prevent it.
11:02 stevenjb  I love it when Don uses the phrase "interesting" :)
11:02 dgarrett  bhthompson: Has there been a waterfall restart after your recent config changes?
11:03 dgarrett  Interesting as in.... not routine.
11:03 bhthompson  it should restart in a couple minutes
11:03 bhthompson  im cleaining up the last of the draining
11:03 amstan  grundler: rialto works because we have our own chrome builder
11:04 amstan  (even though we're headless, we do run chrome, no arc++)
11:04 dgarrett  Ah. 
11:04 dgarrett  There is a good chance this will be fixed on the next successful PFQ run after the restart.
11:04 dgarrett  But I'll try to confirm that.
11:06 bhthompson  waterfall is drained, it appears to be coming back up now
11:06 grundler  amstan: "no arc++" is exactly why I think the prebuilts weren't removed
11:07 dgarrett  Hum. It looks like no PFQ builders are being added or removed.
11:08 dgarrett  Does anyone know which PFQ builder veyron_mighty should get chrome prebuilts from?
11:08 grundler  actually, thinking about, I wonder if the regular builder was likely getting it's prebuilts directly from the -cheets flavor of the same variant...so removing the cheets builds deleted the prebuilts for the regular builds too.
11:08 amstan  i believe all veyrons are grouped together(besides rialto)
11:09 bhthompson  we should have removed the cyan samus and veryon_minnie -cheets builds from the PFQ, it looks like they are gone on the waterfall now as expected
11:11 dgarrett  Did we force a chrome uprev yesterday?
11:11 dgarrett  Josafat was considering it.
11:11 bhthompson  it revved naturally
11:12 dgarrett  I'm wondering if that process went badly for some reason.
11:13 dgarrett Veyron_pinky is on both the PFQ and the CQ, and the CQ failed because of no prebuilt.
11:13 dgarrett  Veyron pinky should have matched between the two.
11:13 dgarrett  So, we didn't use Ningning's uprev tool. Okay, it's in the clear. ;>
11:13 josafat  I did not do it manually .. it was a normal PFQ rev
11:15 josafat  https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/3207
11:15 dgarrett  She's going to look at it anyway based on her expertise.
11:16 dgarrett  Is there a bug for this yet? If not, can we get one?
11:16 dgarrett  And to confirm, all paladins since that PFQ run have been broken.
11:18 dgarrett  Also to ask.... did anyone manually delete any prebuilts? 
11:18 dgarrett  We normally never delete them.
11:27 dgarrett  PS: Local builds for those boards are also probably broken at TOT.