I've marked reef-paladin "experimental" until we can clean this up.

This was a transient ordering issue since 2 CLs needed to land on ToT and M64 in order to fix the electro/basking to reef model label transition.

Everything is now submitted, so it should start to clear on the next DUT runs.

> Everything is now submitted, so it should start to clear on the next DUT runs.

It's not that simple.  Re-labeling only happens when DUTs are
scheduled for testing.  When the electro DUTs lost their model
label, they became ineligible for testing.  So, the system can't
automatically re-label them.

I'm working to figure out a proper answer, which will probably
involve manually relabeling...

I've relabeled all "model:reef" DUTs in the bvt, cq, and suites
pools:

$ M=$(echo $(for p in bvt cq suites; do atest host list -b model:reef,pool:$p; done | awk '/^chromeos/ {print $1}') | sed 's/ /,/g')
$ atest label remove -m $M model:reef 
Removed from label 'model:reef' hosts: 
        chromeos6-row3-rack10-host11, chromeos6-row3-rack10-host3,
        chromeos6-row3-rack12-host11, chromeos6-row3-rack12-host13,
        chromeos6-row3-rack12-host17, chromeos6-row4-rack10-host1,
        chromeos6-row4-rack10-host13, chromeos6-row4-rack10-host19,
        chromeos6-row4-rack10-host20, chromeos6-row4-rack10-host3,
        chromeos6-row4-rack10-host7, chromeos6-row4-rack9-host10,
        chromeos6-row4-rack9-host11, chromeos6-row4-rack9-host12,
        chromeos6-row4-rack9-host13, chromeos6-row4-rack9-host15,
        chromeos6-row4-rack9-host17, chromeos6-row4-rack9-host22,
        chromeos6-row4-rack9-host4, chromeos6-row4-rack9-host5,
        chromeos6-row4-rack9-host6, chromeos6-row4-rack9-host7,
        chromeos6-row4-rack9-host8, chromeos6-row4-rack9-host9
$ atest label add -m $M model:electro 
Added to label 'model:electro' hosts:
        chromeos6-row3-rack10-host11, chromeos6-row3-rack10-host3,
        chromeos6-row3-rack12-host11, chromeos6-row3-rack12-host13,
        chromeos6-row3-rack12-host17, chromeos6-row4-rack10-host1,
        chromeos6-row4-rack10-host13, chromeos6-row4-rack10-host19,
        chromeos6-row4-rack10-host20, chromeos6-row4-rack10-host3,
        chromeos6-row4-rack10-host7, chromeos6-row4-rack9-host10,
        chromeos6-row4-rack9-host11, chromeos6-row4-rack9-host12,
        chromeos6-row4-rack9-host13, chromeos6-row4-rack9-host15,
        chromeos6-row4-rack9-host17, chromeos6-row4-rack9-host22,
        chromeos6-row4-rack9-host4, chromeos6-row4-rack9-host5,
        chromeos6-row4-rack9-host6, chromeos6-row4-rack9-host7,
        chromeos6-row4-rack9-host8, chromeos6-row4-rack9-host9

That's likely to be close enough to right to allow things to go
forward.

Last night I marked the reef-paladin builder as experimental.
The builder never went red (I don't know why), but we're going
to leave it experimental until we can confirm that the builder
actually runs and passes tests.

For reasons I cannot explain, the two most recent reef-paladin
runs chose to test against "--model reef" instead of "--model electro":
    https://uberchromegw.corp.google.com/i/chromeos/builders/reef-paladin/builds/4583
    https://uberchromegw.corp.google.com/i/chromeos/builders/reef-paladin/builds/4582

So, we're still not testing hardware on reef-paladin, and it'll
stay experimental until we can get this sorted.

That's because we changed the model to be reef and updated GE accordingly.
I think the work in comment 5 was backwards.... we wanted all electro/basking to get labeled as reef instead.

> I think the work in comment 5 was backwards.... we wanted
> all electro/basking to get labeled as reef instead.

Have we made all the chromite/GE changes necessary?  Most
especially, have we merged any chromite changes to the
release branches?  Once the label changes are made, any
builders that still say "--model electro" will quit testing,
and may fail.

ok ... did the inverse on comment 5 to make everything model:reef (thanks for the commands).

next run should be reasonably better

GE updates run hourly and hit all of the branches, so yes to comment 9

> ok ... did the inverse on comment 5 to make everything model:reef
> (thanks for the commands).

As of right now, there are still quite a few "electro" and "basking"
DUTs:

    $ atest host list -b board:reef | count_labels -m
          8 basking
         23 electro
         38 reef

sure ... go ahead and do whatever is right here

the net result is everything should be model:reef

this was the goal to resolve the previous loading issues across electro/basking

$ M=$(echo $(atest host list -b board:reef | grep -v model:reef | awk '/^chromeos[246]-/ {print $1}') | sed 's/ /,/g')
$ atest label remove -m $M model:basking model:electro
Removed from label 'model:electro', 'model:basking' hosts: 
	chromeos6-row3-rack10-host1, chromeos6-row3-rack10-host13,
	chromeos6-row3-rack10-host15, chromeos6-row3-rack10-host17,
	chromeos6-row3-rack10-host9, chromeos6-row3-rack12-host1,
	chromeos6-row3-rack12-host11, chromeos6-row3-rack12-host15,
	chromeos6-row3-rack12-host19, chromeos6-row3-rack12-host3,
	chromeos6-row3-rack12-host5, chromeos6-row3-rack12-host7,
	chromeos6-row3-rack12-host9, chromeos6-row4-rack10-host10,
	chromeos6-row4-rack10-host12, chromeos6-row4-rack10-host14,
	chromeos6-row4-rack10-host15, chromeos6-row4-rack10-host16,
	chromeos6-row4-rack10-host4, chromeos6-row4-rack10-host6,
	chromeos6-row4-rack10-host8, chromeos6-row4-rack9-host1,
	chromeos6-row4-rack9-host14, chromeos6-row4-rack9-host18,
	chromeos6-row4-rack9-host19, chromeos6-row4-rack9-host2
$ atest label add -m $M model:reef
Added to label 'model:reef' hosts: 
	chromeos6-row3-rack10-host1, chromeos6-row3-rack10-host13,
	chromeos6-row3-rack10-host15, chromeos6-row3-rack10-host17,
	chromeos6-row3-rack10-host9, chromeos6-row3-rack12-host1,
	chromeos6-row3-rack12-host11, chromeos6-row3-rack12-host15,
	chromeos6-row3-rack12-host19, chromeos6-row3-rack12-host3,
	chromeos6-row3-rack12-host5, chromeos6-row3-rack12-host7,
	chromeos6-row3-rack12-host9, chromeos6-row4-rack10-host10,
	chromeos6-row4-rack10-host12, chromeos6-row4-rack10-host14,
	chromeos6-row4-rack10-host15, chromeos6-row4-rack10-host16,
	chromeos6-row4-rack10-host4, chromeos6-row4-rack10-host6,
	chromeos6-row4-rack10-host8, chromeos6-row4-rack9-host1,
	chromeos6-row4-rack9-host14, chromeos6-row4-rack9-host18,
	chromeos6-row4-rack9-host19, chromeos6-row4-rack9-host2

$ atest host list -b model:reef | count_labels -p
     10 arc-presubmit
      6 bvt
      8 cq
      1 cr50_stress
      1 crosperf
     18 cts
      1 performance
      2 stress
     16 suites
      3 wificell

Those look like good pool counts, so now, all we need is to see a
few builders walk through their paces.  My guess is that we won't
have that kind of result until after close-of-business 12/20, but
Lord willin', this can get marked done tomorrow first thing.

The reef-paladin builder is now no longer experimental

Seems like there's nothing left to do.