Need product input about what to replace caroline-pre-cq with.
Punted to next week.

We can add a caroline paladin, if it is just building that is probably a good idea to do, not clear if we can afford the hardware to support it for HWTests in the cq though.

If we give HWTests to reef with https://chromium-review.googlesource.com/#/c/455991/ that will cover a lot of what might be able to break HWTest wise on caroline (same kernel/ARC setup, though different SoC), that is blocked by https://bugs.chromium.org/p/chromium/issues/detail?id=688585

Cave with HWTest on the cq will also help here https://bugs.chromium.org/p/chromium/issues/detail?id=708313 , cave and caroline are essentially the same on the inside (SKL-Y).

If we decide we want the paladin, first step is https://chromium-review.googlesource.com/#/c/471835/

caroline-pre-cq was added because of  crbug.com/695638 . To replace caroline-pre-cq, you need to have a bort with USE=vulkan. ihf@/zachr@, ideas?

We need Intel, one of the new boards. chell is probably a good stable choice, it is on Android M though. Less stable and on N would be reef, snappy, pyro, asuka.

Chances are that chell has the same problems as caroline though ( issue 708715 ). Not sure if reef, snappy, pyro, asuka would do better.

Could just be that there is no better choice than marking caroline no-vmtest (caroline-no-vmtest-pre-cq). That will provide build coverage, but skip vm which is boring to some degree, except for some Android M sanity tests.

Just to clarify (re #6),  crbug.com/695638  will be avoided just by building the image (=building vulkan).

Ok, here is my suggestion
https://chromium-review.googlesource.com/#/c/472069/1/lib/constants.py

Have you run trybots to verify these two builds?  Do you have timings for samus-pre-cq?

The CL to move vmtest from caroline to samus was merged on Apr 7: https://chromium-review.googlesource.com/c/472069/

I'm still getting alerts about caroline-pre-cq failing (due to VMTest, I spot checked a few of the failures).

I believe this is because we never update the set of pre-cq-configs to be used to vet a CL once they've been determined. So, CLs that were being tested with caroline-pre-cq will continue to be tested with it.
eyeballing the alert list, the failures are becoming less frequent and should tail off on their own.

Actually, the immediate problem has been fixed: pre-cq is no longer borked, but the OP is still valid -- caroline-pre-cq doesn't have a corresponding important paladin.

Of the boards listed in #8, reef is the only one with an important paladin.
So, we should also replase caroline-no-vmtest-pre-cq with reef-no-vmtest-pre-cq

ihf, sounds good?

I don't see samus running vmtest
https://uberchromegw.corp.google.com/i/chromeos/builders/samus-paladin/builds/13444

Does the #11 change need deployment?

To your question, not sure what an important paladin does. But replacing caroline-no-vmtest-pre-cq with reef-no-vmtest-pre-cq should be fine.

By the way, it looks like there is a login failure on newer Intel platforms affecting hwtest starting at 9438. b/37189798

Samus is not affected by b/37189798 though.

Re #15: filed  issue 710169  to run vmtest on samus

Having an important paladin makes sure that
(1) CLs not tested by caroline-pre-cq don't break caroline-pre-cq
(2) We have a clearer signal when caroline-pre-cq fails (currently, we get metrics / alerts about many pre-cq failures, but the paladin failing will get quicker attention)

So I added caroline for bvt-cq in
https://chromium-review.googlesource.com/#/c/509051/

It is mostly purple though, which seems due to excessive build_packages times. The hwtests run reasonably (provision 6 DUTs + about 25 minutes for tests).
https://uberchromegw.corp.google.com/i/chromeos/builders/caroline-paladin

We really glados to not be experimental. cave is also experimental in issue 722961
and we need one more board for arc-bvt-cq to fill the family row in _paladin_hwtest_assignments
https://chromium-review.googlesource.com/#/c/522746/4/cbuildbot/chromeos_config.py

Regarding build packages be slow, we looked at one master run and caroline was 41 minutes on build packages:
https://viceroy.corp.google.com/chromeos/build_details?build_config=caroline-paladin&build_number=169
But samus-paladin from the same master run was 9 minutes:
https://viceroy.corp.google.com/chromeos/build_details?build_config=samus-paladin&build_number=14019

I think this might be from running on bare metal instead of GCE.

Can we move this builder to GCE 64 core?

Based on the current waterfall_layout_dump it appears caroline-paladin should be mapped to a gce_beefy.

The last few runs were on a gce_beefy builder https://uberchromegw.corp.google.com/i/chromeos/builders/caroline-paladin/builds/173 not clear why it was using a baremetal before.

caroline-pre-cq is no longer a default pre-cq builder, and
caroline-paladin is important.