For all that is worth betty-paladin and it is doing well:
https://uberchromegw.corp.google.com/i/chromeos/builders/betty-paladin?numbuilds=200

How would betty-tot-paladin differ?

I'd run it on GCE instead of bare metal, and it doesn't have any changes by being virtue of -tot so it eliminates an untested change as a potential culprit.

Can we have this builder run on GCE to rule out GCE as the cause of the failures?
The current paladin runs on bare metal builders, which might explain why it's more reliable than the pre-cq

I feel it may be a performance issue, but need to verify. So it could be GCE vs. bare metal. We will see.

It sounds like betty-incremental-informational might be a better choice, with SoM being aware of that build.

Most likely the config for this already exists. Will confirm.

cros tryjob --remote betty-incremental

produces:
https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/incremental/675
(still runnning)

Can we add it to the regular scheduled builds on a GCE instance?

Yes, that is the task here. I'm just doing some sanity checking first.

Wait a second.

There already is a betty-incremental on the internal waterfall.

https://uberchromegw.corp.google.com/i/chromeos/builders/betty-incremental

For some reason it isn't being triggered though.

Oh and it has the wrong bot type, you want gce.

Let's tackle the triggering problem first, Issue 796445.

Ok, it's triggering, and running, and failing VMTest quite regularly. https://uberchromegw.corp.google.com/i/chromeos/builders/betty-incremental

This isn't working well enough to put onto SoM I don't think yet.  Adding current sheriffs dianders/ahassani to take a look if they have some time.

Any chance this is a dupe of  bug #797109  ?

It is not a dupe, as this is about adding a builder.

Currently running on build194-m2 which is a baremetal machine. I think the remaining action on this is to use a gce instance instead of a baremetal. Correct?

A relevant change (more relevant to GCE) landed last night.

If we can avoid moving to GCE, I'd prefer it. We only have 21 remaining IPs that are whitelisted for KVM support, and I'm hoping to turn them all into swarming builders.

-> dgarrett What's the latest status of GCE and IP address capacity? (bounce back to me when clarified)

===Pool chromeos_east===
Used: 237
Free: 19

===Pool chromeos_central===
Used: 499
Free: 13


Only the builders in chromeos_east have KVM support, but that is expected to launch globally in about a week.

We have a little more slack than that shows since several waterfalls have unallocated builders already assigned, and there are 40 builders assigned to swarming, which is more than we currently need.

https://chromium-review.googlesource.com/#/c/chromiumos/chromite/+/957481/

blocked on https://ariane.googleplex.com/launch/214948

Passing the buck to jclinton@. This is still probably blocked on external launch?

Nope. All of our build machines are currently opted into the VMTest beta.

We're moving VM testing to Skylab so is it worth doing this at all right now?

chromeos waterfall currently has 7 unallocated GCE builders.

So... all implementing this only requires an update to chromeos_config, then waiting for a waterfall restart.

No big deal either way.

If it's ready with a small config change, then probably worth doing now. VMTest moving to Skylab is looking like Q3.

Alec, this is a simple config update and a good first bug for your first week oncall. Let's chat about it when you have time.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/fb6f9736373855b218598ff92072ac41d3afa6c1

commit fb6f9736373855b218598ff92072ac41d3afa6c1
Author: Alec Thilenius <athilenius@google.com>
Date: Fri Aug 03 18:03:18 2018

Move betty-inremental from baremetal to GCE

BUG= chromium:795976 
TEST=None

Change-Id: I1e2d79423b24e9762cf0343a94a7fc34fed5003f
Reviewed-on: https://chromium-review.googlesource.com/1076674
Commit-Ready: Alec Thilenius <athilenius@google.com>
Tested-by: Alec Thilenius <athilenius@google.com>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/fb6f9736373855b218598ff92072ac41d3afa6c1/config/chromeos_config.py
[modify] https://crrev.com/fb6f9736373855b218598ff92072ac41d3afa6c1/config/waterfall_layout_dump.txt
[modify] https://crrev.com/fb6f9736373855b218598ff92072ac41d3afa6c1/config/config_dump.json

crrev.com/c/1076674 is in, waiting to close this till the weekend waterfall restart.