This might have a conflict - we were planning on using the RandomOrder bots as a testing ground for swarming. Possible options:

1. Spin up 3 new bots for default order:
  - WebKit Linux - DefaultOrder
  - WebKit Mac - DefaultOrder
  - WebKit Win - DefaultOrder
2. Spin up 3 new bots to be the swarming guinea pigs:
  - WebKit Linux - Swarming
  - WebKit Mac - Swarming
  - WebKit Win - Swarming
3. (As originally stated) convert all RandomOrder bots to DefaultOrder bots, which leaves us without swarming testing grounds.

Not sure what the best choice is - CCing relevant people.

Pinging qyearsley & tansell - need your input on this.

How much effort are options (1) or (2) ?

IIUC, there are about 160 tests which are still in RandomOrderExpectations.

In the medium term we do need to either fix or delete these tests, since there's no way we're going to be able to enforce they keep passing if it's not part of the real CQ.

mcgreevy, do you have thoughts?

I don't think they'd be too much effort, we just need to request the builders and create an infra CL. Since we don't have a ton of time I'd advocate for #2 so we can switch the current builders over to being DefaultOrder now.

Update from our discussion: tl;dr we're going with #2.

I'm going to go ahead and create a CL to transition all RandomOrder builders to be DefaultOrder. At the same time I'm going to request VMs for swarming bots. I'm going to try to get the Linux swarming bot up first since that should be able to be provisioned the fastest, and the Mac and Win builders up as they're ready.

Just a note, this will probably entail 2 CLs: one for Linux & Mac and a later one for Windows.

I just realized that the name 'DefaultOrder' isn't appropriate because random will be the new default. I think NaturalOrder makes more sense since that's the flag value (--order=natural).

Additionally, the NaturalOrder bots won't need to run the full layout test suite. They only need to run the small subset tests that have been skipped because of random order flakiness. I'll create a separate bug for that.

Update: we're not going to providing NaturalOrder bots. The Linux & Mac RandomOrder bots are ready to use as swarming testing grounds.

OK, cool. Is the reason for not providing NaturalOrder bots discussed elsewhere?

See dpranke's 2nd comment in this thread: https://chromium-review.googlesource.com/#/c/420850/