PoHsien has been investigating similar issues. I think smoke could probably be split into 4 pieces. That said stability is unclear at that stage. Might be worth setting up an informational builder which tries this for a while first and then compare against betty.

Could we use a single PreCQ builder, but start 4 VM instances and shard the tests across them?

IE: Run all the tests on the same builder, but finish faster through the power of many CPUs?

I have done experiment on VM for suite:arc-cts as they run much longer than normal smoke tests. The experiment run reduced the total amount of run time from 17hr to 10hr if run with 2VM. Would expect more if we shard it more.

Re 3: yes, a builder could easily run 4 VM instances. It might run even 8 or 16, but we could start with 4.

I'll take over it to see if we can make smoke suite run in parallel.

Manually split the suite and test on trybots/locally. 
Tested on cros-standard181-c2 with four suite in parallel takes ~40 mins. Running it as a whole takes ~50 minutes.

And observed that the Android bring up from 4 VMs got extremely slow which takes 5 minutes.

So, you are saying because of increased Android startup times we don't see much runtime improvement right now?

Is it bounded by the performance of the GCE? If so, we can upgrade the GCE to beefy for all vmtest pre-cqs if needed.

No, there is likely a bug impacting establishing communication to multiple Android instances using adb. Could be a networking configuration issue.

Suite1 14tests: >50minutes with 10 minutes waiting for android
Suite2 14tests:  25minutes with  5 minutes waiting for android
Suite3 14tests:  30minutes with 10 minutes waiting for android
Suite4 14tests:  31minutes with  5 minutes waiting for android

So, with good splitting mechanism, we could gain and make VMTestStage finish in 30 minutes.

IIUC, each VM are already in different virtual network. https://chromium-review.googlesource.com/c/chromiumos/chromite/+/780726 but may have other problem in networking.

Wow, this is very disappointing. Lets sit down together and see what is happening with the GCE instance.