Does anyone know of a reason why we wouldn't want to use VMs for these tests instead of real hardware? (Except Android)

#1: Regarding flakiness, I
I totally agree flakiness is so annoying when someone really wants to land a CL.
As per the jobs on the issues I see that most of failed jobs on android_s5_perf_cq are either test crash or compile failure(with and without both failed.) Also we need to investigate if this is something related to specific tests that are failing, because most of the test get passed. Secondly the hardware on which these test are run.

#2: There is no specific reason why we are using real hardware, but I think whenever we put a request to Infra for more hardware, we ask for 5 slaves (3 for bisect and 2 for Perf try jobs with real hardware). And when we decided to create dedicated bots for CQ we borrowed slaves from the bots where we had 5 slaves.

Happening again today: https://codereview.chromium.org/1914543003/

I think the job timed out on s5 cq bot, may be due to a long pending queue on the master. Please note that we only have one slave on each of these perf CQ bots

There is this job running for the same CL on the CQ
https://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_cq/builds/164
Will keep an eye if it gets through!