From Ilja:

Which suite does the Android PFQ run? It needs to be a superset of bvt-cq.

Also, it needs to cover the exact same boards.

The current Android PFQ can be seen here:

https://uberchromegw.corp.google.com/i/chromeos/waterfall?builder=master-android-pfq&builder=cyan-android-pfq&builder=glados-cheets-android-pfq&builder=samus-android-pfq&builder=veyron_minnie-android-pfq&titles=off&reload=30

It builds Cyan, glados-cheets, samus, veyron_minnie.

Offhand, it's not running any HWTests at all.

The build configs are calling for "sanity", and "arc-bvt-cq" to run, but the tests aren't actually showing up in build results. That confuses me.

https://uberchromegw.corp.google.com/i/chromeos/builders/cyan-android-pfq/builds/738

Yes, I only see sanity with basically no test in it
https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-android-pfq/builds/724/steps/HWTest%20%5Bsanity%5D/logs/stdio


So, as discussed arc-bvt-cq was supposed to be killed off and instead bvt-cq used. Somehow this was probably only half way executed.

Lets make sure Android PFQ starts running bvt-cq.

Notice that arc-bvt-cq should still at a minimum contain these tests, which I don't see running above, so it may be mostly defunct?

./cheets_SELinuxTest/control:ATTRIBUTES = "suite:arc-bvt-cq, suite:bvt-cq"
./cheets_SettingsBridge/control:ATTRIBUTES = "suite:arc-bvt-cq, suite:bvt-cq"
./cheets_DownloadsFilesystem/control:ATTRIBUTES = "suite:arc-bvt-cq, suite:bvt-cq"
./cheets_KeyboardTest/control:ATTRIBUTES = "suite:arc-bvt-cq, suite:bvt-cq"
./cheets_ContainerSmokeTest/control:ATTRIBUTES = "suite:arc-bvt-cq, suite:bvt-cq"
./cheets_NotificationTest/control:ATTRIBUTES = "suite:arc-bvt-cq, suite:bvt-cq"

1) I think something is wrong with the builder class driving these builds that's preventing the HWTest stage from running. That may be a difficult bug.

2) Updating the suites to run is easy, code wise, but what will the relative load on the lab be? Will that change be a problem?

I am still wondering if this issue only affected veyron_speedy or all boards as discussed in b/32660818. We should know by Monday.

I did investigation on b/32660818 today and now I'm fairly strongly thinking the veyron_speedy was the same one that affected all boards, as Ilja suspected.

What looked went wrong to me is 1) Android PFQ was not a super set of CQ, and 2) Minnie/Cyan/Samus CQ did not run the CTS test that run on Speedy CQ (the former three bots are watched more carefully by ARC folks; IIU veyron_speedy is not shipped with ARC enabled, so it's strange to have the test only there...)

> 1) Android PFQ was not a super set of CQ
Oops, sorry I wanted to mean was:
1) Chrome PFQ was not a super set of CQ

the veyron_speedy was due to a bad Chrome roll, if my suspect is correct.

This issue boils down to auditing and making consistent
~/trunk/chromite/cbuildbot/config_dump.json

Since the bvt-cq runs the arc tests now, it sounds like what we want to do is deprecate arc-bvt-cq entirely (and replace with just bvt-cq). The presumption was this was not something we needed to rush for the Android PFQ as any container related tests existed within arc-bvt-cq already. 

We do still run HWTests against a RK3288 based system with minnie https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-android-pfq/builds/755/steps/HWTest%20%5Barc-bvt-cq%5D/logs/stdio and this should represent speedy well, however it is still just arc-bvt-cq. 

If we want to be more consistent, perhaps we should combine the android and chrome PFQs into one super PFQ? This is essentially what we do on the release branch (though that does not do much in the way of testing, it is more of a sanity build). This might increase our flake rate, but it would guarantee that a Chrome and Android combination works together, and are validated against a wider set of devices than the Android PFQ handles today, and it would make lab loading on these tests lower/more consistent.

I see two options:
1) Run bvt-cq everywhere.
This is easy to explain and maintain, but it is a bit wasteful.
2) Split bvt-cq into three suites depending on which level in stack they are testing"
a) bvt-cq-os - tests that are not impacted by changes higher in the stack (example graphics_dEQP etc.)
b) bvt-cq-browser - tests that are impacted by os and browser changes (example graphics_WebGLAquarium)
c) bvt-cq-android - test that are impacted by any change (example cheets_CTS)

Traditional bvt-cq = [bvt-cq-os + bvt-cq-browser + bvt-cq-android]
This needs to run for every cq commit.

For every Chrome uprev we would need to run [bvt-cq-browser + bvt-cq-android]
For every Android uprev we would need to run [bvt-cq-android]

Of course we could also have something hybrid, where each test would tag itself by DEPENDENCIES = 'os', DEPENDENCIES = 'android' and DEPENDENCIES = 'browser', but that will need new logic and I'm not sure I would trust it.

I've just put up a series of CLs intended to treat testing of cheets boards more similarly to how we treat other boards. It replaces arc-bvt-cq with bvt-cq, does doesn't adjust the suites any further.

Here's the link to the initial CL. I'm not confident that they are all a good idea, but love removing special casing for cheets in our configs.

https://chromium-review.googlesource.com/#/c/408780/

BTW: The Android PFQ IS correctly running HWTests. I was confused because builds which find there is nothing new to test exit early.

Build running tests:
  https://uberchromegw.corp.google.com/i/chromeos/builders/cyan-android-pfq/builds/771

There is exactly one test in that link's HWTest, namely dummy_PassServer.sanity of suite:sanity. It sounds like that is misconfigured?

It has two HWTest stages. The first is sanity, and the second is arc-bvt-cq.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/379ab587a1c7f05f2536f4d4ed29406ce835a614

commit 379ab587a1c7f05f2536f4d4ed29406ce835a614
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Fri Nov 11 00:46:32 2016

autotest: arc-bvt-cq -> bvt-cq transition.

Make sure every test which is in arc-bvt-cq is also in bvt-cq.
(bvt-inline and bvt-cq are for practical purposes equivalent.)

BUG= chromium:662527 
TEST=None.

Change-Id: I0beb944f0f1bd24e02d2d5005d2a625646b93a04
Reviewed-on: https://chromium-review.googlesource.com/410401
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Rohit Makasana <rohitbm@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>

[modify] https://crrev.com/379ab587a1c7f05f2536f4d4ed29406ce835a614/client/site_tests/security_NetworkListeners/control