yshaul@ ... please take a look

Might be caused by  issue 905704 .

Yes, the bvt-installer failure prevents bvt-perbuild (and bvt-cq) from being kicked off (which is intended and good).

I was a little concerned this happened because of the new test plan stage, but it turns out this was always the case - it's just that installer doesn't fail all that often, so you may not have noticed the behavior. Check out  https://stainless.corp.google.com/search?view=matrix&row=builder_name&col=build&first_date=2018-11-26&last_date=2018-12-02&builder_name=-release%24&suite=%5Ebvt-installer%24&status=GOOD&status=WARN&status=FAIL&status=ERROR&status=ABORT&status=ALERT&status=RUNNING&status=TEST_NA&status=NOSTATUS&status=NOT_RUN&exclude_cts=false&exclude_not_run=false&exclude_non_release=false&exclude_au=false&exclude_acts=false&exclude_retried=false&exclude_non_production=false

Have a look at the failures for daisy_skate-release. It failed for releases R72-11300.0.0 and R72-11302.0.0 (prior to HW test stage), and there's a corresponding hole in the bvt-perbuild matrix.

If you would like bvt-perbuild to run irrespective of the installer result (for release builds), we can rearrange the config so that bvt-perbuild is listed before bvt-installer. Let me know.

Thanks for looking into this. I don't think we want to try to run other suites if bvt-installer fails.

Actually, I take that back. When I look at the falco-release build at http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8928107216625748720, we still ran the bvt-tast-cq ASyncHWTest suite.

I'd assumed that we were running bvt-installer first either to protect DUTs or because none of the other suites would have any chance of running successfully if the installer was broken. This query shows that the tests in bvt-tast-cq are still passing and running, though:

http://stainless/search?view=matrix&row=suite&col=build&first_date=2018-11-28&last_date=2018-12-04&builder_name=%5Efalco-release%24&branch=%5Emaster%24&test=%5Etast%5C..*%5C.&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false

Are there any downsides to running bvt-perbuild regardless of what happens in bvt-installer, as Yaakov suggested in #7? Alternately, I can look into running informational Tast tests in their own suite instead of as part of bvt-perbuild.

It does not surprise me that the json file is inconsistent.

Notice that bvt-installer runs 2 tests autoupdate_Rollback and platform_Powerwash. While these are probably not critical to the lab, they are an indicator that provisions are not going well. 

I think you can reorder the json to do less waiting and more fire and forget, see
https://bugs.chromium.org/p/chromium/issues/detail?id=891467#c14

But make sure there is a little bit of provision testing first, otherwise we may have to ask (and help) the lab technicians one day to manually recover 4k DUTs in the lab.

I've filed issue 911921 against myself to track moving informational Tast tests into their own suite so it's easier to reorder them to run earlier.