11090 is the most recent successful build is:

2018-09-22 11:30	11090.0.0

Still no successful coral builds since 9/22.

+bjanakiraman to further triage.

The current situation doesn't look promising. With the current unibuild setup, a BVT failure of one model prevents any further HWTests to schedule on *all* models in the same unibuild. That's not an ideal situation as we potentially lose the test coverage on many models where we could have caught real regressions/bugs.

The situation with the coral build is just one good example of this fundamental problem.

Please don't file bugs in Infra>Client>ChromeOS. No one will see them: no one is auto CC'd; always use one of the subcomponents.

Please always file release builder fails against the current sheriffs. Release builders are their responsibility to triage.

I took a quick look at this and it seems that there are at least two problems:
* A unit test is failing (autotest_lib.site_utils.dut_status_unittest)
* bvt-inline is failing on autoupdate_EndToEndTest_paygen_au_stable_full

Both of these look like Product failures so over to next week's sheriffs to triage and find owners.

Comment #5 isn't specific to this coral failure. I sort of hijacked this bug to point out a more fundamental problem where we should rethink how tests should be schedule in the world of unibuild.  This is not the first time, and definitely not the last, where one model could block the testing of the all models, and this is statistically more likely to happen as the number of models increases. We simply need some changes to our infra to deal with that better.

Hi Ben, for unibuild test scheduling, please file a new bug in the Infra>Client>ChromeOS>Build component and assign that to shapiroc@ for triage.

This bug is tracking coral-release failing.

Re:#7, agree this is a problem, we need to figure out a way to separate the build from the tests, +shapiroc to follow up on that. 

Ben... I agree this is a long-running problem with lab stability that was greatly exacerbated by unified builds.

There's a plan to improve lab stability, but we may need to entertain a shorter term solution here.

It's a fairly simple change to just make every hwtest phase async for unified builds and ignore all of the test results for the builder.

bhthompson@ ... any objection to this change?

 crbug.com/901871  to track the proposed change to the unibuild release builder semantics.

Sheriffs, please have a look at the other issues with coral-release that are going on right now.

I see different build statuses on legoland and build-annotator

For coral-release on build-annotator I see a couple of successful builds, but I don't see them on legoland. There's one on the 4th and one on the 30th. Are these errors? Are the results supposed to be different?

https://chromiumos-build-annotator.googleplex.com/build_annotations/builds_list/coral-release/


https://cros-goldeneye.corp.google.com/chromeos/legoland/builderHistory?buildConfig=coral-release&buildBranch=master&startCursor=Ci0SJ2oQc35jci1idWlsZGJ1Y2tldHITCxIFQnVpbGQYkMu2mYLGivl7DBgAIAA%3D

The current situation is pretty grim for developers looking to test Chrome changes on coral.

I've put up a CL that will work around the issue in the meanwhile, but I think we need a better solution in the long run; invalidating all coral builds when any HWTest suite fails seems untenable.

https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1324169

we're discussing some options this pm and i'll post the outcome

 Issue 904473  has been merged into this issue.

Getting through the CQ has been pretty difficult the last few days.

Here's the CL for those interested on this bug:
https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1324773

chumped

Finally a green coral-release build 11259.0.0. Closing this bug.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/8b33404f0931ce22de5830f95a6a30fc678f048b

commit 8b33404f0931ce22de5830f95a6a30fc678f048b
Author: Steven Bennetts <stevenjb@chromium.org>
Date: Fri Nov 16 17:20:30 2018

Support full version in Simple Chrome

Currently if a release build fails, no LATEST file is created,
causing Simple Chrome to always fail.

This allows a full version to be specified, e.g. R56-1234.0.0,
allowing a version that generated a valid image and artifacts,
but possibly failed some tests (e.g. due to flake) to be used.

BUG= chromium:898576 
TEST='cros chrome-sdk --internal --board=coral --version=R72-11230.0.0'

Change-Id: Id9e88125ee072c853aa5f471b5edeb3bf969af88
Reviewed-on: https://chromium-review.googlesource.com/c/1324169
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Steven Bennetts <stevenjb@chromium.org>
Tested-by: Steven Bennetts <stevenjb@chromium.org>

[modify] https://crrev.com/8b33404f0931ce22de5830f95a6a30fc678f048b/cli/cros/cros_chrome_sdk.py
[modify] https://crrev.com/8b33404f0931ce22de5830f95a6a30fc678f048b/cli/cros/cros_chrome_sdk_unittest.py