Many of the tests check for things that are not directly critical to the user, this has always been the case though, we also had one on 68 where the unittests were failing due to some builder state that has no user impact and that test did get removed https://bugs.chromium.org/p/chromium/issues/detail?id=847645

In general someone should care, as it implies we are doing something incorrect, so this flags it, but we can have a lot of failures like this that don't prevent shipment.

It seems like what might be in order is a better way of marking such failures as user facing or not, but that is not trivial. 

a red build should mean that, the build is no good, it should be manual check is require to make the determination.   If the test is ok to fail, then the test should be in the suite BVT-dontcare or something.

What would it take to make this happen ? 

If these things really don't matter, we should delete them. If they do, we need the builds to be marked red otherwise no one would fix them.

Please reopen if you think we should delete these tests.

Just because something may not matter enough to block to a release, does not mean it does not matter to someone. 

If we want to define red as 'never able to release under any circumstances' we would want to remove most of the HWTests, as we ship with various test failures all the time.

I hope this resolves itself with greater separation of build from test in the future, a red build (no artifacts) is materially different from a failed test (which can be flaky or broken on the test side). 

We'd have to construct a new way to socially signal to those responsible for maintaining the build that the non-build critical things that's broken needs to be fixed as fast as if it were a broken build. I don't know how to do that. One idea would be that there's some organizational incentive to fix it.

I think this is more a general 'feature' request, may be we need a test review or some kind for the build.  Issues are:

1- not clear on owner for each steps or failure for the builder
   - Last TPM/Test meeting, Berine's method is to look at the output logs and code search and check history to find failure owner. However would be nice to make the builder display the owner and may be a bug link instead.

2- failure in builder anywhere seems to trigger a red build.
  - I don't have any stat but most time a red build does not really mean it. Multiple teamates are require to dig in and usually make the determination that test team can go on and testing.  We would like ot have way to separate the pink vs red for tests.

The CI roadmap has a feature for the UI to offer annotation-based filtering. So, we could have an annotation that a build produced build artifacts and then you could filter builds based on that. That's scheduled for Q2 of next year. Does that sound like what you want?

FWIW I think the failures on 68 were largely treated like they were a broken build.

The unittest fix was put in within an hour of bringing in the author, and it was actually removed so it should never come back. This did not manifest on ToT so it was not raised until I did so, which left a few red builds over the long weekend.

The imagetest failure was also 'fixed' within a day, as it also broke ToT and was quickly merged to the branch, it lingered on the branch due to needing a Chromite roll in the Chrome DEPS. The root problem here was more a lack of understanding of the complexity involved in building Chrome on Chrome OS, the separate Chromite checkout threw us off. 

As to the request here, I am not sure how feasible it would be to have a specific owner for every type of failure that can happen, or to even reliably separate many of these failures between real blocking errors and mere warnings. We can probably do better in expressing what actually failed, and even automatically find a list of people whom may know about it through git history and OWNERS files, but we are unlikely to be 100% covered in this manner.