At Dart we're implementing a first class test results workflow. Our systems can compare with the previous test results and approved test failures that shouldn't turn builders green.

We'd like to be able to notify developers if a builder has a new test failure, even if the builder is already red. We currently use edge triggering with luci-notify, so developers know when builders turn red, however they are not notified about any additional test failures. Likewise the milo console has red builds, where usually only the first in a red column is interesting, but there could be other interesting builds hidden in a red column that add additional regressions.

Buildbot supported the 'Failed Again' outcome, which Milo still supports showing, however the new LUCI recipes system doesn't let us use that color. If Failed Again was supported, our recipe could readily determine whether there are new regressions and make the builder red or make it orange (failed again) if the same test failures were seen. luci-notify could then be put in level triggered mode and mail on every red build, and ignore the orange ones. Developers would then be notified if additional tests break even if the builder is red and it would be easy to spot interesting (red) builds on the milo console.

Some extensibility with additional colors could be useful. For instance we have deflaking and could use a dark green color to signify a build that had flakes that were ignored. This is not as important.

This feature isn't critical to us shipping first class test results, but it will improve how well it works and give our developers a better experience.