Stephen says this could be part of test_results. Adding Sean to add to conversation.

sean isn't in charge of test results. Adding estaab!

This doesn't sound like test-results to me. It sounds like we want a feature in logdog to search for relevant strings and show just those and some context. I'm not sure what the SoM integration would look like but I don't think it would involve test-results, right?

I had thought that it would be a feature for test results to have the logs for a particular test case of a test suite available. So you could ask it for a link to the logs, and it'd be able to tell you. Maybe that's too much integration though.

Some parsing of the logs needs to happen at some point, and I could do that in SoM, but it'd be nice to not have to :)

Proposal:

The best way to do something like this would be to have something connected to the log firehose scanning for things. ATM the only thing thus connected is the Collector, and I'm very wary of adding any additional logic there, especially that could be subject to user configs, since this is a very pivotal service.

We could, however, create a "scanner" subscription to the log topic and a scanner service. It would load a list of per-project "interesting regexps" from "luci-config", sift through all text logs, and send an alert (Pub/Sub?) if it finds one in a log stream. Since this is an independent fleet, it could independently scale without interfering with the Collector, so I think this would work nicely.

Alert would contain: (project, log_stream_path, matched_regexp, log_stream_index)

This is a bit of work - probably a week or two at least. I think it's a cool idea though.

Preemptively indexing all logs is probably not an option, since we have no cloud service to maintain that.

This sounds like it is a FR for SoM since that's where the results would be displayed (correct?). Like an enhanced test step analyzer that knows more about test failures than it currently does.

What is a "datagram" in the context of this bug?

It seems like "1. Understand failing test steps that users would have found interesting if they looked at a fail[ing] build" is the hard part here. Does this mean regex matches, diffs btw failing and last passing, something more complicated that varies based on which particular tests are failing?

Adding SoM milestone UX to this (for triaging) since it sounds like the sheiff-o-matic side of this bug is a UX improvement?