Sheriff-o-matic is looking at which build step failed on a given builder: it doesn't appear to be considering different individual test cases that run in the same step as distinct. So, yeah, if there was an issue with chrome_public_test_apk on a given builder, then it will continue to be associated with that even if the original test is fixed and a different one is now broken..

martiniss: can you take this one?

I have a theory about what's going on.  Maybe these bug associations were old and the actual failure changed from one test to another.  SoM keys failures by test_suite and tester, not by the actual test, right?  So that same key might shift from testX to testY but still have the bug for X associated with it.

Is there any automatic association of bugs to failures that's operating for Android?  If not, then this probably explains what's going on.  Having said that, it would be great if the failure key was by test instead of the whole test suite -- there are tons of instrumentation tests and we don't normally want to lump all failures together as a single build issue.

#3 is correct; we key by test suite at the moment.

I'm working on a refactor to allow these to be keyed by tests, rather than just test suite. It's taking a bit to land, though.

The short term solution is to remove any linked bugs for alerts which haven't been active for a while. This isn't perfect, but it should remove a lot of the confusing cases which happen right now. I'll do this now.

Ping for update on this?

Sorry, been working on landing the refactor recently.

The plan right now is to make the app delete annotations older than 2 days, which should get rid of a lot of the problems we have. It won't solve everything, but it'll get us on feature parity with the old app. 

Ok, change should hopefully land today.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/1b7b164ad27c4ada2b7f1e6d324bb36cae0bb96d

commit 1b7b164ad27c4ada2b7f1e6d324bb36cae0bb96d
Author: Stephen Martinis <martiniss@chromium.org>
Date: Tue Aug 30 22:08:13 2016

SOM: Delete old annotations

This sets up a cron for SOM which deletes old annotations, so that old
annotations don't clutter up new failures. This is a bit simplistic, but
should be accurate for now, if not as helpful as we theoretically could be.

BUG= 639171 

Change-Id: I12bbe199746cf483013530f5cc63c025cee8e53d
Reviewed-on: https://chromium-review.googlesource.com/376184
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/1b7b164ad27c4ada2b7f1e6d324bb36cae0bb96d/go/src/infra/appengine/sheriff-o-matic/cron.yaml
[modify] https://crrev.com/1b7b164ad27c4ada2b7f1e6d324bb36cae0bb96d/go/src/infra/appengine/sheriff-o-matic/main.go
[modify] https://crrev.com/1b7b164ad27c4ada2b7f1e6d324bb36cae0bb96d/go/src/infra/appengine/sheriff-o-matic/main_test.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/1b7b164ad27c4ada2b7f1e6d324bb36cae0bb96d

commit 1b7b164ad27c4ada2b7f1e6d324bb36cae0bb96d
Author: Stephen Martinis <martiniss@chromium.org>
Date: Tue Aug 30 22:08:13 2016

SOM: Delete old annotations

This sets up a cron for SOM which deletes old annotations, so that old
annotations don't clutter up new failures. This is a bit simplistic, but
should be accurate for now, if not as helpful as we theoretically could be.

BUG= 639171 

Change-Id: I12bbe199746cf483013530f5cc63c025cee8e53d
Reviewed-on: https://chromium-review.googlesource.com/376184
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/1b7b164ad27c4ada2b7f1e6d324bb36cae0bb96d/go/src/infra/appengine/sheriff-o-matic/cron.yaml
[modify] https://crrev.com/1b7b164ad27c4ada2b7f1e6d324bb36cae0bb96d/go/src/infra/appengine/sheriff-o-matic/main.go
[modify] https://crrev.com/1b7b164ad27c4ada2b7f1e6d324bb36cae0bb96d/go/src/infra/appengine/sheriff-o-matic/main_test.go

Change is live. This should work well enough for now; it should be solved more fully later.

If this is really a Pri-1, find an owner and update the priority.

This is the result of a bulk edit that moved high priority available bugs to a lower priority in an attempt to be more honest with bug filers.

 Issue 684199  has been merged into this issue.

 Issue 712808  has been merged into this issue.