Looks like the problem is probably coming from the report from the bots, because some summaries of failures are showing "Failure reason" too.  In the attached screenshot the recent failures show that, but the one at the bottom of the image show a reasonable reason, so maybe there was some recent change that caused this.

Deleted: Screen Shot 2017-05-17 at 8.33.09 AM.png 314 KB

	Screen Shot 2017-05-17 at 8.33.09 AM.png 314 KB View Download

The "Failure Reason" step is actually failing so the problem likely isn't with Sheriff-o-Matic right?

Ex: https://build.chromium.org/p/chromium.android/builders/Marshmallow%2064%20bit%20Tester/builds/12209

The "Failure Reason" step fails even though it seems to be reporting the correct result:
1 tests failed: ['chrome_public_test_apk']

Not sure who should be triaging this though.

This is definitely a wacky interaction between the recipes and SoM. Was there also an item for chrome_public_test_apk failing on that bot?

No, I don't see any chrome_public_test_apk failure on the Sheriff-O-Matic page that I still have up.

In case it's useful here's the raw text that was on that page:
-----------------------
Tree closers (2 total) (Help?)
 Collapse all /  Expand all
Failure reason failing on 3 builders
Click to expand
5/17/2017, 10:33 am PDT (4 hours ago)
Examine
 (0)
 LINK/FILE BUG
 SNOOZE
Builders this step failed on:
Android Cronet Builder (dbg)  Android WebView L (dbg)  Android WebView M (dbg)
No test result data available.
Regression ranges:
chromium:
472443 - 472531
Click to see more information.
net_unittests failing on chromium.android/Android Cronet Builder (dbg)
Click to expand
5/17/2017, 10:33 am PDT (4 hours ago)
Examine
 (0)
 LINK/FILE BUG
 SNOOZE
Builders this step failed on:
Android Cronet Builder (dbg)
No test result data available.
Regression ranges:
chromium:
472443 - 472488
Click to see more information.
Consistent failures (2 total) (Help?)
 Collapse all /  Expand all
Failure reason failing on 2 builders
Click to expand
5/17/2017, 3:00 am PDT (11 hours ago)
Examine
 (0)
 LINK/FILE BUG
 SNOOZE
Builders this step failed on:
orderfile-clankium [4 out of the last 4 builds have failed]  perf-clankium-l-tablet
No test result data available.
Regression ranges:
Generate Orderfile failing on internal.client.clank/orderfile-clankium
Bug 721682 Generate Orderfile failing on internal.client.clank/orderfile-clankium 
Click to expand
5/15/2017, 3:00 pm PDT (2 days ago)
Examine
 (0)
 LINK/FILE BUG
 SNOOZE
Builders this step failed on:
orderfile-clankium [16 out of the last 16 builds have failed]
No test result data available.
No regression range information available.
New failures (1 total) (Help?)
 Collapse all /  Expand all
Run CTS failing on 2 builders
 Bug 719974  Run CTS failing on 2 builders since May 4 
Click to expand
5/17/2017, 1:06 pm PDT (an hour ago)
Examine
 (0)
 LINK/FILE BUG
 SNOOZE
Builders this step failed on:
Android WebView L (dbg)  Android WebView M (dbg)
No test result data available.
Regression ranges:
chromium:
472505 - 472531
Click to see more information.
Infra failures (2 total) (Help?)
 Collapse all /  Expand all
chromium.android.Android Cronet Data Reduction Proxy Builder is idle with 53 pending builds.
Click to expand
5/17/2017, 1:02 pm PDT (an hour ago)
 (0)
Useful Links: Builder Last build Last build step
This idle-builder is a job for an Infra-Trooper. go/bug-a-trooper
No test result data available.
No regression range information available.
internal.client.clank.asan-clang-phone is offline.
Click to expand
5/15/2017, 5:15 pm PDT (2 days ago)
 (0)
Useful Links: Builder Last build Last build step
This offline-builder is a job for an Infra-Trooper. go/bug-a-trooper
No test result data available.
No regression range information available.
-----------------------

It's weird that the failure reason step is failing. I don't think it should be?

Although, it is now showing up on the perf waterfall as well...

I found the cause; it was https://codereview.chromium.org/2885333003. 

It makes the failure reason actually red.

Robbie, your CL had unintended consequences.

I'll make a CL to make SOM ignore this step.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/eaf04a7972f2b92128483e80fc5de83acf97de77

commit eaf04a7972f2b92128483e80fc5de83acf97de77
Author: Stephen Martinis <martiniss@chromium.org>
Date: Thu May 18 01:13:07 2017

AD: Ignore "Failure reason" step

Bug:  723682 
Change-Id: Ia13cd2dd71c5d5f78a55ce733aafb738da5cf61b
Reviewed-on: https://chromium-review.googlesource.com/508283
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>

[modify] https://crrev.com/eaf04a7972f2b92128483e80fc5de83acf97de77/go/src/infra/monitoring/analyzer/analyzer.go

Should the CL in c#8 have fixed this? Or are we waiting for a roll/deployment?

SOM is still showing "Failure reason" alerts.

I suppose this is unintended, but it's been broken for a long, long time. It should haven been red from day 1 :).

(we've gotten many bugs over the years from folks who claim their jobs are passing because the last step on the was green).

The CL landed, but I haven't rolled it out. Sorry about that. I'll make sure it gets rolled out soon. 

I'll do a cherry pick push for https://chromium-review.googlesource.com/508283

cherry pick deployed (subsequent alerts about -tainted default version silenced, but that's a separate topic...)

Thanks sean!

I also pushed alerts dispatcher for the chromium.perf tree.