Mhh, it should be noted that the two tests that were retried (security_ProfilePermissions, login_CryptohomeIncognito) each have JOB_RETRIES = 5 specified. I am not sure if that is really meaningful.

But I kinda sorta sense an arms race between gardeners and infra team here. Gardeners trying to find ways to retry, while infra trying to find ways to disable retries. Or am I interpreting this wrong?

I'm not sure that this is a gardner vs. infa thing per-se, it's just that retries are a double edged sword. On the one hand they help reduce flakiness, on the other hand they can mask real failures and increase load. 

Personally I think that any test that requires > 2 attempts is way too flaky and needs to be disabled or fixed.

It would be great to do an audit and document this somewhere (or better yet write a tool to generate the documentation).

It is just these two tests, all others have 2 or 0.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/6ec709fd174e8ec4daaa2d1e6d666c952fca60ed

commit 6ec709fd174e8ec4daaa2d1e6d666c952fca60ed
Author: Ilja H. Friedel <ihf@google.com>
Date: Wed May 31 09:15:31 2017

Revert to 2 retries for CryptohomeIncognito & ProfilePermissions.

BUG=chromium:631640,  chromium:727540 
TEST=None.

Change-Id: Ib64b9f03a6392dc0bffc906fc29975467b5b984c
Reviewed-on: https://chromium-review.googlesource.com/518553
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/6ec709fd174e8ec4daaa2d1e6d666c952fca60ed/client/site_tests/login_CryptohomeIncognito/control
[modify] https://crrev.com/6ec709fd174e8ec4daaa2d1e6d666c952fca60ed/client/site_tests/security_ProfilePermissions/control

David is limiting suite retries to 5 in issue 728738.
Marking this as fixed.