Today, when a test fails on an iOS simulator bot but passes after a retry, it's still causing the test execution to fail (even if all tests eventually passed after retries).
This should be changed to counting the run as a successful run, since there are other infrastructure pieces that can be used to address the flakiness in a more efficient manner: go/flakiness-dashboard and the Try Flakes (https://cs.corp.google.com/chromium_infra/appengine/chromium_try_flakes/) service.
Using those, it's possible to address the flakiness in a more efficient way, since they'll be tracked in bugs which can be assigned and/or used as input for a FixIt.
Spamming random developers with random flakes in e-mail is unlikely to get addressed.
After changing this, it should be ensured that the flaky tests during iOS Simulation runs are actually reported properly to the flakiness dashboard (out of scope for this bug and should probably be tracked in a separate bug).
Comment 1 by bugdroid1@chromium.org
, Mar 31 2017