Defaulting to doing 3 retries was originally done in https://codereview.chromium.org/469533002. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e89251db348fb5b4e26796fc71d452967e408cbf

commit e89251db348fb5b4e26796fc71d452967e408cbf
Author: Caleb Rouleau <crouleau@chromium.org>
Date: Fri Nov 17 20:22:33 2017

[media_perftests] Disable retries.

This prevents the waterfall from retrying failures of media_perftests.

Bug:  785554 
Change-Id: I5d2e044a6749c938bae1bfbf567717971623cb77
Reviewed-on: https://chromium-review.googlesource.com/773464
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Fredrik Hubinette <hubbe@chromium.org>
Reviewed-by: Dale Curtis <dalecurtis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#517533}
[modify] https://crrev.com/e89251db348fb5b4e26796fc71d452967e408cbf/testing/buildbot/gn_isolate_map.pyl

crouleau: Was this issue fixed by your CL?

Hmhh, I believe the general direction is the flags should be moved into the JSON config file instead of the .pyl files.

+Dirk can confirm this.

I disabled retries for media_perftests, which is one of several gtest perf tests. I'll set this to available for others to work on it.

If the flags were moved to the JSON then they would have to be in many places in the JSON. I did it in the gn_isolate_map.pyl to avoid this. Also, IIRC, jmadill@ mentioned that "perf_tests_with_args" in perf_data_generator was for having different args for different bots.

I'll willing to change this to a different way if there's a plan to use the JSON config. What effort is it part of? Is there a design doc?

I do indeed prefer args to be listed in the JSON files, but I don't have a strong objection to using gn_isolate_map.pyl if the flag is used consistently everywhere.

crouleau, would you like to take on the task of disabling retries for angle_perftests? I am a little swamped.

jmadill@, sure, I can do that.

dpranke@, if we just change the default back to 0 retries then we won't have to worry about having any flags. We could gradually ramp down from 3 to 2 to 1 to 0 over a few months and deal with the failures as they come. WDYT?

@crouleau - yes, that would be an option, but as you say it would take quite a bit of time to do stably, and even then we'd have the reverse problem for binaries where it's not safe to turn them off (and I wouldn't want to assume we'll ever get there for some binaries).

Regardless, I am optimistic that in the near future we'll have a better mechanism in place to replace the JSON files and gn_isolate_map, and this won't be an issue.

(infra bug triager here, triaging infra bugs)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/06cce902ab82a04b268a0c0cb99884c5c3ba945a

commit 06cce902ab82a04b268a0c0cb99884c5c3ba945a
Author: Caleb Rouleau <crouleau@chromium.org>
Date: Wed Nov 29 18:20:11 2017

Disable retries for angle_perftests

Bug:  785554 
Change-Id: Ia73174808a48a942bcb2d893c60944f94efc6c4c
Reviewed-on: https://chromium-review.googlesource.com/784124
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Cr-Commit-Position: refs/heads/master@{#520167}
[modify] https://crrev.com/06cce902ab82a04b268a0c0cb99884c5c3ba945a/testing/buildbot/gn_isolate_map.pyl

I've done this for Media and ANGLE. Others are welcome to take it on for other projects.

Thanks Caleb!

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

This bugs still seems worth doing. +Erik FYI for Core Automation team.

With FindIt surfacing flaky tests now, the solution is issue 920002. Once that is done, FindIt will be able to let developers know when their tests become flaky. Then it won't be a problem that the retries turn the overall bot green.