bug 792485  is flake on linux_chromium_rel_ng
 bug 792484  is flake on linux_chromium_rel_ng again (this probably should be merged to  bug 792485 )
bug 791800 is flake on linux_android_rel_ng

The data on the try waterfall doesn't look too bad to me:
https://ci.chromium.org/buildbot/tryserver.chromium.android/linux_android_rel_ng/?limit=200 (3 failures in 200 runs)

https://ci.chromium.org/buildbot/tryserver.chromium.linux/linux_chromium_rel_ng/?limit=200 (7 failures in 200 runs)

I would be fine with removing telemetry_perf_unittests from linux_android_rel_ng since we already have n5x coverage there, but it would be very bad for us to removing telemetry_perf_unittests from all place. 

I will prioritize looking at failure in linux_chromium_rel_ng first, since the error seems more consistent there.

2 things:
-i think we should look at cq try bots, not waterfall. i.e. the former run with DCHECKs, so if the waterfall is only running in release (is that so?) then this won't catch them. is the waterfall running these tests in debug?
-regarding removing it from one cq: that'll relieve some of the symptom. but if these tests are often flaky then this still leaves all the other CQ bots impacted.

Does anyone have a link to the success rate of this test vs others? I used to know the old URLs but they don't work anymore.

jam@ - the numbers nednguyen@ pointed at *are* trybot numbers?

That said, 7 failures in 200 runs == 3.5% flake rate. Assuming we retry a failed build 3 times and the failures are independent, that is still a 1% flake rate, or 20% of our hypothetical 5% total budget.

*However*, I don't actually see seven failed runs that look like flakes. Looking at the most recent 2000 builds on linux_chromium_rel_ng, (602031 - 604048), I saw exactly *1* build where a telemetry test failed and nothing else did: https://ci.chromium.org/buildbot/tryserver.chromium.linux/linux_chromium_rel_ng/603265, the one jam@ filed in  bug 792484 . Even in that case, the CL had retries disabled which as we know aggravates things (though there's not a great reason to suspect that the retry would've worked in this case). 

However, it looks like a subsequent build on that same patchset -- https://chromium-review.googlesource.com/c/chromium/src/+/810205/6 -- did in fact pass? Do you know if the CQ auto-retried it, or if you had to click retry manually?

In fact, looking back over the 2000 builds, it looked like browser_tests and webkit_layout_tests were significantly flakier.

I don't think we run the tests in a debug build anywhere, it'd be too slow (and likely buggy for other reasons). You raise a good point, though: like the other kinds of test suites, just looking at release-only tests won't catch issues w/ dchecks, and those can be very real. OTOH, they potentially also reflect real failures, right?

This stop causing unreasonable amount of flakes so far (also according to Dirk's analysis in #5), so I am closing this bug. Jam: feel free to create new one if you encounter high rate of flake.

Dirk: would be great if we can settle down "is this suite flaky on that builder?" with some hard data/dashboard :-)