One more: Low coverage reported for r562745 (68.99% < 70.00% threshold)

r563183 and r563332 also

I think our lives will be much easier after switching to Chromium builders, which provides nice UI to visualize the results of each step and we can also potentially set up monitoring and alerts.

for r563332,
search for "Retrying, attempt"
https://chromium-coverage.appspot.com/reports/563332/linux/metadata/_bot.log
you will see browser tests failed thrice!, so no data from profdata

for r563183,
also an issue with browser tests, browser_tests binary failed, so partial data
https://chromium-coverage.appspot.com/reports/563183/linux/metadata/browser_tests.log
we might need aggressive return code retry or add another one to 124 ?

We can discuss some solution next week

Notes for myself (probably will get to it post bug-bash).

https://chromium-coverage.appspot.com/reports/563980/linux/metadata/browser_tests.log went through (just over 70%), even though bad report. Has same issue of too many browser tests failing. "Too many badly broken tests" is common message we might rely on. Regular builders browser_tests were green (https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Tests%20%28dbg%29%281%29/72408).

[4508/8953] ExtensionStorageMonitorTest.HostedAppTemporaryFilesystem (TIMED OUT)
[0604/085140.925167:ERROR:kill_posix.cc(83)] Unable to terminate process group 4687: No such process (3)
[0604/085140.931081:ERROR:kill_posix.cc(83)] Unable to terminate process group 4688: No such process (3)
[ RUN      ] ExtensionTabUtilBrowserTest.OpenExtensionsOptionsPage
[4509/8953] ExtensionTabUtilBrowserTest.OpenExtensionsOptionsPage (TIMED OUT)
Too many badly broken tests (879), exiting now.
Sending SIGTERM to 15 child processes... done.
Giving processes a chance to terminate cleanly... done.
Sending SIGKILL to 15 child processes... done.

So, might have to add retry logic and make it fail completely otherwise (no profdata merge at all if too bad results ??)

Yeah, weird that it started to happen more often recently: https://pantheon.corp.google.com/errors/CPHL8Z6A9-aiBQ?time=P30D&project=chromium-coverage

we have reports with low coverage every day, but previously we haven't seen so many of them.

But we're still getting good ones as well, so should be acceptable for now.

Problem is 2 reports >70% went on the dash and didn't quarantine (https://chromium-coverage.appspot.com/reports/563980/linux/chromium/src/report.html, https://chromium-coverage.appspot.com/reports/564080/linux/chromium/src/report.html). I think https://chrome-internal-review.googlesource.com/c/chrome/tools/code-coverage/+/636127 should help. if browser tests are badly broken and can't recover, we shouldnt try profdata and hence it will be definitely <70% then.

i have removed r564080 and r563980 from dash to avoid confusion.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/code-coverage/+/1f0c0416e434cc4fda2dfd297ca8be8cb6763dfe

commit 1f0c0416e434cc4fda2dfd297ca8be8cb6763dfe
Author: Abhishek Arya <inferno@chromium.org>
Date: Wed Jun 06 14:10:18 2018

r564730 had same issue, so removed from dash, hopefully my fix will be uptaken by all bots today.

All bots were updated by EOD yesterday. The most recent "low coverage" error was 11 hours ago and I'm sure it was from a non-updated bot.

Marking fixed for now, will reopen if we see anything on dash with low coverage.

Don't see any error in last 24 hours, but I do see 6 great reports. Awesome! Thanks, Abhishek!

No reports in quarantine for last 2 days, so looks like this fix worked.
gs://chromium-coverage/reports_quarantine