The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/0f67735deeaacc9d569eb0f3883781da3de0a381

commit 0f67735deeaacc9d569eb0f3883781da3de0a381
Author: Yuke Liao <liaoyuke@chromium.org>
Date: Mon Dec 17 05:39:59 2018

[code coverage] Fix table header and TOTALS row column mismatch

Fix table header and TOTALS row column mismatch.

Bug: 915560
Change-Id: I1f056bcd28e2d54bfcdd2c46adf5bd05828064a2
Reviewed-on: https://chromium-review.googlesource.com/c/1379491
Commit-Queue: Yuke Liao <liaoyuke@chromium.org>
Reviewed-by: Shuotao Gao <stgao@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19593}
[modify] https://crrev.com/0f67735deeaacc9d569eb0f3883781da3de0a381/appengine/findit/templates/coverage/summary_view.html

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/25717e6375b2f7147a6bb2ce0094e046ba414beb

commit 25717e6375b2f7147a6bb2ce0094e046ba414beb
Author: Yuke Liao <liaoyuke@chromium.org>
Date: Tue Dec 18 04:09:44 2018

[code coverage] Fix bugs in merge logic

The retry logic didn't work due to some bugs and this CL fixes them.

Bug: 915560
Change-Id: Ib51a837bf3c5c5a4de7fd10d074c5aded2c9683e
Reviewed-on: https://chromium-review.googlesource.com/c/1381331
Commit-Queue: Yuke Liao <liaoyuke@chromium.org>
Reviewed-by: Shuotao Gao <stgao@chromium.org>

[modify] https://crrev.com/25717e6375b2f7147a6bb2ce0094e046ba414beb/scripts/slave/recipe_modules/clang_coverage/unittests/merge_profiles_test.py
[modify] https://crrev.com/25717e6375b2f7147a6bb2ce0094e046ba414beb/scripts/slave/recipe_modules/clang_coverage/resources/merger.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/7f8470741e12059a828d15b28654ddb95eafe62d

commit 7f8470741e12059a828d15b28654ddb95eafe62d
Author: Yuke Liao <liaoyuke@chromium.org>
Date: Tue Dec 18 23:06:16 2018

[code coverage] Implement validate profraws in parallel manner

This CL implements a strategy to validate profraw files before merging,
and it is done in parallel for performance considerations.

Some stats related to this CL:
1. In the most recent failed browser_tests step, there are in total 190
.profraw files need to be merged, and out of them, 3 are invalid and
need to be excluded.

2. With multiprocessing, it only takes 2 minutes in total to merge all
the .profraw files.

Bug: 915560
Change-Id: Ia8a6372bdfa2d98c5df26e768c3d8a45644e32bf
Reviewed-on: https://chromium-review.googlesource.com/c/1382806
Commit-Queue: Yuke Liao <liaoyuke@chromium.org>
Reviewed-by: Shuotao Gao <stgao@chromium.org>

[modify] https://crrev.com/7f8470741e12059a828d15b28654ddb95eafe62d/scripts/slave/recipe_modules/clang_coverage/unittests/merge_profiles_test.py
[modify] https://crrev.com/7f8470741e12059a828d15b28654ddb95eafe62d/scripts/slave/recipe_modules/clang_coverage/resources/merger.py
[modify] https://crrev.com/7f8470741e12059a828d15b28654ddb95eafe62d/scripts/slave/recipe_modules/clang_coverage/resources/merge_profiles.py

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/b6c91b18b6f53de080514732368b2437b0a972f2

commit b6c91b18b6f53de080514732368b2437b0a972f2
Author: Yuke Liao <liaoyuke@google.com>
Date: Fri Dec 21 21:20:49 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/dd778d7bf8464144504b60c42f1d775aa5651b08

commit dd778d7bf8464144504b60c42f1d775aa5651b08
Author: Yuke Liao <liaoyuke@google.com>
Date: Wed Jan 02 21:16:37 2019

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/61f259a6ff106b13c66ef8594f82d82ca84ae1e7

commit 61f259a6ff106b13c66ef8594f82d82ca84ae1e7
Author: Yuke Liao <liaoyuke@google.com>
Date: Wed Jan 02 21:22:17 2019

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/1eb1c4c9ae49cfaed75f4552bce0d596d669040e

commit 1eb1c4c9ae49cfaed75f4552bce0d596d669040e
Author: Yuke Liao <liaoyuke@google.com>
Date: Mon Jan 07 18:12:25 2019

Re-open this bug as even though the percentage matches, the total number of lines don't, chromium-coverage has something like 1.05m, but Findit only has 1.01m.

The root cause is likely to be the list of tests suites, I'll generate the difference, and one thing worth discussing is that if a test target is not run on any CI/CQ builder, should we run it on coverage bots? Would anyone care about the coverage numbers for those targets?

> if a test target is not run on any CI/CQ builder, should we run it on coverage bots?

Good question!

> Would anyone care about the coverage numbers for those targets?

I'd say no, but if they want to care, the test should be included in some CI/CQ builder.

By the way, a big different might be coming from https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_tests.md

Are you guys running these?

Here's the script from the GCE bots: https://chrome-internal.googlesource.com/chrome/tools/code-coverage/+/master/scripts/run_layout_tests.bash

> I'd say no, but if they want to care, the test should be included in some CI/CQ builder.

Agree! If a target is not on CI/CQ, it may break anytime without notice, and running them on coverage builder would be looking for troubles.

> By the way, a big different might be coming from https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_tests.md

We already run it, if not, the difference will be much bigger.

> We already run it, if not, the difference will be much bigger.

Ooh, nice! Yeah, I also realized that the difference would be bigger and hoped that maybe you've got some extra stuff that GCE bots don't run :D

Here is the list of targets that run on gce bot but not on LUCI bot:
audio_unittests // Not running on any Chromium CI/CQ bot.
breakpad_unittests // Android specific test target.
courgette_unittests // Windows specific test target.
crashpad_tests // Non-linux test target.
gl_unittests // Android specific test target.
media_mojo_unittests // Not running on any Chromium CI/CQ bot.
pdfium_embeddertests // Not running on any Chromium CI/CQ bot.
pdfium_unittests // Not running on any Chromium CI/CQ bot.
swiftshader_unittests // GPU FYI desktop specific tests. (Not CI/CQ)
zucchini_unittests // Windows specific test target.

It seems that we shouldn't run them.

The tricky thing is how to confirm/deny that they're the root cause, I'll need to do a little bit more digging later.

Either way, I'd rather track down the owners / others who would be interested in running those and clarify that the targets need to run on some CI/CQ bots.

+tsepez@ for pdfium
+wfh@ for crashpad
+piman@ for swiftshader
+huangs@ for zucchini and courgette
+dalecurtis for audio / media

Sorry if some of you are wrong people to discuss this. If you know a better person, please CC them.

The issue is that code coverage is migrating to Chrome Infra bots and to do things right we have to ensure the following: any test target which is running on code coverage bots, should be also running on some CI/CQ builder(s). Otherwise, the target has to be excluded from the code coverage test suite.

Sg! Thanks for looping the stakeholders in!

+kbr@ for gpu tests.

swiftshader -> sugoi

Re #c10: no, we should have some criteria for a test target to be added to the code coverage bots. And the conditions below should be in the criteria:
1. Run on the same platform.
2. Run on CQ to prevent breaking CLs from landing.
3. Run on CI to let sheriff or Findit revert breaking CLs.

Compile failures are not acceptable, while a few test failures are acceptable but best to not happen.

gl_unittests is run on the commit queue on multiple platforms, not just Android. You can confirm this by looking at test_suites.pyl, waterfalls.pyl and ultimately the generated JSON files.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/ee8f450ae694e6845e3c28720507de3f1c091843

commit ee8f450ae694e6845e3c28720507de3f1c091843
Author: Yuke Liao <liaoyuke@google.com>
Date: Tue Jan 22 20:49:05 2019