Can you point to any doc about how the current tool works?

Without knowing exactly how it works, I'd imagine we'd need a buildbot on the main waterfall that builds with the necessary configuration to generate coverage, runs some tests, and verifies that the generated coverage data is sane.

Once that's reliably green, we'd mirror that configuration on a bot on the chromium.clang waterfall, using tip-of-tree instead of pinned Clang to catch any regressions from upstream.

We support coverage an all Chrome platforms (except Windows, needs testing atm), so the integration tests must be cross-platform as well.

Hans, thanks for the answer. We've started writing a doc on current workflow, we'll update this issue once it's ready!

Great! We should try to make it replace https://www.chromium.org/developers/code-coverage as the top search hit for "Chromium code coverage" :-)

There is a doc from a high level user perspective written by mus+ash Team: https://docs.google.com/document/d/1oF-_5XQ-hoOLFGyVPqgnowJtQYJj6Kd4rlXcPpCg-dU/edit

But I assume you're looking for a more detailed one, which explains what kind of tools we download and run through the script.

Re #5, thanks for the feedback! I think we should replace it with a clang_code_coverage.md.

Wow, I haven't heard about that page! We sure must update it :)

Re #2

I've wrote a doc to explain how code coverage works with Chrome: https://docs.google.com/document/d/1WylZsi6U6UOeekDrmPTYbH2ToFC7TOk_jXXfTmrtnCY/edit#heading=h.jfxzlxz5xk85.

hans@, can you take a look and please let me know if you have any questions.

Thanks! That's a great doc. Is the plan to work that into https://chromium.googlesource.com/chromium/src/+/master/docs/code_coverage.md to help Chromium devs get started?

As for testing this, I suppose what you want is a buildbot that builds with use_clang_coverage=true, runs the coverage tool on some target/s and sanity checks the output.

Once there's such a bot on the main waterfall we can use the same config but with tip-of-tree Clang to make sure it doesn't break in the next Clang roll.

Filed a separate issue for implementing report generation on Chrome Infra bots:  issue 825362 

This issue is for establishing test coverage for the code coverage tools integrated in Chromium.

1) ensure that clang roll script tests llvm-tools-llvm-cov

2) set up any basic test for the coverage script (i.e. integration test)

Clang roll script already tests llvm-tools-llvm-cov as it runs "ninja check-all": https://chromium.googlesource.com/chromium/src/+/master/tools/clang/scripts/update.py#550

An update here, we need a Mac bot as well. Bots should test code coverage script on a unittest target and on a fuzz target.

Yuke has raised a good question: whether we should test component or non-component build or both. I think we should test component build. That should be enough to cover non-component case as well.

I mentioned in c#18 that we need to test both tests and fuzzers. That's still true, but when we get around building fuzz targets on Mac, we'll have to revert https://chromium-review.googlesource.com/c/chromium/src/+/1024444

We don't do release component builds in most cases, and I would worry that debug builds would be too slow, so I'd start with release static builds.

Re c#23, I think we should take what Dirk mentioned into consideration when actually generate code coverage reports, but for the purpose of testing the coverage script, I feel like it's better to test component build as it covers non-component build. WDYT?

I don't know if there's a significant different one way or another, so I'd defer that to you.

re c#24: Yes, absolutely. I fully agree with Dirk's c#23 and this is why I decided to use static builds for report generation.

As for testing code coverage, I agree with Yuke, as test case with component build is slightly more complicated than a non-component one, i.e. if the component case works, a non-component case works for sure.

Yuke, what needs to be done to close this?

Almost ready, everything is up for review.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/082e996347bbe484ec2d90dba33e996a1a662bc0

commit 082e996347bbe484ec2d90dba33e996a1a662bc0
Author: Yuke Liao <liaoyuke@chromium.org>
Date: Fri May 18 15:40:40 2018

Coverage: Add an argument to allow specify coverage tools to use.

This CL adds an argument to allow specify coverage tools to use, and
one use case is for testing the code coverage script against clang
coverage tools tot.

Bug:  821840 
Change-Id: I2ecbd425ed1dcd8494e2e89963bc763505e19a11
Reviewed-on: https://chromium-review.googlesource.com/1064602
Commit-Queue: Yuke Liao <liaoyuke@chromium.org>
Reviewed-by: Max Moroz <mmoroz@chromium.org>
Reviewed-by: Abhishek Arya <inferno@chromium.org>
Cr-Commit-Position: refs/heads/master@{#559903}
[modify] https://crrev.com/082e996347bbe484ec2d90dba33e996a1a662bc0/tools/code_coverage/coverage.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/d2c674f80fbcd7a2df45498a1869a2e51c758b1f

commit d2c674f80fbcd7a2df45498a1869a2e51c758b1f
Author: Yuke Liao <liaoyuke@chromium.org>
Date: Mon May 21 18:08:16 2018

Infra: Establish test for coverage script on clang ToT waterfall.

This CL adds recipes to run coverage script on clang ToT waterfall on
both Mac and Linux platforms.

Bug:  821840 
Change-Id: I18f8942234856c5619cad3e4a7458b8a22e38010
Reviewed-on: https://chromium-review.googlesource.com/1064695
Commit-Queue: Yuke Liao <liaoyuke@chromium.org>
Reviewed-by: Reid Kleckner <rnk@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/d2c674f80fbcd7a2df45498a1869a2e51c758b1f/scripts/slave/README.recipes.md
[add] https://crrev.com/d2c674f80fbcd7a2df45498a1869a2e51c758b1f/scripts/slave/recipes/chromium_clang_coverage_tot.expected/full_chromium_clang_ToTLinuxCoverage.json
[add] https://crrev.com/d2c674f80fbcd7a2df45498a1869a2e51c758b1f/scripts/slave/recipes/chromium_clang_coverage_tot.expected/full_chromium_clang_ToTMacCoverage.json
[add] https://crrev.com/d2c674f80fbcd7a2df45498a1869a2e51c758b1f/scripts/slave/recipes/chromium_clang_coverage_tot.py
[modify] https://crrev.com/d2c674f80fbcd7a2df45498a1869a2e51c758b1f/masters/master.chromium.clang/builders.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/730d28c16612fb8e12522fee0ebda4df3d748bed

commit 730d28c16612fb8e12522fee0ebda4df3d748bed
Author: Yuke Liao <liaoyuke@chromium.org>
Date: Tue May 22 04:46:53 2018

Coverage: Add mb config for code coverage bots.

This CL adds mb configs for the clang tot coverage bots.

Bug:  821840 

Change-Id: I329f0c90871aaf364ab49d4fa53ddf428167176d
Reviewed-on: https://chromium-review.googlesource.com/1068198
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Yuke Liao <liaoyuke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#560485}
[modify] https://crrev.com/730d28c16612fb8e12522fee0ebda4df3d748bed/tools/mb/mb_config.pyl

https://ci.chromium.org/buildbot/chromium.clang/ToTMacCoverage/
https://ci.chromium.org/buildbot/chromium.clang/ToTLinuxCoverage/

Both builders are green now, marking this bug as fixed!