Revisit code coverage with component builds |
|||
Issue descriptionIt looks like with %Nm we're able to get raw dumps from shared libraries. I used the example targets from code coverage documentation: https://chromium.googlesource.com/chromium/src/+/master/docs/code_coverage.md#coverage-script and built those targets in two different modes: gn gen out/coverage2 \ --args='use_clang_coverage=true is_component_build=false' gn gen out/coverage2 \ --args='use_clang_coverage=true is_component_build=true' And generated code coverage reports. Code coverage report for the component build doesn't contain lot of information -- we knew that part, however, there are bunch of .profraw dumps with different signatures in case of component build (I commented out deletion of those files): $ ls -l out/report1 total 26012 -rw-r----- 1 mmoroz primarygroup 1655 Apr 11 18:21 component_view_index.html drwxr-x--- 4 mmoroz primarygroup 4096 Apr 11 18:21 coverage -rw-r----- 1 mmoroz primarygroup 1935016 Apr 11 18:20 coverage.profdata -rw-r----- 1 mmoroz primarygroup 4589496 Apr 11 18:20 crypto_unittests.16759087908695372714_0.profraw -rw-r----- 1 mmoroz primarygroup 4589496 Apr 11 18:20 crypto_unittests.16759087908695372714_1.profraw -rw-r----- 1 mmoroz primarygroup 4589496 Apr 11 18:20 crypto_unittests.16759087908695372714_3.profraw -rw-r----- 1 mmoroz primarygroup 4354 Apr 11 18:20 crypto_unittests_output.txt -rw-r----- 1 mmoroz primarygroup 250 Apr 11 18:21 directory_view_index.html -rw-r----- 1 mmoroz primarygroup 41254 Apr 11 18:21 file_view_index.html -rw-r----- 1 mmoroz primarygroup 200 Apr 11 18:21 index.html -rw-r----- 1 mmoroz primarygroup 2519 Apr 11 18:21 style.css -rw-r----- 1 mmoroz primarygroup 5419792 Apr 11 18:20 url_unittests.17038110202113881002_0.profraw -rw-r----- 1 mmoroz primarygroup 5419792 Apr 11 18:20 url_unittests.17038110202113881002_2.profraw -rw-r----- 1 mmoroz primarygroup 772 Apr 11 18:20 url_unittests_output.txt $ ls -l out/report2 total 43268 -rw-r----- 1 mmoroz primarygroup 1651 Apr 11 18:18 component_view_index.html drwxr-x--- 4 mmoroz primarygroup 4096 Apr 11 18:18 coverage -rw-r----- 1 mmoroz primarygroup 1941192 Apr 11 18:18 coverage.profdata -rw-r----- 1 mmoroz primarygroup 552720 Apr 11 18:18 crypto_unittests.10188749721697527403_0.profraw -rw-r----- 1 mmoroz primarygroup 552720 Apr 11 18:18 crypto_unittests.10188749721697527403_1.profraw -rw-r----- 1 mmoroz primarygroup 552720 Apr 11 18:18 crypto_unittests.10188749721697527403_2.profraw -rw-r----- 1 mmoroz primarygroup 552720 Apr 11 18:18 crypto_unittests.10188749721697527403_3.profraw -rw-r----- 1 mmoroz primarygroup 460392 Apr 11 18:18 crypto_unittests.11598866367102063436_0.profraw -rw-r----- 1 mmoroz primarygroup 460392 Apr 11 18:18 crypto_unittests.11598866367102063436_1.profraw -rw-r----- 1 mmoroz primarygroup 460392 Apr 11 18:18 crypto_unittests.11598866367102063436_2.profraw -rw-r----- 1 mmoroz primarygroup 460392 Apr 11 18:18 crypto_unittests.11598866367102063436_3.profraw -rw-r----- 1 mmoroz primarygroup 675800 Apr 11 18:18 crypto_unittests.13327041009915459474_0.profraw -rw-r----- 1 mmoroz primarygroup 675800 Apr 11 18:18 crypto_unittests.13327041009915459474_1.profraw -rw-r----- 1 mmoroz primarygroup 675800 Apr 11 18:18 crypto_unittests.13327041009915459474_2.profraw -rw-r----- 1 mmoroz primarygroup 675800 Apr 11 18:18 crypto_unittests.13327041009915459474_3.profraw -rw-r----- 1 mmoroz primarygroup 897416 Apr 11 18:18 crypto_unittests.15886912659040564138_0.profraw -rw-r----- 1 mmoroz primarygroup 897416 Apr 11 18:18 crypto_unittests.15886912659040564138_1.profraw -rw-r----- 1 mmoroz primarygroup 897416 Apr 11 18:18 crypto_unittests.15886912659040564138_2.profraw -rw-r----- 1 mmoroz primarygroup 897416 Apr 11 18:18 crypto_unittests.15886912659040564138_3.profraw -rw-r----- 1 mmoroz primarygroup 98776 Apr 11 18:18 crypto_unittests.16708870218917916490_0.profraw -rw-r----- 1 mmoroz primarygroup 98776 Apr 11 18:18 crypto_unittests.16708870218917916490_1.profraw -rw-r----- 1 mmoroz primarygroup 98776 Apr 11 18:18 crypto_unittests.16708870218917916490_2.profraw -rw-r----- 1 mmoroz primarygroup 98776 Apr 11 18:18 crypto_unittests.16708870218917916490_3.profraw -rw-r----- 1 mmoroz primarygroup 376456 Apr 11 18:18 crypto_unittests.17920174803412120650_0.profraw -rw-r----- 1 mmoroz primarygroup 376456 Apr 11 18:18 crypto_unittests.17920174803412120650_1.profraw -rw-r----- 1 mmoroz primarygroup 376456 Apr 11 18:18 crypto_unittests.17920174803412120650_2.profraw -rw-r----- 1 mmoroz primarygroup 376456 Apr 11 18:18 crypto_unittests.17920174803412120650_3.profraw -rw-r----- 1 mmoroz primarygroup 84960 Apr 11 18:18 crypto_unittests.2396517324908736818_0.profraw -rw-r----- 1 mmoroz primarygroup 84960 Apr 11 18:18 crypto_unittests.2396517324908736818_1.profraw -rw-r----- 1 mmoroz primarygroup 84960 Apr 11 18:18 crypto_unittests.2396517324908736818_2.profraw -rw-r----- 1 mmoroz primarygroup 84960 Apr 11 18:18 crypto_unittests.2396517324908736818_3.profraw -rw-r----- 1 mmoroz primarygroup 2680384 Apr 11 18:18 crypto_unittests.4486866546415893753_0.profraw -rw-r----- 1 mmoroz primarygroup 2680384 Apr 11 18:18 crypto_unittests.4486866546415893753_1.profraw -rw-r----- 1 mmoroz primarygroup 2680384 Apr 11 18:18 crypto_unittests.4486866546415893753_2.profraw -rw-r----- 1 mmoroz primarygroup 2680384 Apr 11 18:18 crypto_unittests.4486866546415893753_3.profraw -rw-r----- 1 mmoroz primarygroup 748176 Apr 11 18:18 crypto_unittests.6899576307932548990_0.profraw -rw-r----- 1 mmoroz primarygroup 748176 Apr 11 18:18 crypto_unittests.6899576307932548990_1.profraw -rw-r----- 1 mmoroz primarygroup 748176 Apr 11 18:18 crypto_unittests.6899576307932548990_2.profraw -rw-r----- 1 mmoroz primarygroup 748176 Apr 11 18:18 crypto_unittests.6899576307932548990_3.profraw -rw-r----- 1 mmoroz primarygroup 4357 Apr 11 18:18 crypto_unittests_output.txt -rw-r----- 1 mmoroz primarygroup 250 Apr 11 18:18 directory_view_index.html -rw-r----- 1 mmoroz primarygroup 20079 Apr 11 18:18 file_view_index.html -rw-r----- 1 mmoroz primarygroup 200 Apr 11 18:18 index.html -rw-r----- 1 mmoroz primarygroup 2519 Apr 11 18:18 style.css -rw-r----- 1 mmoroz primarygroup 552720 Apr 11 18:18 url_unittests.10188749721697527403_0.profraw -rw-r----- 1 mmoroz primarygroup 552720 Apr 11 18:18 url_unittests.10188749721697527403_2.profraw -rw-r----- 1 mmoroz primarygroup 167352 Apr 11 18:18 url_unittests.11162461378565426850_0.profraw -rw-r----- 1 mmoroz primarygroup 167352 Apr 11 18:18 url_unittests.11162461378565426850_2.profraw -rw-r----- 1 mmoroz primarygroup 460392 Apr 11 18:18 url_unittests.11598866367102063436_0.profraw -rw-r----- 1 mmoroz primarygroup 460392 Apr 11 18:18 url_unittests.11598866367102063436_2.profraw -rw-r----- 1 mmoroz primarygroup 38208 Apr 11 18:18 url_unittests.12593334497456212627_0.profraw -rw-r----- 1 mmoroz primarygroup 38208 Apr 11 18:18 url_unittests.12593334497456212627_2.profraw -rw-r----- 1 mmoroz primarygroup 110176 Apr 11 18:18 url_unittests.12618661781032985762_0.profraw -rw-r----- 1 mmoroz primarygroup 110176 Apr 11 18:18 url_unittests.12618661781032985762_2.profraw -rw-r----- 1 mmoroz primarygroup 675800 Apr 11 18:18 url_unittests.13327041009915459474_0.profraw -rw-r----- 1 mmoroz primarygroup 675800 Apr 11 18:18 url_unittests.13327041009915459474_2.profraw -rw-r----- 1 mmoroz primarygroup 79512 Apr 11 18:18 url_unittests.14888186743657353593_0.profraw -rw-r----- 1 mmoroz primarygroup 79512 Apr 11 18:18 url_unittests.14888186743657353593_2.profraw -rw-r----- 1 mmoroz primarygroup 933104 Apr 11 18:18 url_unittests.15882516995641766826_0.profraw -rw-r----- 1 mmoroz primarygroup 933104 Apr 11 18:18 url_unittests.15882516995641766826_2.profraw -rw-r----- 1 mmoroz primarygroup 98776 Apr 11 18:18 url_unittests.16708870218917916490_0.profraw -rw-r----- 1 mmoroz primarygroup 98776 Apr 11 18:18 url_unittests.16708870218917916490_2.profraw -rw-r----- 1 mmoroz primarygroup 74416 Apr 11 18:18 url_unittests.17358912728385580130_0.profraw -rw-r----- 1 mmoroz primarygroup 74416 Apr 11 18:18 url_unittests.17358912728385580130_2.profraw -rw-r----- 1 mmoroz primarygroup 376456 Apr 11 18:18 url_unittests.17920174803412120650_0.profraw -rw-r----- 1 mmoroz primarygroup 376456 Apr 11 18:18 url_unittests.17920174803412120650_2.profraw -rw-r----- 1 mmoroz primarygroup 321832 Apr 11 18:18 url_unittests.1970306908406505448_0.profraw -rw-r----- 1 mmoroz primarygroup 321832 Apr 11 18:18 url_unittests.1970306908406505448_2.profraw -rw-r----- 1 mmoroz primarygroup 84960 Apr 11 18:18 url_unittests.2396517324908736818_0.profraw -rw-r----- 1 mmoroz primarygroup 84960 Apr 11 18:18 url_unittests.2396517324908736818_2.profraw -rw-r----- 1 mmoroz primarygroup 14272 Apr 11 18:18 url_unittests.349012374149567957_0.profraw -rw-r----- 1 mmoroz primarygroup 14272 Apr 11 18:18 url_unittests.349012374149567957_2.profraw -rw-r----- 1 mmoroz primarygroup 94544 Apr 11 18:18 url_unittests.4251353132866778890_0.profraw -rw-r----- 1 mmoroz primarygroup 94544 Apr 11 18:18 url_unittests.4251353132866778890_2.profraw -rw-r----- 1 mmoroz primarygroup 2680384 Apr 11 18:18 url_unittests.4486866546415893753_0.profraw -rw-r----- 1 mmoroz primarygroup 2680384 Apr 11 18:18 url_unittests.4486866546415893753_2.profraw -rw-r----- 1 mmoroz primarygroup 423568 Apr 11 18:18 url_unittests.5616890181013341032_0.profraw -rw-r----- 1 mmoroz primarygroup 423568 Apr 11 18:18 url_unittests.5616890181013341032_2.profraw -rw-r----- 1 mmoroz primarygroup 7872 Apr 11 18:18 url_unittests.5949409236498903469_0.profraw -rw-r----- 1 mmoroz primarygroup 7872 Apr 11 18:18 url_unittests.5949409236498903469_2.profraw -rw-r----- 1 mmoroz primarygroup 748176 Apr 11 18:18 url_unittests.6899576307932548990_0.profraw -rw-r----- 1 mmoroz primarygroup 748176 Apr 11 18:18 url_unittests.6899576307932548990_2.profraw -rw-r----- 1 mmoroz primarygroup 772 Apr 11 18:18 url_unittests_output.txt We need to do the following: 1) figure out all shared libraries built in Chromium checkout and used by those targets (can be done manually for this step) 2) pass those targets to llvm-cov command 3) compare code coverage report with the report generated by non-component build If code coverage report produced by the component build shows the proper data, we'll automate steps 1) and 2) and allow use of component builds for code coverage configuration.
,
Apr 12 2018
Well, with unit_tests there is a little difference, but looks good overall. Compare: https://chrome-coverage.storage.googleapis.com/random_reports/unit_tests_component_false/index.html https://chrome-coverage.storage.googleapis.com/random_reports/unit_tests_component_true/index.html I also had to hack in a sanity check to avoid passing non-instrumented libraries to llvm-cov, as it causes an error without a very clear message. We might want to address that upstream. Also, it's interesting why some libraries are not instrumented. Updated the CL (still a draft though): https://chromium-review.googlesource.com/#/c/chromium/src/+/1010464
,
Apr 17 2018
,
Apr 18 2018
I succeeded in generating code coverage report with component build on the weak bot: n1-highmem-16 (16 vCPUs, 104 GB memory). Numbers look good. You can check out revision 551113 on https://chrome-coverage.storage.googleapis.com/index.html My only concerns so far: 1) numbers for fuzz targets are slightly lower than usual, but likely this is fine, e.g. check revision 550851 from a non-component build 2) the whole process took about 30 hours (~20 hours to build targets), when non-component build usually takes 20 hours (~12 hours to build and run targets), but I suspect that was caused by some issue on a bot / on GCE, as it's absolutely surprising. Component build should be faster than a non-component one. I'll generate a couple more reports on both of the bots and will post another update in a day or two.
,
Apr 18 2018
Well, component build does indeed take less time than a non-component one, e.g. 2h:06m vs 2h:55m to build 87 unittests and 285 fuzz targets on n1-highmem-64 (64 vCPUs, 416 GB memory) with RAM disk. However, time to run everything is significantly different in the opposite direction. Non-component build took 6h:22m while component build took 17h:35m for doing the same job. I'll look into differences between various targets. As of now I only checked a large net_unittests binary and the difference wasn't that huge: component: Running net_unittests real 119m52.341s user 92m30.700s sys 12m44.232s vs. non-component Running net_unittests real 109m29.391s user 87m6.248s sys 10m29.384s
,
Apr 19 2018
Some unit tests take much more time when a component build is used. I wonder if it's always a case for them or it's code coverage instrumentation overhead. Some examples: media_unittests 6m1.286s vs 2m56.995s media_blink_unittests 2m40.272s vs 0m42.661s services_unittests 7m25.721s vs 1m57.832s content_unittests 40m32.811s vs 10m28.722s content_browsertests 348m7.246s vs 55m26.286s It seems like the bigger a test is, the more significant is the difference. In that case, I'd blame code coverage instrumentation, as every library needs to dump its raw profile on exit, and in case of a large browser tests there must be lots of processes and libraries involved.
,
Apr 19 2018
unit_tests 69m13.157s vs 24m25.580s
,
Apr 19 2018
That doesn't mean that we should not use component build for code coverage, we can use both, just need to document the difference. I'll also confirm tomorrow whether code coverage instrumentation is indeed the culprit or not.
,
Apr 19 2018
Another downside of running component build on a bot is that I end up with nearly 27K or *.profraw dumps, which surprisingly doesn't cause any problems, but feels unnecessary complicated.
,
Apr 19 2018
Confirmed, component vs non-component builds have relatively the same performance without code coverage instrumentation.
,
Apr 21 2018
Below are two total reports generated using the same revision: Component build: https://chrome-coverage.storage.googleapis.com/chrome/552183c/index.html vs Non component build: https://chrome-coverage.storage.googleapis.com/chrome/552183n/index.html The numbers are approximately the same. Component build has slightly higher percentage on average, but some parts are higher in non-component build (e.g. ui/ directory). The only big difference I see is in content/browser/: component: https://chrome-coverage.storage.googleapis.com/chrome/552183c/coverage/mnt/ram-disk/chromium/src/content/browser/report.html non-component: https://chrome-coverage.storage.googleapis.com/chrome/552183n/coverage/home/mmoroz/chromium/src/content/browser/report.html For example, https://cs.chromium.org/chromium/src/content/browser/host_zoom_map_impl_browsertest.cc should be covered as per logs, but non-component build shows 0 coverage for that file. However, there are 21 crashes happening during execution of content_browsertests, and it's not clear what to expect as usually hard crashes do not produce coverage dump (see issue 829116).
,
Apr 21 2018
My main suspect is issue 801364 . See the merge log for non-component build: + third_party/llvm-build/Release+Asserts/bin/llvm-profdata merge -sparse dumps/content_browsertests1/dump.12745319232068394803_0.profraw dumps/content_browsertests1/dump.12745319232068394803_1.profraw dumps/content_browsertests1/dump.12745319232068394803_2.profraw dumps/content_browsertests1/dump.12745319232068394803_3.profraw dumps/content_browsertests1/dump.14334184836839687410_0.profraw dumps/content_browsertests1/dump.14334184836839687410_1.profraw dumps/content_browsertests1/dump.14334184836839687410_2.profraw dumps/content_browsertests1/dump.14334184836839687410_3.profraw dumps/content_browsertests1/dump.692416824154059690_0.profraw dumps/content_browsertests1/dump.692416824154059690_1.profraw dumps/content_browsertests1/dump.692416824154059690_2.profraw dumps/content_browsertests1/dump.692416824154059690_3.profraw -o content_browsertests.profdata dumps/content_browsertests1/dump.692416824154059690_0.profraw: _ZN2pp13MacroExpander8getTokenEPNS_5TokenE: Function basic block count change detected (counter mismatch) Make sure that all profile data to be merged is generated from the same binary. dumps/content_browsertests1/dump.692416824154059690_1.profraw: _ZN2pp13MacroExpander8getTokenEPNS_5TokenE: Function basic block count change detected (counter mismatch) warning: dumps/content_browsertests1/dump.14334184836839687410_3.profraw: Function basic block count change detected (counter mismatch) warning: dumps/content_browsertests1/dump.692416824154059690_0.profraw: Function basic block count change detected (counter mismatch) Component build also has that warning (though in a slightly different fashion): + third_party/llvm-build/Release+Asserts/bin/llvm-profdata merge -sparse dumps/content_browsertests1/dump.10037411991047182556_0.profraw dumps/content_browsertests1/dump.10037411991047182556_1.profraw dumps/content_browsertests1/dump.10037411991047182556_2.profraw dumps/content_browsertests1/dump.10037411991047182556_3.profraw <... a lot of dumps from various libraries ... > dumps/content_browsertests1/dump.9945865655082957974_2.profraw dumps/content_browsertests1/dump.9945865655082957974_3.profraw -o content_browsertests.profdata warning: dumps/content_browsertests1/dump.979146987538765567_0.profraw: Function basic block count change detected (counter mismatch) warning: dumps/content_browsertests1/dump.979146987538765567_1.profraw: Function basic block count change detected (counter mismatch) warning: dumps/content_browsertests1/dump.979146987538765567_2.profraw: Function basic block count change detected (counter mismatch) warning: dumps/content_browsertests1/dump.979146987538765567_3.profraw: Function basic block count change detected (counter mismatch)
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d73e45fdf4ab585227bdda1f3fd7e3105ae798ee commit d73e45fdf4ab585227bdda1f3fd7e3105ae798ee Author: Max Moroz <mmoroz@chromium.org> Date: Tue Apr 24 18:32:47 2018 [Code Coverage] Add support for component build + update documentation. Bug: 831939 Change-Id: I2804796894045dc69aa8309aa3ab34e6c80b38d7 Reviewed-on: https://chromium-review.googlesource.com/1010464 Commit-Queue: Max Moroz <mmoroz@chromium.org> Reviewed-by: Yuke Liao <liaoyuke@chromium.org> Reviewed-by: Abhishek Arya <inferno@chromium.org> Cr-Commit-Position: refs/heads/master@{#553211} [modify] https://crrev.com/d73e45fdf4ab585227bdda1f3fd7e3105ae798ee/build/config/coverage/coverage.gni [modify] https://crrev.com/d73e45fdf4ab585227bdda1f3fd7e3105ae798ee/docs/code_coverage.md [modify] https://crrev.com/d73e45fdf4ab585227bdda1f3fd7e3105ae798ee/tools/code_coverage/coverage.py
,
Apr 24 2018
|
|||
►
Sign in to add a comment |
|||
Comment 1 by mmoroz@chromium.org
, Apr 12 2018