My first guess is that it spends too much time on writing JSON output, as it does lots of output: https://github.com/llvm-mirror/llvm/blob/master/tools/llvm-cov/CoverageExporterJson.h#L46

Other than that, it's just reading and parsing the .profdata + prepareFileReports() function that works pretty fast on multiple CPUs.

Did some measurements, JSON rendering takes at least 90% of all time spent.

Using ostringbuffer for accumulating data before doing an actual write doesn't help.

After digging deeper, it turns out that coverage segments and expansions are still calculated in -summary-only mode during individual files rendering, and that takes most of the time spent. However, those values are not needed as summary data is already pre-calculated during prepareFileReports() and that's all we need.

I made a change to skip that step. That improves export speed drastically (10-100x) on small tests such as crypto_unittests or url_unittests. I'm measuring time for a bigger target and cleaning up my CL now.

renderFiles() step now takes less than a second even for a huge target. Yesterday, that step + our post-processing in Python (shouldn't add too much) took nearly 8 hours.


$ time third_party/llvm-build/Release+Asserts/bin/llvm-cov-export-superb export -summary-only -instr-profile=dumps/merged.profdata out/coverage/courgette_unittests -object=out/coverage/gpu_unittests \
> -object=out/coverage/headless_unittests \
> -object=out/coverage/audio_unittests \
> -object=out/coverage/media_unittests \
> -object=out/coverage/media_blink_unittests \
> -object=out/coverage/media_mojo_unittests \
> -object=out/coverage/media_service_unittests \
> -object=out/coverage/net_unittests \
> -object=out/coverage/services_unittests \
> -object=out/coverage/service_manager_unittests \
> -object=out/coverage/skia_unittests \
> -object=out/coverage/storage_unittests \
> -object=out/coverage/blink_heap_unittests \
> -object=out/coverage/wtf_unittests \
> -object=out/coverage/blink_common_unittests \
> -object=out/coverage/angle_unittests \
> -object=out/coverage/pdfium_unittests \
> -object=out/coverage/accessibility_unittests \
> -object=out/coverage/gfx_unittests \
> -object=out/coverage/gl_unittests \
> -object=out/coverage/keyboard_unittests \
> -object=out/coverage/snapshot_unittests \
> -object=out/coverage/views_unittests \
> -object=out/coverage/wm_unittests \
> -object=out/coverage/url_unittests \
> -object=out/coverage/breakpad_unittests \
> -object=out/coverage/content_unittests \
> -object=out/coverage/mojo_common_unittests \
> -object=out/coverage/sql_unittests \
> -object=out/coverage/unit_tests \
> -object=out/coverage/cc_blink_unittests \
> -object=out/coverage/crypto_unittests \
> -object=out/coverage/pdf_unittests \
> -object=out/coverage/swiftshader_unittests > merged.json
Started: Fri Jan 12 18:13:13 2018

warning: 905 functions have mismatched data
renderRoot(), getting unique source files: Fri Jan 12 18:20:43 2018

renderRoot(sourceFiles): Fri Jan 12 18:20:49 2018

prepareFileReports: Fri Jan 12 18:20:49 2018

renderFiles: Fri Jan 12 18:30:03 2018

render leftovers: Fri Jan 12 18:30:03 2018


real	16m55.750s
user	302m39.684s
sys	0m10.332s

https://reviews.llvm.org/D42000

Landed upstream: https://reviews.llvm.org/rL322397

Correction to my comment c#4: 8 hours is a wrong estimate. Correct measurements are:

1) before fixing this:

Total time spent on llvm-cov + llvm-export + postprocessing:

real	425m33.540s
user	1615m8.520s
sys	54m3.004s


2) after fixing this:

real  95m53.463s
user  1291m4.512s
sys   53m27.744s