This is great idea btw!!! Leveraging code coverage for performance!

FTR, that was proposed by sadrul@ over email, I've just moved the discussion to here :)

+vmiura@ who actually had the idea! :)

I am slowly starting to look at this.

Given two profiles, are there some tools to show the difference between the two profiles? This would be useful to compare and contrast hotspots with various feature flags.

There aren't any tools that I know of. However, you can do "llvm-cov export" to get coverage data in JSON format. Then you can load two JSONs to compare them. Here is an example of how it looks: https://paste.googleplex.com/5162657116061696

I have a CL ready that would write summary.json as part of the report generated by coverage.py. If you start building some tool / script for diffing, assume that you'd have JSON files generated by the existing script :)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5aa6275b4f8a9668c1376e7ffc36bc496d41f6f5

commit 5aa6275b4f8a9668c1376e7ffc36bc496d41f6f5
Author: Sadrul Habib Chowdhury <sadrul@chromium.org>
Date: Fri May 04 23:07:49 2018

code coverage: Fix writing profiles from child processes.

Notable changes:
. Introduce CLANG_COVERAGE define if use_clang_coverage gn arg is set.
. For child processes to be able to correctly write the coverage profile
  during teardown, it is necessary to allow them to shut-down cleanly.
  So avoid terminating the processes from ~ChildProcessLauncher() when
  CLANG_COVERAGE is set.
. Child processes can do fast-shutdown by calling _exit(0). So force the
  profile file to be written from the child process before calling that.

With both changes, it is still necessary to run with --no-sandbox, since
otherwise the child processes are not able to write to the profile file.

BUG=838582,  834781 

Change-Id: I742521e756a7dead983f40462798f7c4dac2ac02
Reviewed-on: https://chromium-review.googlesource.com/1041271
Reviewed-by: Max Moroz <mmoroz@chromium.org>
Reviewed-by: Antoine Labour <piman@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Sadrul Chowdhury <sadrul@chromium.org>
Cr-Commit-Position: refs/heads/master@{#556236}
[modify] https://crrev.com/5aa6275b4f8a9668c1376e7ffc36bc496d41f6f5/build/config/BUILD.gn
[modify] https://crrev.com/5aa6275b4f8a9668c1376e7ffc36bc496d41f6f5/content/browser/child_process_launcher.cc
[modify] https://crrev.com/5aa6275b4f8a9668c1376e7ffc36bc496d41f6f5/content/child/child_thread_impl.cc

mmoroz@: Thanks! The json should be useful!

+rharrison@ fyi (might be somehow related to issue 817798)

Fwiw, I originally had this idea that if we collected coverage from Top 10k sites in Cluster Telemetry, we could use that data for some interesting things such as:

 - Produce more optimized sets of sites that are still representative.  That could be a small enough set to run on a continuous basis.  i.e. sort sites by how much incremental coverage value they provide.
 - Identify which sites stress particular code the most, for analysis, grouping, etc.

Friendly ping! Any progress here? :)

Also noticed that I haven't copied contents of my email on chrome-code-covererage@ that has some concrete suggestions on how to approach this problem:

Basically, the code coverage we're using is based on LLVM profiler instrumentation. That means, instead of generating code coverage reports, we can also use profiler dumps (== coverage dumps) for analyzing performance or finding hotspots. For example, I took a .profdata file from zlib_uncompress_fuzzer (.profraw would work as well, but .profdata is kind of optimized):

$ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts dumps/zlib_uncompress_fuzzer/dump.profdata 
Instrumentation level: Front-end
Total functions: 67
Maximum function count: 2626960
Maximum internal block count: 697040


# Let's take a look at the top 10 hotspots.

$ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -topn=10 dumps/zlib_uncompress_fuzzer/dump.profdata 
Instrumentation level: Front-end
Total functions: 67
Maximum function count: 2626960
Maximum internal block count: 697040
Top 10 functions with the largest internal block counts: 
  adler32_simd.c:_mm_add_epi32, max count = 2626960
  adler32_simd.c:_mm_madd_epi16, max count = 1044290
  adler32_simd.c:_mm_sad_epu8, max count = 1044290
  adler32_simd.c:_mm_loadu_si128, max count = 1044290
  adler32_simd.c:_mm_maddubs_epi16, max count = 1044290
  inffast_chunk.c:storechunk, max count = 861995
  inffast_chunk.c:loadchunk, max count = 861995
  Cr_z_inflate_table, max count = 697040
  inffast_chunk.c:chunkcopy_core, max count = 670692
  Cr_z_inflate, max count = 635065