Another bunch of warnings from doing the same stuff on a newer revision:

$ time third_party/llvm-build/Release+Asserts/bin/llvm-profdata merge -sparse dumps/*/*.profraw -o dumps/merged.profdata
dumps/angle_unittests/dump.18316999429472973738_5.profraw: pthread_equal: Counter overflow
dumps/angle_unittests/dump.18316999429472973738_3.profraw: pthread_equal: Counter overflow
dumps/angle_unittests/dump.18316999429472973738_2.profraw: pthread_equal: Counter overflow
dumps/gfx_unittests/dump.386878935451104170_6.profraw: pthread_equal: Counter overflow
dumps/media_mojo_unittests/dump.2734634062402409978_0.profraw: pthread_equal: Counter overflow
dumps/media_mojo_unittests/dump.2734634062402409978_1.profraw: pthread_equal: Counter overflow
dumps/media_mojo_unittests/dump.2734634062402409978_3.profraw: pthread_equal: Counter overflow
dumps/media_service_unittests/dump.2315245595604087722_6.profraw: pthread_equal: Counter overflow
dumps/media_service_unittests/dump.2315245595604087722_6.profraw: pthread_equal: Counter overflow
dumps/media_mojo_unittests/dump.2734634062402409978_0.profraw: pthread_equal: Counter overflow
dumps/media_mojo_unittests/dump.2734634062402409978_1.profraw: pthread_equal: Counter overflow
dumps/media_mojo_unittests/dump.2734634062402409978_3.profraw: pthread_equal: Counter overflow
dumps/media_unittests/dump.4991571564513522602_2.profraw: pthread_equal: Counter overflow
dumps/services_unittests/dump.12746703418510548787_1.profraw: _ZN2pp13MacroExpander8getTokenEPNS_5TokenE: Function basic block count change detected (counter mismatch)
Make sure that all profile data to be merged is generated from the same binary.
dumps/services_unittests/dump.12746703418510548787_2.profraw: _ZN2pp13MacroExpander8getTokenEPNS_5TokenE: Function basic block count change detected (counter mismatch)
dumps/services_unittests/dump.12746703418510548787_3.profraw: _ZN2pp13MacroExpander8getTokenEPNS_5TokenE: Function basic block count change detected (counter mismatch)
dumps/content_unittests/dump.1296838580397994922_4.profraw: pthread_equal: Counter overflow
dumps/content_unittests/dump.1296838580397994922_2.profraw: pthread_equal: Counter overflow
dumps/media_unittests/dump.4991571564513522602_6.profraw: pthread_equal: Counter overflow
dumps/services_unittests/dump.12746703418510548787_7.profraw: _ZN2pp13MacroExpander8getTokenEPNS_5TokenE: Function basic block count change detected (counter mismatch)
dumps/services_unittests/dump.12746703418510548787_4.profraw: _ZN2pp13MacroExpander8getTokenEPNS_5TokenE: Function basic block count change detected (counter mismatch)
dumps/gpu_unittests/dump.5461668470095934378_7.profraw: pthread_equal: Counter overflow
dumps/gpu_unittests/dump.5461668470095934378_5.profraw: pthread_equal: Counter overflow
dumps/gpu_unittests/dump.5461668470095934378_7.profraw: pthread_equal: Counter overflow
dumps/gpu_unittests/dump.5461668470095934378_5.profraw: pthread_equal: Counter overflow
dumps/unit_tests/dump.11404473204906193834_0.profraw: pthread_equal: Counter overflow
dumps/unit_tests/dump.11404473204906193834_1.profraw: pthread_equal: Counter overflow
dumps/unit_tests/dump.11404473204906193834_2.profraw: pthread_equal: Counter overflow
dumps/unit_tests/dump.11404473204906193834_3.profraw: pthread_equal: Counter overflow
dumps/unit_tests/dump.11404473204906193834_0.profraw: pthread_equal: Counter overflow
dumps/unit_tests/dump.11404473204906193834_1.profraw: pthread_equal: Counter overflow
dumps/unit_tests/dump.11404473204906193834_6.profraw: pthread_equal: Counter overflow
dumps/unit_tests/dump.11404473204906193834_7.profraw: pthread_equal: Counter overflow
warning: dumps/views_unittests/dump.6610692226059982330_0.profraw: Counter overflow
warning: dumps/views_unittests/dump.6610692226059982330_1.profraw: Counter overflow
warning: dumps/views_unittests/dump.6610692226059982330_2.profraw: Counter overflow
warning: dumps/views_unittests/dump.6610692226059982330_3.profraw: Counter overflow
warning: dumps/views_unittests/dump.6610692226059982330_4.profraw: Counter overflow
warning: dumps/views_unittests/dump.6610692226059982330_5.profraw: Counter overflow
warning: dumps/views_unittests/dump.6610692226059982330_6.profraw: Counter overflow
warning: dumps/views_unittests/dump.6610692226059982330_7.profraw: Counter overflow
warning: dumps/wm_unittests/dump.15835933278296446817_3.profraw: Counter overflow
warning: dumps/wm_unittests/dump.5416851889370493866_0.profraw: Counter overflow
warning: dumps/wm_unittests/dump.5416851889370493866_2.profraw: Counter overflow
warning: dumps/wm_unittests/dump.5416851889370493866_3.profraw: Counter overflow
warning: dumps/wtf_unittests/dump.16755835025050891178_0.profraw: Counter overflow
warning: dumps/wtf_unittests/dump.16755835025050891178_2.profraw: Counter overflow
warning: dumps/wtf_unittests/dump.16755835025050891178_4.profraw: Counter overflow
warning: dumps/wtf_unittests/dump.16755835025050891178_7.profraw: Counter overflow
warning: dumps/swiftshader_unittests/dump.12746703418510548787_5.profraw: Function basic block count change detected (counter mismatch)
warning: dumps/swiftshader_unittests/dump.12746703418510548787_6.profraw: Function basic block count change detected (counter mismatch)
error: dumps/service_manager_unittests/dump.14882662155174628336_0.profraw: Invalid instrumentation profile data (file header is corrupt)

real	2m54.935s
user	19m23.004s
sys	3m54.036s

And another bunch of warnings observed when merging dumps from unittests + fuzzers:

$ time third_party/llvm-build/Release+Asserts/bin/llvm-profdata merge -sparse dumps/*/*.profraw fuzzer_dumps/*/dump.profraw -o tests_and_fuzzers.profdata
dumps/angle_unittests/dump.18316999429472973738_5.profraw: pthread_equal: Counter overflow
dumps/angle_unittests/dump.18316999429472973738_2.profraw: pthread_equal: Counter overflow
dumps/angle_unittests/dump.18316999429472973738_3.profraw: pthread_equal: Counter overflow
dumps/gfx_unittests/dump.386878935451104170_6.profraw: pthread_equal: Counter overflow
dumps/media_mojo_unittests/dump.2734634062402409978_1.profraw: pthread_equal: Counter overflow
dumps/media_mojo_unittests/dump.2734634062402409978_0.profraw: pthread_equal: Counter overflow
dumps/media_mojo_unittests/dump.2734634062402409978_3.profraw: pthread_equal: Counter overflow
dumps/media_service_unittests/dump.2315245595604087722_6.profraw: pthread_equal: Counter overflow
dumps/media_service_unittests/dump.2315245595604087722_6.profraw: pthread_equal: Counter overflow
dumps/media_mojo_unittests/dump.2734634062402409978_1.profraw: pthread_equal: Counter overflow
dumps/media_mojo_unittests/dump.2734634062402409978_0.profraw: pthread_equal: Counter overflow
dumps/media_mojo_unittests/dump.2734634062402409978_3.profraw: pthread_equal: Counter overflow
dumps/media_unittests/dump.4991571564513522602_2.profraw: pthread_equal: Counter overflow
dumps/services_unittests/dump.12746703418510548787_2.profraw: _ZN2pp13MacroExpander8getTokenEPNS_5TokenE: Function basic block count change detected (counter mismatch)
Make sure that all profile data to be merged is generated from the same binary.
dumps/services_unittests/dump.12746703418510548787_3.profraw: _ZN2pp13MacroExpander8getTokenEPNS_5TokenE: Function basic block count change detected (counter mismatch)
dumps/services_unittests/dump.12746703418510548787_4.profraw: _ZN2pp13MacroExpander8getTokenEPNS_5TokenE: Function basic block count change detected (counter mismatch)
dumps/content_unittests/dump.1296838580397994922_2.profraw: pthread_equal: Counter overflow
dumps/content_unittests/dump.1296838580397994922_4.profraw: pthread_equal: Counter overflow
dumps/media_unittests/dump.4991571564513522602_6.profraw: pthread_equal: Counter overflow
dumps/sql_unittests/dump.16602651782091699114_4.profraw: pthread_equal: Counter overflow
dumps/sql_unittests/dump.16602651782091699114_4.profraw: pthread_equal: Counter overflow
dumps/gpu_unittests/dump.5461668470095934378_7.profraw: pthread_equal: Counter overflow
dumps/gpu_unittests/dump.5461668470095934378_5.profraw: pthread_equal: Counter overflow
dumps/services_unittests/dump.12746703418510548787_7.profraw: _ZN2pp13MacroExpander8getTokenEPNS_5TokenE: Function basic block count change detected (counter mismatch)
dumps/wm_unittests/dump.5416851889370493866_7.profraw: pthread_equal: Counter overflow
dumps/gpu_unittests/dump.5461668470095934378_7.profraw: pthread_equal: Counter overflow
dumps/gpu_unittests/dump.5461668470095934378_5.profraw: pthread_equal: Counter overflow
dumps/wm_unittests/dump.5416851889370493866_7.profraw: pthread_equal: Counter overflow
dumps/wtf_unittests/dump.16755835025050891178_7.profraw: pthread_equal: Counter overflow
dumps/wtf_unittests/dump.16755835025050891178_7.profraw: pthread_equal: Counter overflow
fuzzer_dumps/gpu_swiftshader_fuzzer/dump.profraw: _ZN2pp13MacroExpander8getTokenEPNS_5TokenE: Function basic block count change detected (counter mismatch)
dumps/unit_tests/dump.11404473204906193834_2.profraw: pthread_equal: Counter overflow
fuzzer_dumps/mediasource_WEBM_VP9_pipeline_integration_fuzzer/dump.profraw: pthread_equal: Counter overflow
dumps/unit_tests/dump.11404473204906193834_1.profraw: pthread_equal: Counter overflow
fuzzer_dumps/mediasource_WEBM_VP9_pipeline_integration_fuzzer/dump.profraw: pthread_equal: Counter overflow
dumps/unit_tests/dump.11404473204906193834_3.profraw: pthread_equal: Counter overflow
fuzzer_dumps/blink_text_codec_UTF_8_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/blink_text_codec_WINDOWS_1252_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/blink_png_decoder_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/clear_site_data_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
dumps/unit_tests/dump.11404473204906193834_2.profraw: pthread_equal: Counter overflow
fuzzer_dumps/css_parser_proto_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/feature_policy_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
dumps/unit_tests/dump.11404473204906193834_3.profraw: pthread_equal: Counter overflow
fuzzer_dumps/content_security_policy_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
dumps/unit_tests/dump.11404473204906193834_1.profraw: pthread_equal: Counter overflow
dumps/unit_tests/dump.11404473204906193834_0.profraw: pthread_equal: Counter overflow
fuzzer_dumps/css_parser_fast_paths_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
dumps/unit_tests/dump.11404473204906193834_6.profraw: pthread_equal: Counter overflow
fuzzer_dumps/html_preload_scanner_fuzzer/dump.profraw: _ZN2pp13MacroExpander8getTokenEPNS_5TokenE: Function basic block count change detected (counter mismatch)
dumps/unit_tests/dump.11404473204906193834_7.profraw: pthread_equal: Counter overflow
fuzzer_dumps/profiling_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/xml_parser_fuzzer/dump.profraw: pthread_equal: Counter overflow
fuzzer_dumps/origin_manifest_parser_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/origin_trial_token_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/mhtml_parser_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/xml_parser_fuzzer/dump.profraw: pthread_equal: Counter overflow
fuzzer_dumps/template_url_parser_fuzzer/dump.profraw: pthread_equal: Counter overflow
fuzzer_dumps/swiftshader_vertex_routine_fuzzer/dump.profraw: _ZN2pp13MacroExpander8getTokenEPNS_5TokenE: Function basic block count change detected (counter mismatch)
fuzzer_dumps/renderer_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/template_url_parser_fuzzer/dump.profraw: pthread_equal: Counter overflow
fuzzer_dumps/web_icon_sizes_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/blink_harfbuzz_shaper_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/stylesheet_contents_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/text_resource_decoder_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/blink_json_parser_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/blink_html_tokenizer_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
fuzzer_dumps/v8_serialized_script_value_fuzzer/dump.profraw: _ZN2sh17ConstructCompilerEj12ShShaderSpec14ShShaderOutput: Function basic block count change detected (counter mismatch)
warning: fuzzer_dumps/v8_wasm_names_section_fuzzer/dump.profraw: Counter overflow
warning: fuzzer_dumps/v8_wasm_types_section_fuzzer/dump.profraw: Counter overflow
warning: fuzzer_dumps/vp8_depacketizer_fuzzer/dump.profraw: Counter overflow
warning: fuzzer_dumps/vp9_qp_parser_fuzzer/dump.profraw: Counter overflow
warning: fuzzer_dumps/webcrypto_ec_import_key_pkcs8_fuzzer/dump.profraw: Counter overflow
warning: fuzzer_dumps/webcrypto_rsa_import_key_spki_fuzzer/dump.profraw: Counter overflow
warning: fuzzer_dumps/web_icon_sizes_fuzzer/dump.profraw: Counter overflow
warning: fuzzer_dumps/webusb_descriptors_fuzzer/dump.profraw: Counter overflow
warning: fuzzer_dumps/xml_parser_fuzzer/dump.profraw: Counter overflow
warning: fuzzer_dumps/zlib_deflate_set_dictionary_fuzzer/dump.profraw: Counter overflow
warning: fuzzer_dumps/zlib_uncompress_fuzzer/dump.profraw: Counter overflow
warning: fuzzer_dumps/zucchini_patch_fuzzer/dump.profraw: Function basic block count change detected (counter mismatch)
warning: fuzzer_dumps/sign_response_data_fuzzer/dump.profraw: Counter overflow
warning: fuzzer_dumps/skia_color_space_fuzzer/dump.profraw: Function basic block count change detected (counter mismatch)
warning: fuzzer_dumps/skia_image_filter_proto_fuzzer/dump.profraw: Counter overflow
warning: fuzzer_dumps/skia_path_fuzzer/dump.profraw: Counter overflow
warning: fuzzer_dumps/sqlite3_ossfuzz_fuzzer/dump.profraw: Function basic block count change detected (counter mismatch)
warning: fuzzer_dumps/sqlite3_prepare_v2_fuzzer/dump.profraw: Function basic block count change detected (counter mismatch)
warning: fuzzer_dumps/string_to_int_fuzzer/dump.profraw: Function basic block count change detected (counter mismatch)
warning: fuzzer_dumps/stylesheet_contents_fuzzer/dump.profraw: Counter overflow

real	4m28.893s
user	30m11.836s
sys	3m7.184s

Messages like

  dumps/angle_unittests/dump.18316999429472973738_5.profraw: pthread_equal: Counter overflow

come from https://github.com/llvm-mirror/llvm/blob/master/tools/llvm-profdata/llvm-profdata.cpp#L197 and mean that we have an overflown counter for function pthread_equal.


Warnings like:

  warning: dumps/wtf_unittests/dump.16755835025050891178_7.profraw: Counter overflow

come from https://github.com/llvm-mirror/llvm/blob/master/tools/llvm-profdata/llvm-profdata.cpp#L301 and are also related to pthread_equal function.




$ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -function=pthread_equal out/2_report-viz-4m_2/viz_unittests.6671870263282226682_3.profraw 
Counters:
  pthread_equal:
    Hash: 0x000000000000061f
    Counters: 1
    Function count: 15098326761600978439
    Block counts: []
  pthread_equal:
    Hash: 0x000000000000061f
    Counters: 1
    Function count: 15098326761600978439
    Block counts: []
Instrumentation level: Front-end
Functions shown: 2
Total functions: 345581
Maximum function count: 15098326761600978439
Maximum internal block count: 398229543


While other functions have more adequate counts:

$ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -function=printf out/2_report-viz-4m_2/viz_unittests.6671870263282226682_3.profraw 
Counters:
  _ZN16TCMalloc_Printer6printfEPKcz:
    Hash: 0xa72fe7cac932f2eb
    Counters: 4
    Function count: 0
    Block counts: [0, 0, 0]
  vprintf:
    Hash: 0x0000000000000018
    Counters: 1
    Function count: 1670
    Block counts: []
  _ZN8SkString6printfEPKcz:
    Hash: 0xf22a4346d5f42103
    Counters: 6
    Function count: 14
    Block counts: [0, 0, 14, 0, 0]
<...>
  stringprintf.cc:_ZN4base12_GLOBAL__N_114StringAppendVTINSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEEEEvPT_PKNS9_10value_typeEP13__va_list_tag:
    Hash: 0xf999efcca27ff438
    Counters: 11
    Function count: 214219
    Block counts: [214219, 214219, 0, 0, 0, 0, 0, 0, 0, 0]
  stringprintf.cc:_ZN4base12_GLOBAL__N_110vsnprintfTEPcmPKcP13__va_list_tag:
    Hash: 0x0000000000000018
    Counters: 1
    Function count: 214219
    Block counts: []
<...>

That large value is written into the .profraw file, i.e. it's been somehow generated when the target program was running.

Given that crazy large value for pthread_equal count, I suspect that there is some use of uninitialized value sneaked in. Some information proving my theory:

>>> hex(15098326761600978439)
'0xd188083200000e07L'
>>> data = open('out/2_report-viz-4m_2/viz_unittests.6671870263282226682_3.profraw').read()
>>> 

>>> data.find('d188083200000e07'.decode('hex')[::-1])
16617376
>>> data.count('d188083200000e07'.decode('hex')[::-1])
1

i.e. there is only one counter with that particular value. Ok, what if higher 4 bytes are junk, lets look for lower 4 bytes only:

>>> 0xe07
3591
>>> data.count('0344cb'.decode('hex')[::-1])
10

3591 is quite a reasonable value, right? Let's see if there are functions having that count value:

$ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -all-functions out/2_report-viz-4m_2/viz_unittests.6671870263282226682_3.profraw | egrep 'count: 3591' -B 3
  _ZNSt3__16__treeINS_12__value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEES7_EENS_19__map_value_compareIS7_S8_NS_4lessIvEELb1EEENS5_IS8_EEE16__find_leaf_highERPNS_15__tree_end_nodeIPNS_16__tree_node_baseIPvEEEERKS7_:
    Hash: 0x01467bf72cf1d6b4
    Counters: 6
    Function count: 3591
--
  _ZNSt3__122__tree_key_value_typesINS_12__value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEES7_EEE9__get_keyERKS8_:
    Hash: 0x0000000000000018
    Counters: 1
    Function count: 3591
--
  xmlIsMainThread:
    Hash: 0x029f49129f498458
    Counters: 3
    Function count: 3591


The last one looks suspicious: xmlIsMainThread. Let's see its source code: https://cs.chromium.org/chromium/src/third_party/libxml/src/threads.c?q=xmlIsMainThread&sq=package:chromium&dr=CSs&l=793

Wow, it seems to return (pthread_equal(mainthread,pthread_self()));

That means we have two options:

1) pthread_equal is being called 0xd188083200000000 (15098326761600974848) times from other places

2) higher 4 bytes of the pthread_equal count are either overwritten of uninitialized

I'm leaning towards the second version. Looking more.

And another detail that bother me is that the overflow is happening when two records for pthread_equal are merged. Basically, 15098326761600978439 + 15098326761600978439 doesn't fit into 64 bits.

There are some other functions that also have two records, but all of them seem to have 0 count:

$ third_party/llvm-build/Release+Asserts/bin/llvm-profdata merge -sparse out/2_report-viz-4m_2/viz_unittests.6671870263282226682_3.profraw -o 3.profdata
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_setzero_si128
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_set1_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_set1_epi32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_undefined_si128
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_unpackhi_epi32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_unpacklo_epi32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_madd_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_add_epi32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_srai_epi32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_packs_epi32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_unpacklo_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_unpackhi_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_add_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_sub_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_srai_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_cvtsi32_si128
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_unpacklo_epi8
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_packus_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_cvtsi128_si32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:dct_const_round_shift
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_set_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_slli_epi32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_sub_epi32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_unpacklo_epi64
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_unpackhi_epi64
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_adds_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_loadl_epi64
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_storel_epi64
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_sse2.c:_mm_load_si128
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_set1_epi16
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_unpacklo_epi64
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_loadl_epi64
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_unpacklo_epi8
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_cmpeq_epi8
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_unpackhi_epi64
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:abs_diff
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_max_epu8
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_setzero_si128
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_cmpgt_epi16
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_packs_epi16
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_adds_epu8
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_unpackhi_epi8
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_srli_epi16
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_subs_epu8
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_and_si128
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_set_epi8
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_set1_epi8
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_xor_si128
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_subs_epi8
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_adds_epi8
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_srai_epi16
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_andnot_si128
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_cvtsi128_si32
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_unpacklo_epi16
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_unpackhi_epi32
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_unpackhi_epi16
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_unpacklo_epi32
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_load_si128
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_castps_si128
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_loadh_pi
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_castsi128_ps
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_undefined_si128
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_subs_epi16
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_adds_epi16
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_add_epi16
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_packus_epi16
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_sub_epi16
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_or_si128
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_storel_epi64
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_loadu_si128
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_cmpgt_epi8
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_storeu_si128
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:transpose8x16
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:transpose
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:filter8_mask
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:filter_add2_sub2
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_slli_epi16
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:filter16_mask
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_storel_pd
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_castsi128_pd
0x0 + 0x0
// We're updating a function we've seen before.
loopfilter_sse2.c:_mm_storeh_pd
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_setzero_si128
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_set1_epi32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_set1_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_set_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_unpacklo_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_unpackhi_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_unpacklo_epi32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_unpackhi_epi32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_unpacklo_epi64
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_unpackhi_epi64
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_madd_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_add_epi32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_srai_epi32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_packs_epi32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_add_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_sub_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_adds_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_srai_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_loadl_epi64
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_unpacklo_epi8
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_packus_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_storel_epi64
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_mulhrs_epi16
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:add_sub_butterfly
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:recon_and_store
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:store_buffer_8x32
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm_ssse3.c:_mm_load_si128
0x0 + 0x0
// We're updating a function we've seen before.
media_param_traits.cc:_ZN3IPCL10WriteParamIiEEvPN4base6PickleERKT_
0x0 + 0x0
// We're updating a function we've seen before.
media_param_traits.cc:_ZN3IPCL9ReadParamIiEEbPKN4base6PickleEPNS1_14PickleIteratorEPT_
0x0 + 0x0
// We're updating a function we've seen before.
media_param_traits.cc:_ZN3IPCL10WriteParamIbEEvPN4base6PickleERKT_
0x0 + 0x0
// We're updating a function we've seen before.
media_param_traits.cc:_ZN3IPCL10WriteParamINSt3__16vectorIhNS1_9allocatorIhEEEEEEvPN4base6PickleERKT_
0x0 + 0x0
// We're updating a function we've seen before.
media_param_traits.cc:_ZN3IPCL10WriteParamIjEEvPN4base6PickleERKT_
0x0 + 0x0
// We're updating a function we've seen before.
media_param_traits.cc:_ZN3IPCL9ReadParamIbEEbPKN4base6PickleEPNS1_14PickleIteratorEPT_
0x0 + 0x0
// We're updating a function we've seen before.
media_param_traits.cc:_ZN3IPCL9ReadParamINSt3__16vectorIhNS1_9allocatorIhEEEEEEbPKN4base6PickleEPNS6_14PickleIteratorEPT_
0x0 + 0x0
// We're updating a function we've seen before.
media_param_traits.cc:_ZN3IPCL9ReadParamIjEEbPKN4base6PickleEPNS1_14PickleIteratorEPT_
0x0 + 0x0
// We're updating a function we've seen before.
media_param_traits.cc:_ZN3IPCL8LogParamIiEEvRKT_PNSt3__112basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEE
0x0 + 0x0
// We're updating a function we've seen before.
media_param_traits.cc:_ZN3IPCL8LogParamIbEEvRKT_PNSt3__112basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEE
0x0 + 0x0
// We're updating a function we've seen before.
media_param_traits.cc:_ZN3IPCL8LogParamINSt3__16vectorIhNS1_9allocatorIhEEEEEEvRKT_PNS1_12basic_stringIcNS1_11char_traitsIcEENS3_IcEEEE
0x0 + 0x0
// We're updating a function we've seen before.
media_param_traits.cc:_ZN3IPCL8LogParamIjEEvRKT_PNSt3__112basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEE
0x0 + 0x0
// We're updating a function we've seen before.
media_param_traits.cc:_ZL16SkNO_RETURN_HINTv
0x0 + 0x0
// We're updating a function we've seen before.
video_encode_accelerator.mojom.cc:_ZL16SkNO_RETURN_HINTv
0x0 + 0x0
// We're updating a function we've seen before.
pthread_equal
0xd188083200000e07 + 0xd188083200000e07
out/2_report-viz-4m_2/viz_unittests.6671870263282226682_3.profraw: pthread_equal: Counter overflow (instr)
// We're updating a function we've seen before.
gfx_param_traits.cc:_ZN3IPCL10WriteParamImEEvPN4base6PickleERKT_
0x0 + 0x0
// We're updating a function we've seen before.
gfx_param_traits.cc:_ZN3IPCL9ReadParamImEEbPKN4base6PickleEPNS1_14PickleIteratorEPT_
0x0 + 0x0
// We're updating a function we've seen before.
gfx_param_traits.cc:_ZN3IPCL8LogParamIiEEvRKT_PNSt3__112basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEE
0x0 + 0x0
// We're updating a function we've seen before.
gfx_param_traits.cc:_ZN3IPCL10WriteParamIiEEvPN4base6PickleERKT_
0x0 + 0x0
// We're updating a function we've seen before.
gfx_param_traits.cc:_ZN3IPCL9ReadParamIiEEbPKN4base6PickleEPNS1_14PickleIteratorEPT_
0x0 + 0x0
// We're updating a function we've seen before.
gfx_param_traits.cc:_ZN3IPCL8LogParamIiEEvRKT_PNSt3__112basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEE
0x0 + 0x0
// We're updating a function we've seen before.
intrapred.c:d117_predictor
0x0 + 0x0
// We're updating a function we've seen before.
intrapred.c:d135_predictor
0x0 + 0x0
// We're updating a function we've seen before.
intrapred.c:d153_predictor
0x0 + 0x0
// We're updating a function we've seen before.
intrapred.c:v_predictor
0x0 + 0x0
// We're updating a function we've seen before.
intrapred.c:h_predictor
0x0 + 0x0
// We're updating a function we've seen before.
intrapred.c:dc_128_predictor
0x0 + 0x0
// We're updating a function we've seen before.
intrapred.c:dc_left_predictor
0x0 + 0x0
// We're updating a function we've seen before.
intrapred.c:dc_top_predictor
0x0 + 0x0
// We're updating a function we've seen before.
intrapred.c:clip_pixel
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm.c:check_range
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm.c:clip_pixel_add
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm.c:dct_const_round_shift
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm.c:highbd_clip_pixel_add
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm.c:clip_pixel
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm.c:clip_pixel_highbd
0x0 + 0x0
// We're updating a function we've seen before.
inv_txfm.c:clamp
0x0 + 0x0
// We're updating a function we've seen before.
prob.c:tree_merge_probs_impl
0x0 + 0x0
// We're updating a function we've seen before.
prob.c:mode_mv_merge_probs
0x0 + 0x0
// We're updating a function we've seen before.
prob.c:get_prob
0x0 + 0x0
// We're updating a function we've seen before.
prob.c:weighted_prob
0x0 + 0x0
// We're updating a function we've seen before.
reconintra.c:once
0x0 + 0x0
// We're updating a function we've seen before.
yv12extend.c:extend_plane
0x0 + 0x0
// We're updating a function we've seen before.
display.cc:_ZL16SkNO_RETURN_HINTv
0x0 + 0x0

Looked through pthread_equal counts in dumps generated with %p:

0xe07 + 0xe07
0xcb + 0xcb
0x249 + 0x249
0x272 + 0x272
0x2f + 0x2f
0x4d + 0x4d
0x19 + 0x19
0x303 + 0x303
0x226c + 0x226c
0xba7 + 0xba7
0x78b + 0x78b
0x112 + 0x112
0xf6 + 0xf6
0x13 + 0x13
0xbf + 0xbf
0x140 + 0x140
0x7f + 0x7f
0x12e + 0x12e
0x70 + 0x70
0x69 + 0x69
0x2a + 0x2a
0x134 + 0x134
0xf5 + 0xf5
0x102 + 0x102
0xd1 + 0xd1
0x362 + 0x362
0x1e1 + 0x1e1
0x1f + 0x1f

(others are 0x0 + 0x0)

total is 0xb9d6 (47574)

-----------------------------------

For comparison, below are numbers from 3 different runs with %4m:

0x35e2000000000 + 0x35e2000000000
0x1596f94640000e07 + 0x1596f94640000e07
0x307000000000 + 0x307000000000
0x35251b9000000000 + 0x35251b9000000000

total: 0x957f46cc80001c0eL (10772406677828279310L)

0xc0e57000000000 + 0xc0e57000000000
0x3293c00000e07 + 0x3293c00000e07
0x91aecdb20000000 + 0x91aecdb20000000
0x10849000000000 + 0x10849000000000

total: 0x13df002e40001c0e (1431863405169351694)

0xe3000000000 + 0xe3000000000
0x26adbcb00000000 + 0x26adbcb00000000
0x1ff3000000000 + 0x1ff3000000000
0xd188083200000e07 + 0xd188083200000e07

total: 0x1a7e9e2ba00001c0eL (30546195236108901390L)

Pretty weird, looks like merge pool indeed puts some incorrect values into higher bits.

------------------------------------

And one more run with %p:

0xe07 + 0xe07
0xcb + 0xcb
0x249 + 0x249
0x272 + 0x272
0x2f + 0x2f
0x4d + 0x4d
0x19 + 0x19
0x303 + 0x303
0x226c + 0x226c
0xba7 + 0xba7
0x78b + 0x78b
0x112 + 0x112
0xf6 + 0xf6
0x13 + 0x13
0xbf + 0xbf
0x140 + 0x140
0x7f + 0x7f
0x12e + 0x12e
0x70 + 0x70
0x69 + 0x69
0x2a + 0x2a
0x134 + 0x134
0xf5 + 0xf5
0x102 + 0x102
0xd1 + 0xd1
0x362 + 0x362
0x1e1 + 0x1e1
0x1f + 0x1f

total: 0xb9d6 (47574)


----------------------------------

I like that in %m case, there is always a dump with count value ending with "e07", which seems to be coming from the first tests process. After that, merge pool is definitely doing something weird. We'll investigate more tomorrow.

Not much of a news, but I can confirm that nothing wrong happens before coverage data get written on the disk. I've dumped counters for pthread_equal from every single process, and all of them look good: there are no overflows and the total value matches 47574 obtained with %p profile file.

I have some ideas on what's corrupting the data on later stages, continue investigating.

I know where incorrect numbers come from. It's because of that line: https://github.com/llvm-mirror/compiler-rt/blame/d89810afdc2862ea865f5c297ef4ea5839941d25/lib/profile/InstrProfilingMerge.c#L112

It differs from how pointers to the Counters are calculated in all other places.

It should be something like:

  SrcCounters = (uint64_t*)SrcData->CounterPtr;

However, that doesn't seem to be the right fix. It may be that other places are wrong, or I'm missing something else. I think I'll figure that out soon.

First of all, I've been wrong in c#11, the existing code does the right thing, as it adjust pointers when profile data is read from the disk. The code I suggested works fine only in-memory, before the dump has been written on the disk.

For my future reference, counters from different processes:

In-Memory Counter #0 holding Value: 246
In-Memory Counter #0 holding Value: 246
In-Memory Counter #0 holding Value: 2983
In-Memory Counter #0 holding Value: 2983
In-Memory Counter #0 holding Value: 585
In-Memory Counter #0 holding Value: 585
In-Memory Counter #0 holding Value: 191
In-Memory Counter #0 holding Value: 77
In-Memory Counter #0 holding Value: 191
In-Memory Counter #0 holding Value: 77
In-Memory Counter #0 holding Value: 1931
In-Memory Counter #0 holding Value: 771
In-Memory Counter #0 holding Value: 203
In-Memory Counter #0 holding Value: 771
In-Memory Counter #0 holding Value: 1931
In-Memory Counter #0 holding Value: 203
In-Memory Counter #0 holding Value: 274
In-Memory Counter #0 holding Value: 47
In-Memory Counter #0 holding Value: 626
In-Memory Counter #0 holding Value: 47
In-Memory Counter #0 holding Value: 274
In-Memory Counter #0 holding Value: 626
In-Memory Counter #0 holding Value: 8812
In-Memory Counter #0 holding Value: 8812
In-Memory Counter #0 holding Value: 112
In-Memory Counter #0 holding Value: 112
In-Memory Counter #0 holding Value: 320
In-Memory Counter #0 holding Value: 320
In-Memory Counter #0 holding Value: 127
In-Memory Counter #0 holding Value: 127
In-Memory Counter #0 holding Value: 42
In-Memory Counter #0 holding Value: 302
In-Memory Counter #0 holding Value: 42
In-Memory Counter #0 holding Value: 302
In-Memory Counter #0 holding Value: 105
In-Memory Counter #0 holding Value: 105
In-Memory Counter #0 holding Value: 19
In-Memory Counter #0 holding Value: 19
In-Memory Counter #0 holding Value: 258
In-Memory Counter #0 holding Value: 481
In-Memory Counter #0 holding Value: 258
In-Memory Counter #0 holding Value: 481
In-Memory Counter #0 holding Value: 245
In-Memory Counter #0 holding Value: 245
In-Memory Counter #0 holding Value: 209
In-Memory Counter #0 holding Value: 209
In-Memory Counter #0 holding Value: 308
In-Memory Counter #0 holding Value: 31
In-Memory Counter #0 holding Value: 308
In-Memory Counter #0 holding Value: 31
In-Memory Counter #0 holding Value: 866
In-Memory Counter #0 holding Value: 866
In-Memory Counter #0 holding Value: 25
In-Memory Counter #0 holding Value: 25
In-Memory Counter #0 holding Value: 3591
In-Memory Counter #0 holding Value: 3591

These values result in a "correct" number 47574 when merged altogether:

$ third_party/llvm-build/Release+Asserts/bin/llvm-profdata_orig show -counts -function=pthread_equal tp/merged.profdata
Counters:
  pthread_equal:
    Hash: 0x000000000000061f
    Counters: 1
    Function count: 47574
    Block counts: []
Instrumentation level: Front-end
Functions shown: 1
Total functions: 96814
Maximum function count: 3631555577
Maximum internal block count: 3168401839


As you can see, each value is duplicated, as we have two profile data blocks for pthread_equal function in each *.profraw dump, e.g.:

$ third_party/llvm-build/Release+Asserts/bin/llvm-profdata_orig show -counts -function=pthread_equal tp/viz.6588.profraw
Counters:
  pthread_equal:
    Hash: 0x000000000000061f
    Counters: 1
    Function count: 31
    Block counts: []
  pthread_equal:
    Hash: 0x000000000000061f
    Counters: 1
    Function count: 31
    Block counts: []
Instrumentation level: Front-end
Functions shown: 2
Total functions: 355511
Maximum function count: 1148749
Maximum internal block count: 312905


That doesn't feel right, as during merge via %Nm, those counters are duplicated multiple times. For example, let's say we have 4 dumps with the following counter values:

0 + 0
3 + 3
8 + 8
4 + 4

merge of %p dumps would give us: 30  (0 + 0 + 3 + 3 + 8 + 8 + 4 + 4)

merge of %1m would give us: 64, see the steps below:

>>> 0 + 0 # Counters in memory, no file is written yet, simply dump it on the disk.
0
>>> 3 + 3 + 0 + 0 # Another process has 3 + 3 in memory, plus 0 + 0 on the disk. Read file from the disk and merge into memory, then overwrite the file.
6
>>> 8 + 8 + 6 + 6 # Another process has 8 + 8 in memory, plus 6 + 6 on the disk. Read file from the disk and merge into memory, then overwrite the file. See the problem? 3 + 3 has been already counted twice!
28
>>> 4 + 4 + 28 + 28 # Same as before, 4 + 4 in memory and 28 + 28 from the disk. Now we have 3 + 3 counted 4 times and 8 + 8 counted twice. Those extra 18 + 16 result in delta of 34, which is 64 - 30.
64


So, it seems that there are no races or other mistakes in the merge process, the root cause is duplicated profile data for pthread_equal function.

I have one more idea to try out, and if it doesn't work I'll file a bug at LLVM and discuss it with other developers.

Oh, interesting, there was something similar ~2.5y ago: https://bugs.llvm.org/show_bug.cgi?id=23499

Reading.

Filed https://bugs.llvm.org/show_bug.cgi?id=36805, as it seems to be a linker issue rather than coverage tooling.

that's really bad news :(

To clarify, if it's a linker issue, then we should still be able to repro with the following set up:

use %9m and set --test-launcher-job=9

Is that correct?

I've ran viz_unittests with the above mentioned set up 80 times, and they all generate the same sha1sum, how should we explain that?

The problem is that the result is not correct even when we use %p. It simply doesn't overflow because values are small, and it gives always the same result as merge process is linear (each dump is added once) and predictable.

In %Nm case, each of the dumps can be merged multiple times. It's not an issue itself, but with duplicated counters, some values are multiplied by 2 ** P, where P is in somewhere between 1 and (number_of_total_processes - N).

It's possible that your combination of options affects reproducibility of the merge process, i.e. the dumps from different processes are merged together in the same order. However, I suspect that you're running on Mac, where that linker issue isn't present, it's a Linux-only problem.

Could you please run the following command on any single profraw dump:

$ llvm-profdata show -counts -function=pthread_equal tp/viz.6588.profraw

1. I was running on Linux.

2. I just verified the order of the test launcher jobs, they are different between two different runs, so I would say the merge order is not fixed.

3. I ran the command you provided (two different types of profraws were generated in a single run):

liaoyuke@liaoyuke2 ~/chromium/src (master) $ ls out/viz_report_14/
component_view_index.html                     viz_unittests.15835931327307552609_6.profraw
coverage/                                     viz_unittests.15835931327307552609_7.profraw
coverage.profdata                             viz_unittests.15835931327307552609_8.profraw
directory_view_index.html                     viz_unittests.6667102620810597882_0.profraw
file_view_index.html                          viz_unittests.6667102620810597882_1.profraw
index.html                                    viz_unittests.6667102620810597882_2.profraw
style.css                                     viz_unittests.6667102620810597882_3.profraw
viz_unittests.15835931327307552609_0.profraw  viz_unittests.6667102620810597882_4.profraw
viz_unittests.15835931327307552609_1.profraw  viz_unittests.6667102620810597882_5.profraw
viz_unittests.15835931327307552609_2.profraw  viz_unittests.6667102620810597882_6.profraw
viz_unittests.15835931327307552609_3.profraw  viz_unittests.6667102620810597882_7.profraw
viz_unittests.15835931327307552609_4.profraw  viz_unittests.6667102620810597882_8.profraw
viz_unittests.15835931327307552609_5.profraw  viz_unittests_output.txt
liaoyuke@liaoyuke2 ~/chromium/src (master) $ ls out/viz_report_14/^C
liaoyuke@liaoyuke2 ~/chromium/src (master) $ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -function=pthread_equal out/viz_report_14/viz_unittests.15835931327307552609_0.profraw 
Instrumentation level: Front-end
Functions shown: 0
Total functions: 9930
Maximum function count: 347686854
Maximum internal block count: 239828385
liaoyuke@liaoyuke2 ~/chromium/src (master) $ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -function=pthread_equal out/viz_report_14/viz_unittests.6667102620810597882_8.profraw 
Counters:
  pthread_equal:
    Hash: 0x000000000000061f
    Counters: 1
    Function count: 1185138347527
    Block counts: []
  pthread_equal:
    Hash: 0x000000000000061f
    Counters: 1
    Function count: 1185138347527
    Block counts: []
Instrumentation level: Front-end
Functions shown: 2
Total functions: 345518
Maximum function count: 1185138347527
Maximum internal block count: 262980668
liaoyuke@liaoyuke2 ~/chromium/src (master) $

Well, there is also a chance that  issue 801387  is not necessary caused by Counter Overflow warnings :( Remember, we've noticed there was a correlation on my desktop, but it still can be a different problem...

However, I guess you don't get Counter Overflow warning when using %9m? If you don't, it still doesn't break our hypothesis that Counter Overflow warnings may lead to results mismatch.

One more thing I'd like to ask you to try.

1) use *.profraw files from two different runs (or more than two)

2) merge them into corresponding *.profdata files

3) run the following commands on each of the *.profdata files:

$ llvm-profdata show -counts -function=pthread_equals tp/merged.profdata
Instrumentation level: Front-end
Functions shown: 0
Total functions: 96814  # <---- I'd expect this to be different across different runs
Maximum function count: 3631555577
Maximum internal block count: 3168401839


and


$ llvm-profdata show -counts -function=xmlIsMainThread tp/merged.profdata
Counters:
  xmlIsMainThread:
    Hash: 0x029f49129f498458
    Counters: 3
    Function count: 3591  # <---- I'd expect this to be the same across different runs
    Block counts: [1, 0]
Instrumentation level: Front-end
Functions shown: 1
Total functions: 96814
Maximum function count: 3631555577
Maximum internal block count: 3168401839


Thanks!

Yes, I didn't see any Counter Overflow warning when using %9m.

liaoyuke@liaoyuke2 ~/chromium/src (master) $ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -function=pthread_equals out/viz_report_1/coverage.profdata 
Instrumentation level: Front-end
Functions shown: 0
Total functions: 96685
Maximum function count: 205953080334
Maximum internal block count: 3168401839
liaoyuke@liaoyuke2 ~/chromium/src (master) $ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -function=pthread_equals out/viz_report_2/coverage.profdata 
Instrumentation level: Front-end
Functions shown: 0
Total functions: 96684
Maximum function count: 82380354702
Maximum internal block count: 3168401839
liaoyuke@liaoyuke2 ~/chromium/src (master) $ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -function=pthread_equals out/viz_report_3/coverage.profdata 
Instrumentation level: Front-end
Functions shown: 0
Total functions: 96685
Maximum function count: 168591976718
Maximum internal block count: 3168401839

liaoyuke@liaoyuke2 ~/chromium/src (master) $ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -function=xmlIsMainThread out/viz_report_1/coverage.profdata 
Counters:
  xmlIsMainThread:
    Hash: 0x029f49129f498458
    Counters: 3
    Function count: 3591
    Block counts: [1, 0]
Instrumentation level: Front-end
Functions shown: 1
Total functions: 96685
Maximum function count: 205953080334
Maximum internal block count: 3168401839
liaoyuke@liaoyuke2 ~/chromium/src (master) $ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -function=xmlIsMainThread out/viz_report_2/coverage.profdata 
Counters:
  xmlIsMainThread:
    Hash: 0x029f49129f498458
    Counters: 3
    Function count: 3591
    Block counts: [1, 0]
Instrumentation level: Front-end
Functions shown: 1
Total functions: 96684
Maximum function count: 82380354702
Maximum internal block count: 3168401839
liaoyuke@liaoyuke2 ~/chromium/src (master) $ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -function=xmlIsMainThread out/viz_report_3/coverage.profdata 
Counters:
  xmlIsMainThread:
    Hash: 0x029f49129f498458
    Counters: 3
    Function count: 3591
    Block counts: [1, 0]
Instrumentation level: Front-end
Functions shown: 1
Total functions: 96685
Maximum function count: 168591976718
Maximum internal block count: 3168401839

Looks like the "Total functions" match across different runs with "%9m and --test-launcher-jobs=9"

Oops, my bad, I meant pthread_equal, not pthread_equals. We need to take a look at function count for that particular function, not the total function. I’m on a phone now, will post my output later from PC.

liaoyuke@liaoyuke2 ~/chromium/src (master) $ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -function=pthread_equal out/viz_report_1/coverage.profdata 
Counters:
  pthread_equal:
    Hash: 0x000000000000061f
    Counters: 1
    Function count: 205953080334
    Block counts: []
Instrumentation level: Front-end
Functions shown: 1
Total functions: 96685
Maximum function count: 205953080334
Maximum internal block count: 3168401839
liaoyuke@liaoyuke2 ~/chromium/src (master) $ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -function=pthread_equal out/viz_report_2/coverage.profdata 
Counters:
  pthread_equal:
    Hash: 0x000000000000061f
    Counters: 1
    Function count: 82380354702
    Block counts: []
Instrumentation level: Front-end
Functions shown: 1
Total functions: 96684
Maximum function count: 82380354702
Maximum internal block count: 3168401839
liaoyuke@liaoyuke2 ~/chromium/src (master) $ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -function=pthread_equal out/viz_report_3/coverage.profdata 
Counters:
  pthread_equal:
    Hash: 0x000000000000061f
    Counters: 1
    Function count: 168591976718
    Block counts: []
Instrumentation level: Front-end
Functions shown: 1
Total functions: 96685
Maximum function count: 168591976718
Maximum internal block count: 3168401839

Yeah, you're right, the function count are all different.

Thanks for checking, Yuke!

Here are my numbers as well:

%p

$ llvm-profdata show -counts -function=pthread_equal tp/merged.profdata 
Counters:
  pthread_equal:
    Hash: 0x000000000000061f
    Counters: 1
    Function count: 47574
    Block counts: []
Instrumentation level: Front-end
Functions shown: 1
Total functions: 96814
Maximum function count: 3631555577
Maximum internal block count: 3168401839


%4m:

$ llvm-profdata show -counts -function=pthread_equal tm/merged.profdata 
Counters:
  pthread_equal:
    Hash: 0x000000000000061f
    Counters: 1
    Function count: 12645019374583808
    Block counts: []
Instrumentation level: Front-end
Functions shown: 1
Total functions: 69053
Maximum function count: 12645019374583808
Maximum internal block count: 1560987736

Vedant from Apple (an author of llvm-cov) asked if we have a small reproducer. Not sure how we can create it, probably have to see which part of viz_unittests is calling pthread_equal and how it is linked.

While trying to create a reproducer, I realized that we may have that issue due to some build weirdness we're doing. After taking a deeper look into where that extra pthread_equal comes from, it appears to be libxml.

A little hacky patch below helped me to avoid that trouble for viz_unittests, so now I'm curious how many things may break if I land it. Will try with unit_tests and then hopefully upload for a review.

$ third_party/llvm-build/Release+Asserts/bin/llvm-profdata show -counts -function=pthread_equal tm/merged.profdata
Counters:
  pthread_equal:
    Hash: 0x000000000000061f
    Counters: 1
    Function count: 25931
    Block counts: []
Instrumentation level: Front-end
Functions shown: 1
Total functions: 97165
Maximum function count: 3963738311
Maximum internal block count: 3459445312




patch:

$ git diff
diff --git a/third_party/libxml/BUILD.gn b/third_party/libxml/BUILD.gn
index 80b82fb9859e..efbc07075e8d 100644
--- a/third_party/libxml/BUILD.gn
+++ b/third_party/libxml/BUILD.gn
@@ -47,7 +47,7 @@ config("libxml_warnings") {
 
       # threads.c attempts to forward declare a pthread_equal which doesn't
       # match the prototype in pthreads.h
-      "-Wno-ignored-attributes",
+      # "-Wno-ignored-attributes",
 
       # libxml casts from int to long to void*.
       "-Wno-int-to-void-pointer-cast",
diff --git a/third_party/libxml/src/config.h.in b/third_party/libxml/src/config.h.in
index a751769cdba8..4dcca9f89e08 100644
--- a/third_party/libxml/src/config.h.in
+++ b/third_party/libxml/src/config.h.in
@@ -114,7 +114,7 @@
 #undef HAVE_PRINTF
 
 /* Define if <pthread.h> is there */
-#undef HAVE_PTHREAD_H
+#define HAVE_PTHREAD_H
 
 /* Define to 1 if you have the `putenv' function. */
 #undef HAVE_PUTENV
diff --git a/third_party/libxml/src/threads.c b/third_party/libxml/src/threads.c
index 0433ac0f5919..50bd0643b540 100644
--- a/third_party/libxml/src/threads.c
+++ b/third_party/libxml/src/threads.c
@@ -66,7 +66,7 @@ static int libxml_is_threaded = -1;
 #pragma weak pthread_cond_init
 #pragma weak pthread_cond_destroy
 #pragma weak pthread_cond_wait
-#pragma weak pthread_equal
+// #pragma weak pthread_equal
 #pragma weak pthread_self
 #pragma weak pthread_key_create
 #pragma weak pthread_key_delete

Hmm, looks promising. Tested locally on unit_tests and and content_unittests.

Runs with my patch didn't have have Counter Overflow warnings and produced a higher result compared to current master branch. Also, the result seems to be much more stable and reproducible:

# current master, content_unittests with -f content, 1st run:
content/  55.47% (260291/469254)  55.06% (22667/41168)  58.41% (470625/805736)
# 2nd run:
content/  54.22% (254428/469254)  54.09% (22266/41168)  56.48% (455043/805736)

# patched version, content_unittests -f content, 1st run:
content/  65.39% (306855/469254)  61.83% (25454/41168)  73.73% (594095/805736)
# 2nd run:
content/  65.39% (306855/469254)  61.83% (25454/41168)  73.73% (594095/805736)


# current master, unit_tests with -f components, 1st run:
components/  22.96% (73688/320949)  29.56% (9036/30568)  16.99% (46854/275842)
# 2nd run:
components/  20.19% (64809/320949)  26.72% (8168/30568)  15.90% (43855/275842)

# patched version, unit_tests with -f components, 1st run:
components/  26.26% (84292/320949)  32.63% (9975/30568)  22.50% (62077/275842)
# 2nd run:
components/  26.26% (84288/320949)  32.63% (9975/30568)  22.50% (62076/275842)

Looks like someone tried fixing that in libxml upstream 3.5 years ago, but not succeeded: http://lists.busybox.net/pipermail/buildroot/2014-September/105880.html

I'll upload a patch for the configuration used in Chromium.

I don't see Counter Overflow messages anymore since pthread got disabled in libxml. Coverage numbers are stable and pretty good now. Woohoo!

TIL: whenever we notice Counter Overflow again, we should fix it ASAP.