New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 866546 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Jul 30
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug
Build-Toolchain

Blocking:
issue 852111



Sign in to add a comment

Deadlock in libFuzzer

Project Member Reported by metzman@chromium.org, Jul 23

Issue description

David noticed a deadlock in libFuzzer in Chrome OS. The deadlock seems to have been fixed in upstream. 
If this is the case, then the compiler-rt version should be rolled or the fixes backported. Otherwise this deadlock should be fixed.
Attatched is the file that causes the deadlock on virglrenderer_fuzzer. To repro you need to run the fuzzer in a loop with this input, since it does not deterministically reproduce.
 
oom-e69d0dfc056e4430608df7481580db63020312fb
25 bytes View Download
Adding more details, the input causes an out of memory failure and another process is spawned to trace and perform leak detection on the first.  The second process deadlocks on a mutex.

First process stdout (can't be traced because second one is tracing it):
INFO: Seed: 4049999721
INFO: Loaded 2 modules   (34984 inline 8-bit counters): 34961 [0x7fd5141669f8, 0x7fd51416f289), 23 [0x558095e92526, 0x558095e9253d),
INFO: Loaded 2 PC tables (34984 PCs): 34961 [0x7fd51416f290,0x7fd5141f7ba0), 23 [0x558095e92540,0x558095e926b0),
INFO: 18 Clang Coverage Counters
/usr/libexec/fuzzers/virgl_fuzzer: Running 1 inputs 1 time(s) each.
Running: oom-e69d0dfc056e4430608df7481580db63020312fb
gl_version 30 - es profile enabled
WARNING: running without ARB/KHR robustness in place may crash
vrend_renderer.c:2518:17: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior vrend_renderer.c:2518:17 in
==163146== ERROR: libFuzzer: out-of-memory (malloc(16777216036))
   To change the out-of-memory limit use -rss_limit_mb=<N>

    #0 0x558095e1f1d7 in __sanitizer_print_stack_trace (/usr/libexec/fuzzers/virgl_fuzzer+0xfd1d7)
    #1 0x558095d4cca8 in _init (/usr/libexec/fuzzers/virgl_fuzzer+0x2aca8)
    #2 0x558095d4cbaa in _init (/usr/libexec/fuzzers/virgl_fuzzer+0x2abaa)
    #3 0x558095e25607  (/usr/libexec/fuzzers/virgl_fuzzer+0x103607)
    #4 0x558095d718cd  (/usr/libexec/fuzzers/virgl_fuzzer+0x4f8cd)
    #5 0x558095d71afa  (/usr/libexec/fuzzers/virgl_fuzzer+0x4fafa)
    #6 0x558095e1680c in calloc (/usr/libexec/fuzzers/virgl_fuzzer+0xf480c)
    #7 0x7fd513ecde89 in vrend_create_shader /build/amd64-generic/tmp/portage/media-libs/virglrenderer-0.6.0_p20180716-r2/work/virglrenderer-0fb73b11e4cdadced885e52848002b2e9c79e3f5/src/vrend_renderer.c:2592:16
    #8 0x7fd513fd0e87 in vrend_decode_create_shader /build/amd64-generic/tmp/portage/media-libs/virglrenderer-0.6.0_p20180716-r2/work/virglrenderer-0fb73b11e4cdadced885e52848002b2e9c79e3f5/src/vrend_decode.c:110:10
    #9 0x7fd513fd0e87 in vrend_decode_create_object /build/amd64-generic/tmp/portage/media-libs/virglrenderer-0.6.0_p20180716-r2/work/virglrenderer-0fb73b11e4cdadced885e52848002b2e9c79e3f5/src/vrend_decode.c:698
    #10 0x7fd513fd0e87 in vrend_decode_block /build/amd64-generic/tmp/portage/media-libs/virglrenderer-0.6.0_p20180716-r2/work/virglrenderer-0fb73b11e4cdadced885e52848002b2e9c79e3f5/src/vrend_decode.c:1210
    #11 0x558095e43938 in LLVMFuzzerTestOneInput /build/amd64-generic/tmp/portage/media-libs/virglrenderer-0.6.0_p20180716-r2/work/virglrenderer-0fb73b11e4cdadced885e52848002b2e9c79e3f5/tests/fuzzer/virgl_fuzzer.c:181:4
    #12 0x558095d4ecfc in _init (/usr/libexec/fuzzers/virgl_fuzzer+0x2ccfc)
    #13 0x558095d3f686 in _init (/usr/libexec/fuzzers/virgl_fuzzer+0x1d686)
    #14 0x558095d4552c in _init (/usr/libexec/fuzzers/virgl_fuzzer+0x2352c)
    #15 0x558095d6fe32  (/usr/libexec/fuzzers/virgl_fuzzer+0x4de32)
==163146== ERROR: libFuzzer: out-of-memory (used: 2137Mb; limit: 2048Mb)
   To change the out-of-memory limit use -rss_limit_mb=<N>

Live Heap Allocations: 16814868403 bytes in 4512 chunks; quarantined: 114268966 bytes in 12579 chunks; 8095 other chunks; total chunks: 25186; showing top 95% (at most 8 unique contexts)
16777216036 byte(s) (99%) in 1 allocation(s)

Second process that is deadlocked:
(gdb) where
#0  0x0000558095e2caa7 in __sanitizer::BlockingMutex::Lock() ()
#1  0x0000558095e37f02 in __sanitizer::Symbolizer::SymbolizePC(unsigned long) ()
#2  0x0000558095e36ba9 in __sanitizer::StackTrace::Print() const ()
#3  0x0000558095e177d1 in __asan::HeapProfile::Print(unsigned long, unsigned long) ()
#4  0x0000558095e17688 in __asan::MemoryProfileCB(__sanitizer::SuspendedThreadsList const&, void*) ()
#5  0x0000558095e34b44 in __sanitizer::TracerThread(void*) ()
#6  0x0000558095e2d422 in __sanitizer::internal_clone(int (*)(void*), void*, int, void*, int*, void*, int*) ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

I find that I can only reproduce by creating additional CPU load (I spawned another instance of the fuzzer with 50 workers) with the other fuzzer in a loop.
Labels: OS-Chrome
Owner: yunlian@chromium.org
Yunlian,
Cqan you cherry-pick r331825 to llvm and compiler-rt?
This looks like a UBSan+OOM variant of https://github.com/google/sanitizers/issues/788.  I the cherry-pick doesn't fix this, we probably need to fix this in UBSan with something similar to https://reviews.llvm.org/D46277.
Blocking: 852111
Status: Fixed (was: Untriaged)
This should have been fixed by the cherry-pick + llvm update in https://chromium-review.googlesource.com/1147840. Please re-open if that is not the case.
Looks fixed from my tests.

Sign in to add a comment