New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 658578 link

Starred by 1 user
Status: Fixed
Owner:
kcc@chromium.org
Closed: Oct 2016
Cc:
mmoroz@chromium.org
och...@chromium.org
█ aizatsky@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocking:
issue 539572
issue 654868



Sign in to add a comment

pdf_jpx_fuzzer oom issue

Project Member Reported by infe...@chromium.org, Oct 23 2016 
See attached file. Unsure if these top allocations indicating some issue in libFuzzer.

4698787 byte(s) (31%) in 9995 allocation(s)
#0 0x4f210b in operator new(unsigned long) (/mnt/scratch0/clusterfuzz/slave-bot/builds/chromium-browser-libfuzzer_linux-release-asan_ae530a86793cd6b8b56ce9af9159ac101396e802/revisions/libfuzzer-linux-release-426991/pdf_jpx_fuzzer+0x4f210b)
#1 0x4f5cb7 in __allocate buildtools/third_party/libc++/trunk/include/new:168:10
#2 0x4f5cb7 in allocate buildtools/third_party/libc++/trunk/include/memory:1729
#3 0x4f5cb7 in allocate buildtools/third_party/libc++/trunk/include/memory:1488
#4 0x4f5cb7 in std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >::allocate(unsigned long) buildtools/third_party/libc++/trunk/include/vector:930
#5 0x50becd in std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >::vector(std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > const&) buildtools/third_party/libc++/trunk/include/vector:1204:9
#6 0x5158c8 in construct<std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > &> buildtools/third_party/libc++/trunk/include/memory:1740:31
#7 0x5158c8 in __construct<std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > &> buildtools/third_party/libc++/trunk/include/memory:1656
#8 0x5158c8 in construct<std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > &> buildtools/third_party/libc++/trunk/include/memory:1502
#9 0x5158c8 in __construct_range_forward<std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > *, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > *> buildtools/third_party/libc++/trunk/include/memory:1586
#10 0x5158c8 in _ZNSt3__16vectorINS0_IhNS_9allocatorIhEEEENS1_IS3_EEE18__construct_at_endIPS3_EENS_9enable_ifIXsr21__is_forward_iteratorIT_EE5valueEvE4typeES9_S9_m buildtools/third_party/libc++/trunk/include/vector:1022
#11 0x510df9 in fuzzer::Fuzzer::FindExtraUnits(std::__1::vector<std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >, std::__1::allocator<std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > > const&, std::__1::vector<std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >, std::__1::allocator<std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > > const&) third_party/libFuzzer/src/FuzzerLoop.cpp:593:20
#12 0x5119ee in fuzzer::Fuzzer::Merge(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) third_party/libFuzzer/src/FuzzerLoop.cpp:652:14
#13 0x4f9c5e in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) third_party/libFuzzer/src/FuzzerDriver.cpp:497:7
#14 0x515e48 in main third_party/libFuzzer/src/FuzzerMain.cpp:20:10
#15 0x7f38de81aec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287
pdf_jpx_fuzzer
15.5 KB View Download

Comment 1 by kcc@chromium.org, Oct 23 2016 

>> Unsure if these top allocations indicating some issue in libFuzzer.
Unlikely. 
It says: 4698787 byte(s) (31%) in 9995 allocation(s)
So, libFuzzer has allocated 5Mb to store the corpus files in memory, 
pretty reasonable. At the same time, the process used more than 3Gb and they did not show up in the profile. 

Most likely, this is caused by a single huge memory allocation
that was still happening while the process was killed by OOM and so it
did not show up in the profile. 
(This is a missing functionality in libFuzzer which I am aware of). 

Please remind me, where do I get the contents of  /mnt/scratch0/clusterfuzz/slave-bot/inputs/fuzzer-testcases-disk/pdf_jpx_fuzzer_initial_corpus ?

Comment 2 by aarya@google.com, Oct 24 2016 

Thanks Kostya. gs://clusterfuzz-corpus/libfuzzer/pdf_jpx_fuzzer is the location of corpus. Yes, if we can see that allocation stack of that huge memory allocation, then easy to assign bug to developer.

Comment 3 by kcc@chromium.org, Oct 25 2016

Cc: -kcc@chromium.org
Owner: kcc@chromium.org
Ok, I see the problem. 
It is the long-standing issue with asan allocator where in some cases
it accumulates too much RAM and does not release it back to the system. 
You've never seen it before on "big" cluster fuzz because all processes
there  are short lived. (or have you?)

I have been trying to address this in Q3 with partial success. 
If you try this: 
ASAN_OPTIONS=quarantine_size_mb=10:allocator_release_to_os=1 ./fuzzer ... 
the merge process for this target will succeed with peak RSS of 2130Mb. 
Since there is at least one input that requires 1.7Gb to be processed, 
this is pretty good. 

Please try using this flag for merges. 
If it helps -- great. If it does not -- let's look for other workarounds. 
I hope to finish this work in Q1 and make this behavior the default. 

Independently, I think we need to buganize the memory eaters in this target. 
E.g. 135c5a9d7648905e8b5f7f9045a736d2feadfe2e (306 bytes) from the corpus consumes 1.779 Gb at peak.

Comment 4 by aarya@google.com, Oct 25 2016

Status: Fixed (was: Untriaged)
Decreased per testcase rss limit for 2048 to 1024 mb (still think 1 gb is too high, will probably lower it more). Also added allocator_release_to_os flag in https://chromereviews.googleplex.com/532787013/

This fixes the merge, yay! stats updated on dashboard.
►

Sign in to add a comment