Detailed report: https://cluster-fuzz.appspot.com/testcase?key=5952584351481856

Fuzzer: libfuzzer_net_url_request_fuzzer
Job Type: libfuzzer_chrome_msan
Platform Id: linux

Crash Type: Out-of-memory
Crash Address: 
Crash State:
  net_url_request_fuzzer
  
Regressed: https://cluster-fuzz.appspot.com/revisions?job=libfuzzer_chrome_msan&range=395689:395794

Minimized Testcase (0.47 Kb): https://cluster-fuzz.appspot.com/download/AMIfv96UweE_mBAGH19JtZ_Nxzrf2nit_m704taogjlcgsp8FTUvyuFwekHz1GPdg1bZ_hxAXCRixKD9oUUnRTZJiMP97OYvocmrTjwvIpKj3hCC_DQdr7pF2f9djTt6X_zsnkLNfBE5lXLprTyfW7rmBdY3IDuFEQ?testcase_id=5952584351481856

Filer: mmohammad

See https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/reproducing.md for more information.

ClusterFuzz has detected this issue as fixed in range 413228:413328.

Detailed report: https://cluster-fuzz.appspot.com/testcase?key=4601109104820224

Fuzzer: libfuzzer_net_url_request_fuzzer
Job Type: libfuzzer_chrome_msan
Platform Id: linux

Crash Type: Out-of-memory
Crash Address: 
Crash State:
  net_url_request_fuzzer
  
Regressed: https://cluster-fuzz.appspot.com/revisions?job=libfuzzer_chrome_msan&range=395689:395794
Fixed: https://cluster-fuzz.appspot.com/revisions?job=libfuzzer_chrome_msan&range=413228:413328

Minimized Testcase (0.47 Kb): https://cluster-fuzz.appspot.com/download/AMIfv97rbiYfd8XGRLoj6XeijqxC1hgp0bZlOnWetEae_4bhdGyDWCB4cUqFqpv_bKT93AE-o77TvKZurBvjLAT3QOYB7jXZ65Zi6sCmCdjqDMGhHyGpp-a7eN-amyIdTzMevbBJ2_7xM84u4VDXOOrHEOQbHXV6SA?testcase_id=4601109104820224

See https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/reproducing.md for more information.

If you suspect that the result above is incorrect, try re-doing that job on the test case report page.

ClusterFuzz has detected this issue as fixed in range 413228:413328.

Detailed report: https://cluster-fuzz.appspot.com/testcase?key=5952584351481856

Fuzzer: libfuzzer_net_url_request_fuzzer
Job Type: libfuzzer_chrome_msan
Platform Id: linux

Crash Type: Out-of-memory
Crash Address: 
Crash State:
  net_url_request_fuzzer
  
Regressed: https://cluster-fuzz.appspot.com/revisions?job=libfuzzer_chrome_msan&range=395689:395794
Fixed: https://cluster-fuzz.appspot.com/revisions?job=libfuzzer_chrome_msan&range=413228:413328

Minimized Testcase (0.47 Kb): https://cluster-fuzz.appspot.com/download/AMIfv96UweE_mBAGH19JtZ_Nxzrf2nit_m704taogjlcgsp8FTUvyuFwekHz1GPdg1bZ_hxAXCRixKD9oUUnRTZJiMP97OYvocmrTjwvIpKj3hCC_DQdr7pF2f9djTt6X_zsnkLNfBE5lXLprTyfW7rmBdY3IDuFEQ?testcase_id=5952584351481856

See https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/reproducing.md for more information.

If you suspect that the result above is incorrect, try re-doing that job on the test case report page.

ClusterFuzz testcase is verified as fixed, closing issue.

If this is incorrect, please add ClusterFuzz-Wrong label and re-open the issue.

Detailed report: https://cluster-fuzz.appspot.com/testcase?key=5956357377490944

Fuzzer: libfuzzer_net_url_request_fuzzer
Job Type: libfuzzer_chrome_msan
Platform Id: linux

Crash Type: Out-of-memory
Crash Address: 
Crash State:
  net_url_request_fuzzer
  
Regressed: https://cluster-fuzz.appspot.com/revisions?job=libfuzzer_chrome_msan&range=413683:413827

Minimized Testcase (0.31 Kb):
Download: https://cluster-fuzz.appspot.com/download/AMIfv95n6t96EoZxicyTaFyVZyBCmPHP3VBxM33upDES0DSPPemTPe0mS8yYF-uS5o_q0QwNIa7VhUqBpgmi40SFqhI8q-zYl1Pk8Qsp3NMPL8I6TdQ8mGFfC7FZCNNaOSD-MO4LMktqKABRf6Gx4ZLJkQDTp44c9g?testcase_id=5956357377490944
5HHTTPHig(PTHTh.2.:1 Hdiges/2}H:T
Content-Encoding: deflate
Content-Encoding: deflate
Content-Encoding:bR

� �0J(�p���tR'filelo�P=926
LHTocqtioWNYoWAWntentEnoHTTP/1.1 100 404! wHTT39
C307 * �onteO
BK
int-ndoncEg:bR

� 1�JT�-(file`.*!in: gzapAutnetworkNdicate:n)0�on�htTP 616
Location0 pa;8kt C:/
Expires:[)��-jA


Issue manually filed by: mmohammad

See https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/reproducing.md for more information.

Re-Opening the issue as Clusterfuzz has detected the crash again, Clusterfuzz update in the above comment.Thank you 

Removing EditIssue view restrictions from ClusterFuzz filed bugs. If you believe that this issue should still be restricted, please reapply the label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Assigning to concern owner from CL,
https://chromium.googlesource.com/chromium/src/+log/dd6d1b62de7b449d997d159d63207abe87d3cb46..6da2b3835157a2178a5951ff42f2e1a7d3404608?pretty=fuller

Suspecting Commit#
https://chromium.googlesource.com/chromium/src/+/37ef985f4359f4b5b4b26489b5880d0025a61772'

@csharrisom -- Could you please look into the issue, kindly re-assign if this is not related to your changes.
Thank You.

The fuzzer report has no stack trace. Would current net triager be willing to repro?

The report has no stack trace because it is Out-of-Memory crash, i.e. the fuzzer doesn't crash itself, it is being intentionally killed once rss_limt_mb is exceeded.

csharrison@, let me assign this back to you, since it should be reproducible. I've tried to reproduce it locally and got another crash (issue 674060), so we are blocked on that for now.

Oops, forgot to assign. Please see my previous comment.

I understand the fuzzer ran out of memory, it would just be nice to know which allocation triggered the OOM in the clusterfuzz report.

I'm afraid that in most of the cases OOMs are caused by many different allocations, so you cannot say which allocation triggered the OOM. LibFuzzer doesn't track every allocation, it just calls getrusage (https://cs.chromium.org/chromium/src/third_party/libFuzzer/src/FuzzerUtil.cpp?q=GetPeakRSSMb+file:%5Esrc/third_party/libFuzzer/src/&sq=package:chromium&l=279) once per second (https://cs.chromium.org/chromium/src/third_party/libFuzzer/src/FuzzerDriver.cpp?q=RssThread+file:%5Esrc/third_party/libFuzzer/src/&sq=package:chromium&l=253) and compares resource usage against the limit.

Ah ok great that clears things up.

libFuzzer does print the heap profile when fails with OOM -- but only in asan mode. We did not hook up the heap profiler with msan

If I run the test case in the asan+libfuzzer mode it passes consuming 1.2Gb RSS. 
If I the re-run it with -rss_limit_mb=1000 I get an OOM failure and a nice heap profile: 

./out/libfuzzer-asan/net_url_request_fuzzer  ~/Downloads/fuzz-3-net_url_request_fuzzer -rss_limit_mb=1000
==16118== ERROR: libFuzzer: out-of-memory (used: 1033Mb; limit: 1000Mb)
   To change the out-of-memory limit use -rss_limit_mb=<N>

Live Heap Allocations: 1620702513 bytes from 4734 allocations; showing top 95%
1616904192 byte(s) (99%) in 2 allocation(s)
    #0 0x4f840b in operator new(unsigned long) (/usr/local/google/home/kcc/chromium/src/out/libfuzzer-asan/net_url_request_fuzzer+0x4f840b)
    #1 0x7fa4de31a79b in __allocate buildtools/third_party/libc++/trunk/include/new:168:10
    #2 0x7fa4de31a79b in allocate buildtools/third_party/libc++/trunk/include/memory:1729
    #3 0x7fa4de31a79b in allocate buildtools/third_party/libc++/trunk/include/memory:1488
    #4 0x7fa4de31a79b in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__grow_by_and_replace(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, char const*) buildtools/third_party/libc++/trunk/include/string:2325
    #5 0x7fa4de319cf4 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::append(char const*, unsigned long) buildtools/third_party/libc++/trunk/include/string:2582:9
    #6 0x2db1071 in net::TestDelegate::OnReadCompleted(net::URLRequest*, int) net/url_request/url_request_test_util.cc:315:22
    #7 0x2db0653 in net::TestDelegate::OnResponseStarted(net::URLRequest*, int) net/url_request/url_request_test_util.cc:275:7
    #8 0x346fcaa in net::URLRequestJob::NotifyHeadersComplete() net/url_request/url_request_job.cc:516:13
    #9 0x37886cb in net::URLRequestHttpJob::NotifyHeadersComplete() net/url_request/url_request_http_job.cc:429:18
    #10 0x378df1e in net::URLRequestHttpJob::SaveCookiesAndNotifyHeadersComplete(int) net/url_request/url_request_http_job.cc:768:3
    #11 0x378c47e in net::URLRequestHttpJob::OnStartCompleted(int) net/url_request/url_request_http_job.cc:909:5
    #12 0x3165353 in Run base/callback.h:85:12
    #13 0x3165353 in net::HttpCache::Transaction::DoLoop(int) net/http/http_cache_transaction.cc:885
    #14 0x31a3e6f in Run base/callback.h:85:12
    #15 0x31a3e6f in DoCallback net/http/http_network_transaction.cc:659
    #16 0x31a3e6f in net::HttpNetworkTransaction::OnIOComplete(int) net/http/http_network_transaction.cc:665
    #17 0x35f3bd0 in Run base/callback.h:85:12
    #18 0x35f3bd0 in net::HttpStreamParser::OnIOComplete(int) net/http/http_stream_parser.cc:386
    #19 0x50be33 in Run base/callback.h:85:12
    #20 0x50be33 in net::FuzzedSocket::OnReadComplete(base::Callback<void (int), (base::internal::CopyMode)1, (base::internal::RepeatMode)1> const&, int) net/socket/fuzzed_socket.cc:256
    #21 0x2e95b39 in Run base/callback.h:68:12
    #22 0x2e95b39 in base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) base/debug/task_annotator.cc:52
    #23 0x2de3dce in base::MessageLoop::RunTask(base::PendingTask*) base/message_loop/message_loop.cc:413:19
    #24 0x2de5099 in base::MessageLoop::DeferOrRunPendingTask(base::PendingTask) base/message_loop/message_loop.cc:422:5
    #25 0x2de67ba in base::MessageLoop::DoWork() base/message_loop/message_loop.cc:515:13
    #26 0x2df6264 in base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) base/message_loop/message_pump_libevent.cc:218:31
    #27 0x2de2e89 in base::MessageLoop::RunHandler() base/message_loop/message_loop.cc:378:10
    #28 0x2e019f9 in base::RunLoop::Run() base/run_loop.cc:37:10
    #29 0x4fb72b in LLVMFuzzerTestOneInput net/url_request/url_request_fuzzer.cc:42:19

That sounds like a nice trick to be added to both Chromium and OSS-Fuzz
docs!

Thanks for the stack trace!

Hm, compression bomb maybe? Still, moving back to internals network queue for triage. My CL just fixed all net fuzzers (they were broken at the time with non compiling builds), so it not directly related to this OOM.

Just a note to explain why msan died with OOM and asan did not (with the default limit).
As we can see here the target allocates 1.6Gb of ram and probably does not
actually use all of it.
With asan there is ~ 1/8 RAM overhead for the shadow, so asan process 
fits into 2Gb limit.
msan with track origins has up to 3x RAM overhead
(1:1 memory shadow for uninit bits and 1:1 shadow for origins) and so msan
process consumes 4.5Gb on this input.

Maybe we should simply ignore OOMs from msan? 
OTOH this bugs seems to be proper. 

These all look like deflate/brotli bomb scenarios - at least the two test cases both have 3 content-encodings.

Seems like there are two potential fixes:
1)  Don't buffer entire response body into RAM, in the fuzzer.
2)  Fix Chrome:  Don't allow multiple gzip/brotli filters (And only allow one SDCH as well).  This approach seems better, as the issue has come up before, but we should probably gather metrics first.

Punting out of triage queue, though I may take it on during my (current) shift.

For more information, please see https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/reference.md.

The link referenced in the description is no longer valid.

(bulk edit)

ClusterFuzz has detected this issue as fixed in range 462409:462423.

Detailed report: https://clusterfuzz.com/testcase?key=5956357377490944

Fuzzer: libFuzzer_net_url_request_fuzzer
Job Type: libfuzzer_chrome_msan
Platform Id: linux

Crash Type: Out-of-memory (exceeds 2048 MB)
Crash Address: 
Crash State:
  net_url_request_fuzzer
  
Sanitizer: memory (MSAN)

Regressed: https://clusterfuzz.com/revisions?job=libfuzzer_chrome_msan&range=413683:413827
Fixed: https://clusterfuzz.com/revisions?job=libfuzzer_chrome_msan&range=462409:462423

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=5956357377490944

See https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/reference.md for more information.

If you suspect that the result above is incorrect, try re-doing that job on the test case report page.

The fuzzer is disabled, anyways.