New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 802137 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux , Mac
Pri: 2
Type: Bug



Sign in to add a comment

Timeout in net_data_job_fuzzer

Project Member Reported by ClusterFuzz, Jan 16 2018

Issue description

Detailed report: https://clusterfuzz.com/testcase?key=6590649561513984

Fuzzer: libFuzzer_net_data_job_fuzzer
Job Type: libfuzzer_chrome_msan
Platform Id: linux

Crash Type: Timeout (exceeds 25 secs)
Crash Address: 
Crash State:
  net_data_job_fuzzer
  
Sanitizer: memory (MSAN)

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=6590649561513984

Issue filed automatically.

See https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/reference.md for more information.

Note: This crash might not be reproducible with the provided testcase. That said, for the past 14 days we've been seeing this crash frequently. If you are unable to reproduce this, please try a speculative fix based on the crash stacktrace in the report. The fix can be verified by looking at the crash statistics in the report, a day after the fix is deployed. We will auto-close the bug if the crash is not seen for 14 days.
 
Project Member

Comment 1 by ClusterFuzz, Jan 17 2018

Labels: OS-Mac
Cc: brajkumar@chromium.org
Components: Internals>Network
Labels: Test-Predator-Wrong CF-NeedsTriage
Unable to find actual suspect through code search and also no regressed revision range is seen in the detailed report, hence marking it as untriaged.


Thanks!

Comment 3 by mmenke@chromium.org, Jan 22 2018

These are probably because the ClusterFuzz started sending 1 MB+ strings to these fuzzers - probably shouldn't try to look for a regression range when the properties of the fuzzer have been changed.

Comment 4 by eroman@chromium.org, Jan 29 2018

Labels: -Pri-1 Pri-2
Status: Available (was: Untriaged)
Samples: 1M of event 'cycles', Event count (approx.): 1340814119637                                                                                 
  Children      Self  Command          Shared Object               Symbol                                                            
+   55.72%     0.00%  net_data_job_fu  net_data_job_ [.] fuzzer::FuzzerDriver                                                        
+   55.72%     0.00%  net_data_job_fu  net_data_job_ [.] main                                                                        
+   55.72%     0.00%  net_data_job_fu  libc-2.24.so  [.] __libc_start_main                                                           
+   55.72%     0.00%  net_data_job_fu  [unknown]     [k] 0xec9e258d4c544155                                                          
+   55.72%     0.00%  net_data_job_fu  net_data_job_ [.] fuzzer::RunOneTest                                                          
+   55.72%     0.00%  net_data_job_fu  net_data_job_ [.] fuzzer::Fuzzer::ExecuteCallback                                             
+   55.72%     0.00%  net_data_job_fu  net_data_job_ [.] LLVMFuzzerTestOneInput                                                      
+   55.72%     0.00%  net_data_job_fu  net_data_job_ [.] URLRequestDataJobFuzzerHarness::CreateAndReadFromDataURLRequest             
+   55.63%     0.00%  net_data_job_fu  libbase.so    [.] base::MessageLoop::Run                                                      
+   55.63%     0.00%  net_data_job_fu  libbase.so    [.] base::RunLoop::Run                                                          
+   55.60%     0.04%  net_data_job_fu  libbase.so    [.] base::MessagePumpLibevent::Run                                              
+   50.51%     0.03%  net_data_job_fu  libbase.so    [.] base::MessageLoop::DoWork                                                   
+   43.88%     0.00%  TaskSchedulerFo  libbase.so    [.] base::(anonymous namespace)::ThreadFunc                                     
+   43.88%     0.00%  TaskSchedulerFo  libpthread-2. [.] start_thread                                                                
+   43.75%     0.06%  TaskSchedulerFo  libbase.so    [.] base::internal::SchedulerWorker::Thread::ThreadMain                         
+   42.64%     0.01%  net_data_job_fu  libbase.so    [.] base::MessageLoop::DeferOrRunPendingTask                                    
+   42.58%     0.02%  net_data_job_fu  libbase.so    [.] base::MessageLoop::RunTask                                                  
+   41.86%     0.01%  net_data_job_fu  libbase.so    [.] base::internal::IncomingTaskQueue::RunTask                                  
+   41.17%     0.03%  net_data_job_fu  libbase.so    [.] base::debug::TaskAnnotator::RunTask                                         
+   40.96%     0.01%  net_data_job_fu  libbase.so    [.] base::OnceCallback<void ()>::Run                                            
+   39.92%     0.01%  net_data_job_fu  libbase.so    [.] base::internal::Invoker<base::internal::BindState<void (base::(anonymous nam
+   39.89%     0.00%  net_data_job_fu  libbase.so    [.] base::internal::Invoker<base::internal::BindState<void (base::(anonymous nam
+   39.87%     0.01%  net_data_job_fu  libbase.so    [.] base::internal::InvokeHelper<false, void>::MakeItSo<void (base::(anonymous n
+   39.85%     0.01%  net_data_job_fu  libbase.so    [.] base::internal::FunctorTraits<void (base::(anonymous namespace)::PostTaskAnd
+   39.83%     0.01%  net_data_job_fu  libbase.so    [.] base::(anonymous namespace)::PostTaskAndReplyRelay::RunReplyAndSelfDestruct 
+   33.94%     0.01%  net_data_job_fu  libnet.so     [.] base::internal::Invoker<base::internal::BindState<void (net::URLRequestJob::
+   33.90%     0.01%  net_data_job_fu  libnet.so     [.] base::internal::Invoker<base::internal::BindState<void (net::URLRequestJob::
+   33.85%     0.01%  net_data_job_fu  libnet.so     [.] base::internal::InvokeHelper<true, void>::MakeItSo<void (net::URLRequestJob:
+   33.83%    33.73%  net_data_job_fu  net_data_job_ [.] __sanitizer_cov_trace_pc_guard                                              
+   33.22%     0.01%  net_data_job_fu  libnet.so     [.] base::internal::FunctorTraits<void (net::URLRequestJob::*)(int), void>::Invo
+   32.19%     0.02%  net_data_job_fu  libnet.so     [.] net::URLRequestJob::ReadRawDataComplete                                     
+   31.45%     0.02%  net_data_job_fu  libnet.so     [.] base::RepeatingCallback<void (int)>::Run                                    
+   29.97%     0.01%  net_data_job_fu  libnet.so     [.] base::internal::Invoker<base::internal::BindState<void (net::URLRequestJob::
+   29.94%     0.01%  net_data_job_fu  libnet.so     [.] base::internal::Invoker<base::internal::BindState<void (net::URLRequestJob::
+   29.89%     0.01%  net_data_job_fu  libnet.so     [.] base::internal::InvokeHelper<true, void>::MakeItSo<void (net::URLRequestJob:

Comment 5 by mmenke@chromium.org, Feb 28 2018

Owner: mmenke@chromium.org
Status: Assigned (was: Available)
Cc: eroman@chromium.org csharrison@chromium.org
[+eroman]:  Plan to have you review this CL.
[+csharrison]:  Just in case you have any thoughts, since you wrote the fuzzer (Feel free to ignore this, however, without guilt)

The data job fuzzer is slow only because the data copy off thread, so it's all due to threading overhead and binding/callback overhead.

So, options:
1) We could do fewer reads (Once we've done more than, say, 1000 reads, we could increase the size of subsequent reads to a minimum of 32k or something).
2) We could skip the thread hop for URLRequestJobs than can be handled on-thread. With the advent of the network service, eventually we won't be reading any off thread data, anyways (Not even sure we'll still need to support data URLs, long term, so the point may be moot).
3) We could just limit input size.  Since we do need to handle long data URLs, I'm not a fan of this option.

Option 1) and 2) are both pretty easy to implement.  Option 2) potentially improves real world performance, at the cost of adding yet more funky cases to URLRequestSimpleJob that we may not care about long term, so I'm inclined to go with option 1).
Where do we pass data between threads? I thought this test would all run single-threaded (it's been a while...)

I wonder if we should just limit to url::kMaxUrlChars (2MB), which is the GURL IPC limit.
URLRequestSimpleJob copies data off-thread.  It's used for data URLs, chrome URLs, and other sorts of data.  Chrome URLs use memory-mapped files, and we don't want to block the IO thread on file IO.

This test is only reading about 1 MB of data, but 1 million PostTaskAndReplies in a memory instrumented build is slow.
One other alternative is to just abort the fuzzer if we calculate we'd make too many reads (over some large N). (1) is likely equally easy to do though, so might as well do that.
That means we'd be unlikely to ever fuzz reads over a very large response body, though.  While it's unlikely we're suddenly run into a bug reading the 1 millionth byte, anyways, I'd rather still test that case.

And as you say, (1) should be a pretty easy change, regardless.
Project Member

Comment 11 by bugdroid1@chromium.org, Mar 5 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fde0811e64ba588de994b029782c941dd32dd83c

commit fde0811e64ba588de994b029782c941dd32dd83c
Author: Matt Menke <mmenke@chromium.org>
Date: Mon Mar 05 20:44:37 2018

Limit the net_data_job_fuzzer to 20000 reads of fuzzed length.

The remaining reads will all be 32k.  This is because hundreds of
thousands of 1-byte reads cause the test to timeout, since each read
requests a pair of thread hops.  Once the network service ships, we
can consider removing the thread hops, if the network service is
still used for data URLs (The hops are needed for other types of
URLs that currently share code with data URLs).

Bug:  802137 
Change-Id: Iaf26006da9d9a45984c76bba9a5503bf2ccd5098
Reviewed-on: https://chromium-review.googlesource.com/944667
Commit-Queue: Matt Menke <mmenke@chromium.org>
Reviewed-by: Eric Roman <eroman@chromium.org>
Cr-Commit-Position: refs/heads/master@{#540924}
[modify] https://crrev.com/fde0811e64ba588de994b029782c941dd32dd83c/net/url_request/url_request_data_job_fuzzer.cc

Status: Fixed (was: Assigned)

Sign in to add a comment