New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 763098 link

Starred by 3 users
Status: Fixed
Owner:
█ ckrasic@chromium.org
Last visit > 30 days ago
Closed: May 2018
Cc:
█ xunji...@chromium.org
rch@chromium.org
agl@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux , Windows , Chrome , Mac , Fuchsia
Pri: 1
Type: Bug

Blocking:
issue 738275



Sign in to add a comment

URLRequestQuicTest is flaky

Project Member Reported by mattm@chromium.org, Sep 7 2017 
Run net_unittests on mac or windows with command line:
out/default/net_unittests --test-launcher-retry-limit=0 --test-launcher-jobs=20 --log-level=0 --gtest_filter="*Cert*:*X509*:*URLReq*:*HTTP*:*PathBu*"

With high reliability, one or more URLRequestQuicTest will fail & crash. (Easily reproducible on mac or windows, not so much on linux. Dunno why.)

It appears there are two problems:
1) the test uses a fixed port number, causing failures if multiple tests are run in parallel
2) the test uses EXPECT_EQ in some places it should use ASSERT_EQ, so that it crashes instead of just failing


[ RUN      ] URLRequestQuicTest.TestGetRequest
[28082:775:0907/124217.960383:1107239701121023:ERROR:quic_simple_server.cc(99)] Listen() failed: net::ERR_ADDRESS_IN_USE
../../net/url_request/url_request_quic_unittest.cc:141: Failure
Expected: (rv) >= (0), actual: -147 vs 0
Quic server fails to start
../../net/url_request/url_request_quic_unittest.cc:191: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.connect_start.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:193: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.connect_end.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:195: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.ssl_start.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:197: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.ssl_end.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:203: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.dns_start.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:205: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.dns_end.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:250: Failure
Value of: request->status().is_success()
  Actual: false
Expected: true
../../net/url_request/url_request_quic_unittest.cc:251: Failure
      Expected: kHelloBodyValue
      Which is: "Hello from QUIC Server"
To be equal to: delegate.data_received()
      Which is: ""
Received signal 11 SEGV_MAPERR 0000000000c8
 [0x00010964d4ae]
 [0x00010964d56d]
 [0x00010964b81c]
 [0x00010964d299]
 [0x7fffa6cf6b3a]
 [0x7fc6a8008eb0]
 [0x0001055c5d6e]
 [0x000104fe6507]
 [0x0001056b1f9e]
 [0x00010568d3a2]
 [0x00010568d31b]
 [0x00010568e46d]
 [0x00010568f7cc]
 [0x00010569cd8a]
 [0x0001056b53de]
 [0x00010569c972]
 [0x00010569c852]
 [0x0001058eac41]
 [0x0001058e9fd1]
 [0x000104ef780d]
 [0x000104ef7754]
 [0x000104ef76e3]
 [0x000104ef761c]
 [0x00010592e28d]
 [0x00010592b828]
 [0x00010592b673]
 [0x000104ef7259]
 [0x7fffa6ae7235]
[end of stack trace]
[937/1562] URLRequestQuicTest.TestGetRequest (CRASHED)


[ RUN      ] URLRequestQuicTest.DoNotCancelPushIfNotFoundInCache
[28212:775:0907/124225.286290:1107247027393211:ERROR:quic_simple_server.cc(99)] Listen() failed: net::ERR_ADDRESS_IN_USE
../../net/url_request/url_request_quic_unittest.cc:141: Failure
Expected: (rv) >= (0), actual: -147 vs 0
Quic server fails to start
../../net/url_request/url_request_quic_unittest.cc:191: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.connect_start.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:193: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.connect_end.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:195: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.ssl_start.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:197: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.ssl_end.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:203: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.dns_start.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:205: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.dns_end.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:428: Failure
Value of: request->status().is_success()
  Actual: false
Expected: true
../../net/url_request/url_request_quic_unittest.cc:434: Failure
      Expected: 4u
      Which is: 4
To be equal to: entries.size()
      Which is: 0
Received signal 11 SEGV_MAPERR 000000000020
 [0x00010abd34ae]
 [0x00010abd356d]
 [0x00010abd181c]
 [0x00010abd3299]
 [0x7fffa6cf6b3a]
 [0x7fffa6c804de]
 [0x000106566bc4]
 [0x000106c35f9e]
 [0x000106c113a2]
 [0x000106c112d6]
 [0x000106c1246d]
 [0x000106c137cc]
 [0x000106c20d8a]
 [0x000106c393de]
 [0x000106c20972]
 [0x000106c20852]
 [0x000106e6ec41]
 [0x000106e6dfd1]
 [0x00010647b80d]
 [0x00010647b754]
 [0x00010647b6e3]
 [0x00010647b61c]
 [0x000106eb228d]
 [0x000106eaf828]
 [0x000106eaf673]
 [0x00010647b259]
 [0x7fffa6ae7235]
[end of stack trace]
[1522/1562] URLRequestQuicTest.DoNotCancelPushIfNotFoundInCache (CRASHED)


[ RUN      ] URLRequestQuicTest.TestTwoRequests
[26652:775:0907/123018.767146:1106520471989570:ERROR:quic_simple_server.cc(99)] Listen() failed: net::ERR_ADDRESS_IN_USE
../../net/url_request/url_request_quic_unittest.cc:141: Failure
Expected: (rv) >= (0), actual: -147 vs 0
Quic server fails to start
../../net/url_request/url_request_quic_unittest.cc:191: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.connect_start.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:193: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.connect_end.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:195: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.ssl_start.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:197: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.ssl_end.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:203: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.dns_start.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:205: Failure
      Expected: session_reused
      Which is: false
To be equal to: load_timing_info.connect_timing.dns_end.is_null()
      Which is: true
../../net/url_request/url_request_quic_unittest.cc:186: Failure
      Expected: session_reused
      Which is: true
To be equal to: load_timing_info.socket_reused
      Which is: false
../../net/url_request/url_request_quic_unittest.cc:482: Failure
Value of: request->status().is_success()
  Actual: false
Expected: true
../../net/url_request/url_request_quic_unittest.cc:483: Failure
Value of: request2->status().is_success()
  Actual: false
Expected: true
../../net/url_request/url_request_quic_unittest.cc:484: Failure
      Expected: kHelloBodyValue
      Which is: "Hello from QUIC Server"
To be equal to: delegate.data_received()
      Which is: ""
../../net/url_request/url_request_quic_unittest.cc:485: Failure
      Expected: kHelloBodyValue
      Which is: "Hello from QUIC Server"
To be equal to: delegate2.data_received()
      Which is: ""
Received signal 11 SEGV_MAPERR 0000000000c8
 [0x000111a414ae]
 [0x000111a4156d]
 [0x000111a3f81c]
 [0x000111a41299]
 [0x7fffa6cf6b3a]
 [0x7fff5562f0f8]
 [0x00010d9bdd6e]
 [0x00010d3de507]
 [0x00010daa9f9e]
 [0x00010da853a2]
 [0x00010da8531b]
 [0x00010da8646d]
 [0x00010da877cc]
 [0x00010da94d8a]
 [0x00010daad3de]
 [0x00010da94972]
 [0x00010da94852]
 [0x00010dce2c41]
 [0x00010dce1fd1]
 [0x00010d2ef80d]
 [0x00010d2ef754]
 [0x00010d2ef6e3]
 [0x00010d2ef61c]
 [0x00010dd2628d]
 [0x00010dd23828]
 [0x00010dd23673]
 [0x00010d2ef259]
 [0x7fffa6ae7235]
[end of stack trace]
[1543/1562] URLRequestQuicTest.TestTwoRequests (CRASHED)

Comment 1 by w...@chromium.org, Dec 6 2017

Cc: agl@chromium.org rch@chromium.org xunji...@chromium.org
Components: Tests>Flaky
Labels: -Pri-2 M-65 OS-Chrome OS-Fuchsia OS-Linux Pri-1
Just got bitten by this on the Fuchsia bot (currently FYI, due to being more sensitive to flaky tests, ironically):
https://ci.chromium.org/buildbot/chromium.fyi/Fuchsia/11944

Looks like the port is fixed so that certificates will match. Is that something we could fake/redirect at a lower level, as we do for the HostResolver?

Comment 2 by w...@chromium.org, Dec 6 2017

Blocking: 738275

Comment 3 by w...@chromium.org, Dec 7 2017

Owner: rch@chromium.org
Status: Assigned (was: Untriaged)
Ryan, can you find someone to look at this ASAP? Normally I'd disable the tests with a TODO linked to this bug, but that would mean disabling the whole URLRequestQuicTest suite in this case. :-/

Comment 4 by rch@chromium.org, Dec 7 2017

Owner: ckrasic@chromium.org
ckrasic: Can you take a look at this?

Comment 5 by ckrasic@chromium.org, Dec 8 2017 

ok, looking.

Comment 6 by ckrasic@chromium.org, Dec 8 2017 

I think the crash is from QuicSimpleServer::Shutdown: socket_ is nullptr in the case that Listen failed, and QuicSimpleServer::Shutdown derefs socket_ without checking.

Spoke to Ryan, and I *think* the fixed port is unrelated to certs, so a fix there should set port to 0 and allow the platform to pick an available port, of course we have to add the get address thingy that exposes the chosen port in that case.
Project Member

Comment 7 by bugdroid1@chromium.org, Dec 8 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/70b06ef6ee7584f8220fd70b73ff66ce1b25b1c6

commit 70b06ef6ee7584f8220fd70b73ff66ce1b25b1c6
Author: Charles 'Buck' Krasic <ckrasic@chromium.org>
Date: Fri Dec 08 03:29:05 2017

QUIC - fix flaky URLRequestQuicTest.

o root cause test was needlessly using a fixed port with
  QuicSimpleServer, causing server initialization to fail
  if tests run in parallel.
o In case QuicSimpleServer fails during startup, Shutdown would
  crash because it null-dereferenced socket_.

R=rch@chromium.org

Bug:  763098 
Change-Id: Ic73ea2ec4aa710473829aac3243074a4c1131322
Reviewed-on: https://chromium-review.googlesource.com/816294
Reviewed-by: Ryan Hamilton <rch@chromium.org>
Commit-Queue: Buck Krasic <ckrasic@chromium.org>
Cr-Commit-Position: refs/heads/master@{#522699}
[modify] https://crrev.com/70b06ef6ee7584f8220fd70b73ff66ce1b25b1c6/net/tools/quic/quic_simple_server.cc
[modify] https://crrev.com/70b06ef6ee7584f8220fd70b73ff66ce1b25b1c6/net/url_request/url_request_quic_unittest.cc

Comment 8 by zhongyi@chromium.org, May 9 2018

Status: Fixed (was: Assigned)
Looks the change landed in Comment 7 fixed the flakiness. Closing this thread now, feel free to file new bugs if you believe the tests are still problematic.
►

Sign in to add a comment