New issue
Advanced search Search tips
Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Fuchsia
Pri: 1
Type: Bug

Blocked on:
issue 918724



Sign in to add a comment
link

Issue 913037: URLRequestTestHTTP.Redirect302Tests hangs under Fuchsia

Reported by w...@chromium.org, Dec 7 Project Member

Issue description

net_unittests now mostly run reliably, with the exception of Redirect302Tests, which sometimes hangs, e.g. in https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/fuchsia-fyi-x64-rel/6369 run:

...
Still waiting for the following processes to finish:
	/pkg/net_unittests__exec --gtest_flagfile=/tmp/.org.chromium.Chromium.gkhhgC/.org.chromium.Chromium.NhNGDP --single-process-tests --test-launcher-bot-mode --test-launcher-filter-file=/data/test_filter.txt --test-launcher-jobs=4 --test-launcher-output=/tmp/.org.chromium.Chromium.IDNNKa/test_results.xml --test-launcher-retry-limit=0 --test-launcher-summary-output=/data/test_summary.json
Still waiting for the following processes to finish:
	/pkg/net_unittests__exec --gtest_flagfile=/tmp/.org.chromium.Chromium.gkhhgC/.org.chromium.Chromium.NhNGDP --single-process-tests --test-launcher-bot-mode --test-launcher-filter-file=/data/test_filter.txt --test-launcher-jobs=4 --test-launcher-output=/tmp/.org.chromium.Chromium.IDNNKa/test_results.xml --test-launcher-retry-limit=0 --test-launcher-summary-output=/data/test_summary.json
[ RUN      ] UR2018-12-07 03:54:23,094:WARNING:root:Process exited with status code 1.
2018-12-07 03:54:23,094:INFO:root:Terminating kernel log reader.
2018-12-07 03:54:40,102:INFO:root:Shutting down QEMU.
LRequestTestHTTP.Redirect302Tests
[30496/30496] URLRequestTestHTTP.Redirect302Tests (TIMED OUT)
2 tests timed out:
    URLRequestTestHTTP.Redirect301Tests (../../net/url_request/url_request_unittest.cc:7869)
    URLRequestTestHTTP.Redirect302Tests (../../net/url_request/url_request_unittest.cc:7889)

Sometimes only Redirect302Tests times-out, but in other runs we also see Redirect301Tests timeout; it's unclear whether both tests are flaky, or just that one causes the other to flake, but from this run it looks like both.
 

Comment 1 by bugdroid1@chromium.org, Dec 7

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8e2d409ca8ec4acd698d75a1803f770f8e1bda92

commit 8e2d409ca8ec4acd698d75a1803f770f8e1bda92
Author: Wez <wez@chromium.org>
Date: Fri Dec 07 20:03:12 2018

[Fuchsia] Filter flaky HTTP redirect tests.

Bug: 913037
Change-Id: I2086df02bc0323c0a4b10cf3f24b628e62e28e39
Reviewed-on: https://chromium-review.googlesource.com/c/1368405
Reviewed-by: Fabrice de Gans-Riberi <fdegans@chromium.org>
Commit-Queue: Wez <wez@chromium.org>
Cr-Commit-Position: refs/heads/master@{#614793}
[modify] https://crrev.com/8e2d409ca8ec4acd698d75a1803f770f8e1bda92/testing/buildbot/filters/fuchsia.net_unittests.filter

Comment 2 by bugdroid1@chromium.org, Dec 20

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e31b72ded8986ce93d6ddba8c9f852d77610ce19

commit e31b72ded8986ce93d6ddba8c9f852d77610ce19
Author: Fabrice de Gans-Riberi <fdegans@chromium.org>
Date: Thu Dec 20 22:37:45 2018

[Fuchsia] Disable more URLRequestTestHTTP.Redirect* tests

These tests hang and time out regularly.

Bug: 913037
Change-Id: I0cca43ecf5ce6320371885d07c50c90986afd137
Reviewed-on: https://chromium-review.googlesource.com/c/1387389
Commit-Queue: Sergey Ulanov <sergeyu@chromium.org>
Reviewed-by: Sergey Ulanov <sergeyu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#618359}
[modify] https://crrev.com/e31b72ded8986ce93d6ddba8c9f852d77610ce19/testing/buildbot/filters/fuchsia.net_unittests.filter

Comment 3 by bugdroid1@chromium.org, Dec 21

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bbc985177229c1b20bc3c04d3c5689c3b0e178da

commit bbc985177229c1b20bc3c04d3c5689c3b0e178da
Author: Fabrice de Gans-Riberi <fdegans@chromium.org>
Date: Fri Dec 21 19:48:12 2018

[Fuchsia] Disable URLRequestTestHTTP.Redirect308Tests

It flaked twice in recent runs.

Bug: 913037
Change-Id: I2e7441c15118c8f458dd6d4bc18046aa52a72536
Reviewed-on: https://chromium-review.googlesource.com/c/1388555
Commit-Queue: Fabrice de Gans-Riberi <fdegans@chromium.org>
Reviewed-by: Sergey Ulanov <sergeyu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#618572}
[modify] https://crrev.com/bbc985177229c1b20bc3c04d3c5689c3b0e178da/testing/buildbot/filters/fuchsia.net_unittests.filter

Comment 4 by w...@chromium.org, Jan 9

Blockedon: 918724
Planning to re-enable these tests once CL for  issue 918724  lands.

Comment 5 by bugdroid1@chromium.org, Jan 11

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5798283ab35aa9153a708b092b7ab1a9281e5ff3

commit 5798283ab35aa9153a708b092b7ab1a9281e5ff3
Author: Wez <wez@chromium.org>
Date: Fri Jan 11 18:46:40 2019

[Fuchsia] Un-filter URLRequestTestHTTP.Redirect* tests.

These tests were flakily hanging, with unsufficient diagnostic output
for us to diagnose the issue. Now that we have per-Run() timeouts we
can re-enable these tests to get diagnostic call-stacks when they hang.

Bug: 913037
Change-Id: I5ef11d59e78304ac7182e3953d7a693f7ec502e4
Reviewed-on: https://chromium-review.googlesource.com/c/1406139
Commit-Queue: Wez <wez@chromium.org>
Reviewed-by: Sergey Ulanov <sergeyu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#622071}
[modify] https://crrev.com/5798283ab35aa9153a708b092b7ab1a9281e5ff3/testing/buildbot/filters/fuchsia.net_unittests.filter

Comment 6 by w...@chromium.org, Jan 12

Cc: w...@chromium.org
Components: Internals>SequenceManager
Owner: alexclarke@chromium.org
Status: Assigned (was: Started)
Un-filtering these tests we got a flake today:

[ RUN      ] URLRequestTestHTTP.Redirect307Tests
[385761:1315669333:0112/005706.684197:336026527:FATAL:scoped_task_environment.cc(323)] Run() timed out.
#00: base::debug::StackTrace::StackTrace(unsigned long) at stack_trace_fuchsia.cc:?
#01: logging::LogMessage::~LogMessage() at logging.cc:?
#02: base::internal::Invoker<base::internal::BindState<base::test::ScopedTaskEnvironment::ScopedTaskEnvironment(base::test::ScopedTaskEnvironment::MainThreadType, base::test::ScopedTaskEnvironment::ExecutionMode, base::test::ScopedTaskEnvironment::NowSource, base::trait_helpers::NotATraitTag)::$_0>, void ()>::Run(base::internal::BindStateBase*) at scoped_task_environment.cc:?
#03: base::internal::Invoker<base::internal::BindState<void (*)(base::RunLoop*, base::RepeatingCallback<void ()>), base::internal::UnretainedWrapper<base::RunLoop>, base::RepeatingCallback<void ()> >, void ()>::RunOnce(base::internal::BindStateBase*) at run_loop.cc:?
#04: base::OnceCallback<void (int)>::Run(int) && at elements_upload_data_stream_unittest.cc:?
#05: base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) at task_annotator.cc:?
#06: base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl(base::TimeTicks*) at thread_controller_with_message_pump_impl.cc:?
#07: base::MessagePumpFuchsia::Run(base::MessagePump::Delegate*) at message_pump_fuchsia.cc:?
#08: non-virtual thunk to base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run(bool) at thread_controller_with_message_pump_impl.cc:?
#09: base::RunLoop::Run() at run_loop.cc:?
#10: net::TestDelegate::RunUntilComplete() at url_request_test_util.cc:?
#11: net::URLRequestTestHTTP::HTTPRedirectMethodTest(GURL const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool) at url_request_unittest.cc:?
#12: net::URLRequestTestHTTP_Redirect307Tests_Test::TestBody() at url_request_unittest.cc:?
#13: testing::Test::Run() at gtest.cc:?
#14: testing::TestInfo::Run() at gtest.cc:?
#15: testing::TestCase::Run() at gtest.cc:?
#16: testing::internal::UnitTestImpl::RunAllTests() at gtest.cc:?
#17: testing::UnitTest::Run() at gtest.cc:?
#18: base::TestSuite::Run() at test_suite.cc:?
#19: int base::internal::Invoker<base::internal::BindState<int (base::TestSuite::*)(), base::internal::UnretainedWrapper<NetTestSuite> >, int ()>::RunImpl<int (base::TestSuite::* const&)(), std::__1::tuple<base::internal::UnretainedWrapper<NetTestSuite> > const&, 0ul>(int (base::TestSuite::* const&)(), std::__1::tuple<base::internal::UnretainedWrapper<NetTestSuite> > const&, std::__1::integer_sequence<unsigned long, 0ul>) at run_all_unittests.cc:?
#20: base::OnceCallback<bool ()>::Run() && at host_resolver_impl.cc:?
#21: base::(anonymous namespace)::LaunchUnitTestsInternal(base::OnceCallback<int ()>, unsigned long, int, bool, base::OnceCallback<void ()>) at unit_test_launcher.cc:?
#22: base::LaunchUnitTests(int, char**, base::OnceCallback<int ()>) at unit_test_launcher.cc:?
#23: main at run_all_unittests.cc:?
#24: pc 0x6fa19cd71f68 (libc.so,0x19f68)

[7140/31074] URLRequestTestHTTP.Redirect307Tests (CRASHED)

These tests have been round for a long time, and only recently started flaking; 

It's possible that this is a networking issue with our infrastructure, but the flakes started around the same time as the SequenceManager switch in ScopedTaskEnvironment.  alexclarke@, can you take a look?

Comment 7 Deleted

Comment 8 by bugdroid1@chromium.org, Jan 12

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/20dd5544bab21ac02936d2a6d0fba10368fdad49

commit 20dd5544bab21ac02936d2a6d0fba10368fdad49
Author: Wez <wez@chromium.org>
Date: Sat Jan 12 02:28:35 2019

Revert "[Fuchsia] Un-filter URLRequestTestHTTP.Redirect* tests."

This reverts commit 5798283ab35aa9153a708b092b7ab1a9281e5ff3.

Reason for revert: Tests continue to flake, and we at least have some example failure stacks now.

Original change's description:
> [Fuchsia] Un-filter URLRequestTestHTTP.Redirect* tests.
> 
> These tests were flakily hanging, with unsufficient diagnostic output
> for us to diagnose the issue. Now that we have per-Run() timeouts we
> can re-enable these tests to get diagnostic call-stacks when they hang.
> 
> Bug: 913037
> Change-Id: I5ef11d59e78304ac7182e3953d7a693f7ec502e4
> Reviewed-on: https://chromium-review.googlesource.com/c/1406139
> Commit-Queue: Wez <wez@chromium.org>
> Reviewed-by: Sergey Ulanov <sergeyu@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#622071}

TBR=wez@chromium.org,sergeyu@chromium.org,fdegans@chromium.org

Change-Id: I2fe40c2d71d3c5be93b8624ede11856371829eae
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 913037
Reviewed-on: https://chromium-review.googlesource.com/c/1407573
Reviewed-by: Wez <wez@chromium.org>
Commit-Queue: Wez <wez@chromium.org>
Cr-Commit-Position: refs/heads/master@{#622273}
[modify] https://crrev.com/20dd5544bab21ac02936d2a6d0fba10368fdad49/testing/buildbot/filters/fuchsia.net_unittests.filter

Comment 9 by w...@chromium.org, Jan 12

Cc: jkarlin@chromium.org
+jkarlin, since https://chromium-review.googlesource.com/c/1335869 affected redirect handling.

Comment 10 by estaab@chromium.org, Jan 25

Labels: Test-Flaky

Sign in to add a comment