All headless_browsertests fail flakily at the same time on Win10 Tests x64 (dbg) |
|||
Issue descriptionAll headless_browsertests fail at the same time on Win10 Tests x64 (dbg): https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=headless_browsertests&builder=chromium.win%3AWin10%20Tests%20x64%20(dbg) I've never seen this pattern of failures before. This only happens on this specific bot (win10 debug). Here's a link to an example failing run: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29/1749 The stderr is not helpful: [ RUN ] EnableResourceSchedulerTest.RunAsyncTest [9128:2616:0706/062502.834:21402750:ERROR:registration_protocol_win.cc(84)] TransactNamedPipe: The pipe has been ended. (0x6D) [9128:2616:0706/062502.834:21402750:WARNING:resource_bundle.cc(358)] locale_file_path.empty() for locale [9128:2616:0706/062502.881:21402796:ERROR:gpu_process_transport_factory.cc(642)] Switching to software compositing. [9128:2616:0706/062502.881:21402796:ERROR:gpu_process_transport_factory.cc(1007)] Lost UI shared context. [0706/062503.043:ERROR:registration_protocol_win.cc(56)] CreateFile: The system cannot find the file specified. (0x2) ../../headless/test/headless_browser_test.cc(252): error: Failed Abnormal renderer termination Stack trace: Backtrace: StackTraceGetter::CurrentStackTrace [0x00007FF7F50967F0+80] testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop [0x00007FF7F50AFF6A+90] testing::internal::AssertHelper::operator= [0x00007FF7F50AFA6A+90] headless::HeadlessAsyncDevTooledBrowserTest::RenderProcessExited [0x00007FF7F4CB8B20+192] headless::HeadlessWebContentsImpl::RenderProcessExited [0x00007FF9A4A20355+421] content::RenderProcessHostImpl::ProcessDied [0x00007FF98D869326+1222] content::RenderProcessHostImpl::OnChannelError [0x00007FF98D86B073+179] IPC::ChannelProxy::Context::OnDispatchError [0x00007FF9A8F90EAD+45] ??$Invoke@P8Context@ChannelProxy@IPC@@EAAXXZAEBV?$scoped_refptr@VContext@ChannelProxy@IPC@@@@$$V@?$FunctorTraits@P8Context@ChannelProxy@IPC@@EAAXXZX@internal@base@@SAXP8Context@ChannelProxy@IPC@@EAAXXZAEBV?$scoped_refptr@VContext@ChannelProxy@IPC@@@@@Z [0x00007FF9A8F97D33+67] base::internal::InvokeHelper<0,void>::MakeItSo<void (__cdecl IPC::ChannelProxy::Context::*const & __ptr64)(void) __ptr64,scoped_refptr<IPC::ChannelProxy::Context> const & __ptr64> [0x00007FF9A8F97CA6+86] base::internal::Invoker<base::internal::BindState<void (__cdecl IPC::ChannelProxy::Context::*)(void) __ptr64,scoped_refptr<IPC::ChannelProxy::Context> >,void __cdecl(void)>::RunImpl<void (__cdecl IPC::ChannelProxy::Context::*const & __ptr64)(void) __ptr64 [0x00007FF9A8F97C49+73] base::internal::Invoker<base::internal::BindState<void (__cdecl IPC::ChannelProxy::Context::*)(void) __ptr64,scoped_refptr<IPC::ChannelProxy::Context> >,void __cdecl(void)>::Run [0x00007FF9A8F97B7C+60] base::OnceCallback<void __cdecl(void)>::Run [0x00007FF99706C301+97] base::debug::TaskAnnotator::RunTask [0x00007FF9970D3C33+915] base::internal::IncomingTaskQueue::RunTask [0x00007FF99716CA34+212] I see that there is some win-specific code in the headless directory. I wonder if that could be flaky? It should be pretty simple to test using a windows checkout. skyostil, can you triage this?
,
Jul 9
That's really suspicious. Do you mean that in a given test run on this particular bot, all the tests fail in the same way? Sounds like we might be leaking some OS resource.
,
Jul 9
,
Jul 18
This showed up again today in the sheriff queue. The first 10 tests completed successfully, and then subsequent tests timed out. The log starts with [1/86] HeadlessBrowserTest.CreateAndDoNotDestroyBrowserContext (305 ms) [2/86] HeadlessBrowserTest.CreateAndDestroyBrowserContext (529 ms) [3/86] HeadlessBrowserTest.CreateAndDestroyWebContents (528 ms) [4/86] HeadlessBrowserTest.WebContentsAreDestroyedWithContext (529 ms) [5/86] HeadlessBrowserTest.CreateAndDoNotDestroyWebContents (372 ms) [6/86] HeadlessBrowserTest.DestroyAndCreateTwoWebContents (416 ms) [7/86] HeadlessBrowserTest.CreateWithBadURL (312 ms) [8/86] HeadlessBrowserTest.ClipboardCopyPasteText (279 ms) [9/86] HeadlessBrowserTest.PermissionManagerAlwaysASK (307 ms) [10/86] HeadlessBrowserTest.TraceUsingBrowserDevToolsTarget (384 ms) Still waiting for the following processes to finish: ".\headless_browsertests.exe" --brave-new-test-launcher --cfi-diag=0 --disable-gpu-process-for-dx12-vulkan-info-collection --gtest_also_run_disabled_tests --gtest_filter=HeadlessBrowserTest.WindowPrint --single_process --test-launcher-bot-mode --test-launcher-output="C:\Users\CHROME~1\AppData\Local\Temp\scoped_dir1704_23668\results1704_920\test_results.xml" --test-launcher-summary-output="C:\b\s\w\ioawr74m\output.json" and then it continues to wait for that one; other tests afterwards fail with the message in the original description. See https://chromium-swarm.appspot.com/task?id=3ec67d0476ef8b10&refresh=10&show_raw=1
,
Jul 20
,
Aug 20
This is failing on CFI with a possibly more actionable report: https://ci.chromium.org/buildbot/chromium.clang/ToTLinuxUBSanVptr/3664 [ RUN ] HeadlessBrowserTest.CreateAndDoNotDestroyWebContents [10536:10536:0820/031747.981981:10949318335:WARNING:resource_bundle.cc(358)] locale_file_path.empty() for locale [0820/031748.075398:WARNING:resource_bundle.cc(358)] locale_file_path.empty() for locale [10536:10536:0820/031748.086749:10949423096:ERROR:gpu_process_transport_factory.cc(984)] Lost UI shared context. ../../headless/lib/browser/headless_content_browser_client.cc:214:67: runtime error: member call on address 0x05b636d39d80 which does not point to an object of type 'headless::HeadlessBrowserImpl' 0x05b636d39d80: note: object has invalid vptr 00 00 00 00 f1 f8 21 c9 49 fa ff ff f1 0d 14 c9 49 fa ff ff 00 00 00 00 00 00 00 00 00 00 00 00 ^~~~~~~~~~~~~~~~~~~~~~~ invalid vptr #0 0x2120fa3 (/b/s/w/ir/out/Release/headless_browsertests+0x2120fa3) #1 0x612dd28 (/b/s/w/ir/out/Release/headless_browsertests+0x612dd28) #2 0x547731e (/b/s/w/ir/out/Release/headless_browsertests+0x547731e) #3 0x5476375 (/b/s/w/ir/out/Release/headless_browsertests+0x5476375) #4 0x666e8f8 (/b/s/w/ir/out/Release/headless_browsertests+0x666e8f8) #5 0x67419ac (/b/s/w/ir/out/Release/headless_browsertests+0x67419ac) #6 0x67d37dc (/b/s/w/ir/out/Release/headless_browsertests+0x67d37dc) #7 0x67400fc (/b/s/w/ir/out/Release/headless_browsertests+0x67400fc) #8 0x67381ce (/b/s/w/ir/out/Release/headless_browsertests+0x67381ce) #9 0x6737523 (/b/s/w/ir/out/Release/headless_browsertests+0x6737523) #10 0x67d414d (/b/s/w/ir/out/Release/headless_browsertests+0x67d414d) #11 0x7f81e97a4183 (/lib/x86_64-linux-gnu/libpthread.so.0+0x8183) #12 0x7f81e398803c (/lib/x86_64-linux-gnu/libc.so.6+0xfe03c) Does that help with fixing? If not, can you disable the two tests failing there?
,
Dec 27
|
|||
►
Sign in to add a comment |
|||
Comment 1 by vasi...@chromium.org
, Jul 9