Issue metadata
Sign in to add a comment
|
browser_tests "task-of-death" WebViewTest test cases are hard failing on OSX |
||||||||||||||||||||||||
Issue descriptionThis has been on-going for a long time. This cause a lot of CQ slow down with unnecessary retry. This needs to be fixed now. If you look at the task summary, it caused two different bots to die in the exact same manner, with what looks like a severe problem: Task summary: https://chromium-swarm.appspot.com/task?id=3a8e4db5792fd010 Try #1: https://chromium-swarm.appspot.com/task?id=3a8e4db5792fd011 Try #2: https://chromium-swarm.appspot.com/task?id=3a8e4db5792fd012 To be clear: the exact same code ran in both tries. This badly affects the Chromium infrastructure. It's clear that something really wrong is happening with this test case. Both cases fail in similar way, except the other failed with another WebViewTests test case. --- CUT HERE --- [622/788] WebViewTests/WebViewTest.ReloadAfterCrash/1 (3424 ms) [1220/120824.816176:ERROR:kill_posix.cc(84)] Unable to terminate process group 22495: No such process (3) [ RUN ] WebViewTests/WebViewTest.ReloadAfterCrash/0 [22495:55043:1220/120819.315862:WARNING:mac_util.mm(226)] Failed to set backup exclusion for file '/private/var/folders/9x/6c6sv3cj4j53wzpzthbp4ksm0000gm/T/.org.chromium.Chromium.g10ghY/dyhQDEd/Default/History': Error Domain=NSOSStatusErrorDomain Code=-50 "The operation couldn’t be completed. (OSStatus error -50.)" (paramErr: error in user parameter list) (-50) [22495:5123:1220/120819.323712:WARNING:notification_platform_bridge_mac.mm(533)] AlertNotificationService: XPC connection invalidated. [22495:55043:1220/120819.464286:WARNING:mac_util.mm(226)] Failed to set backup exclusion for file '/private/var/folders/9x/6c6sv3cj4j53wzpzthbp4ksm0000gm/T/.org.chromium.Chromium.g10ghY/dyhQDEd/Default/Favicons': Error Domain=NSOSStatusErrorDomain Code=-50 "The operation couldn’t be completed. (OSStatus error -50.)" (paramErr: error in user parameter list) (-50) [22504:771:1220/120820.400234:WARNING:mac_util.mm(157)] Couldn't get the main display's color space, using generic [22504:771:1220/120820.403990:ERROR:gl_initializer_mac.cc(59)] Error choosing pixel format. [22504:771:1220/120820.404020:ERROR:gl_initializer_mac.cc(138)] GLSurfaceCGL::InitializeOneOff failed. [22504:771:1220/120820.410730:WARNING:vt_video_decode_accelerator_mac.cc(178)] Failed to create VTDecompressionSession: Error Domain=NSOSStatusErrorDomain Code=-8973 "The operation couldn’t be completed. (OSStatus error -8973.)" (codecOpenErr) (-8973) [22504:771:1220/120820.411771:WARNING:vt_video_decode_accelerator_mac.cc(200)] Failed to create hardware VideoToolbox session [22504:771:1220/120820.466700:ERROR:gl_initializer_mac.cc(59)] Error choosing pixel format. [22504:771:1220/120820.466727:ERROR:gl_initializer_mac.cc(138)] GLSurfaceCGL::InitializeOneOff failed. [22504:771:1220/120820.472054:ERROR:vt_video_encode_accelerator_mac.cc(517)] VTCompressionSessionCreate failed: -12908 [22504:771:1220/120820.472205:ERROR:viz_main_impl.cc(196)] Exiting GPU process due to errors during initialization [22495:44803:1220/120820.493403:ERROR:browser_gpu_channel_host_factory.cc(120)] Failed to launch GPU process. 2017-12-20 12:08:20.787 browser_tests[22495:303] An uncaught exception was raised 2017-12-20 12:08:20.787 browser_tests[22495:303] Error (268435459) creating CGSWindow 2017-12-20 12:08:20.788 browser_tests[22495:303] ( 0 CoreFoundation 0x00007fff9466125c __exceptionPreprocess + 172 1 browser_tests 0x000000010454a2ff _ZN6chromeL25ObjcExceptionPreprocessorEP11objc_object + 1039 2 libobjc.A.dylib 0x00007fff89793e75 objc_exception_throw + 43 3 CoreFoundation 0x00007fff9466110c +[NSException raise:format:] + 204 4 AppKit 0x00007fff8cee6f81 _NXCreateWindowWithStyleMask + 366 5 AppKit 0x00007fff8cee6d7d _NSCreateWindow + 64 6 AppKit 0x00007fff8cd9f72c -[NSWindow _commonAwake] + 2963 7 AppKit 0x00007fff8cc7b400 -[NSWindow _commonInitFrame:styleMask:backing:defer:] + 882 8 AppKit 0x00007fff8cc7a882 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 1054 9 AppKit 0x00007fff8cc7a458 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 45 10 browser_tests 0x00000001082792c8 _ZN15StatusBubbleMac6CreateEv + 184 11 browser_tests 0x00000001082791ad _ZN15StatusBubbleMacC2EP8NSWindowP11objc_object + 141 12 browser_tests 0x00000001081b6ec8 -[BrowserWindowController initWithBrowser:takeOwnership:] + 2248 13 browser_tests 0x00000001081c1ea2 _ZN13BrowserWindow19CreateBrowserWindowEP7Browserb + 50 14 browser_tests 0x000000010802213b _ZN7BrowserC2ERKNS_12CreateParamsE + 2363 15 browser_tests 0x000000010806b7aa _ZN25StartupBrowserCreatorImpl17OpenTabsInBrowserEP7BrowserbRKNSt3__16vectorI10StartupTabNS2_9allocatorIS4_EEEE + 186 16 browser_tests 0x000000010806c5c5 _ZN25StartupBrowserCreatorImpl22RestoreOrCreateBrowserERKNSt3__16vectorI10StartupTabNS0_9allocatorIS2_EEEENS_19BrowserOpenBehaviorEjbb + 629 17 browser_tests 0x000000010806b38d _ZN25StartupBrowserCreatorImpl22DetermineURLsAndLaunchEbRKNSt3__16vectorI4GURLNS0_9allocatorIS2_EEEE + 589 18 browser_tests 0x000000010806ac22 _ZN25StartupBrowserCreatorImpl6LaunchEP7ProfileRKNSt3__16vectorI4GURLNS2_9allocatorIS4_EEEEb + 1122 19 browser_tests 0x0000000108067c3a _ZN21StartupBrowserCreator13LaunchBrowserERKN4base11CommandLineEP7ProfileRKNS0_8FilePathEN6chrome7startup16IsProcessStartupENSA_10IsFirstRunE + 410 20 browser_tests 0x00000001080679c2 _ZN21StartupBrowserCreator18ProcessCmdLineImplERKN4base11CommandLineERKNS0_8FilePathEbP7ProfileRKNSt3__16vectorIS8_NS9_9allocatorIS8_EEEE + 2146 21 browser_tests 0x0000000108067017 _ZN21StartupBrowserCreator5StartERKN4base11CommandLineERKNS0_8FilePathEP7ProfileRKNSt3__16vectorIS8_NS9_9allocatorIS8_EEEE + 103 22 browser_tests 0x00000001044a730a _ZN22ChromeBrowserMainParts25PreMainMessageLoopRunImplEv + 4522 23 browser_tests 0x00000001044a605e _ZN22ChromeBrowserMainParts21PreMainMessageLoopRunEv + 62 24 browser_tests 0x000000010241b143 _ZN7content15BrowserMainLoop21PreMainMessageLoopRunEv + 67 25 browser_tests 0x0000000102963e75 _ZN7content17StartupTaskRunner14RunAllTasksNowEv + 117 26 browser_tests 0x000000010241962f _ZN7content15BrowserMainLoop18CreateStartupTasksEv + 687 27 browser_tests 0x000000010241e57a _ZN7content21BrowserMainRunnerImpl10InitializeERKNS_18MainFunctionParamsE + 650 28 browser_tests 0x00000001024175f2 _ZN7content11BrowserMainERKNS_18MainFunctionParamsE + 210 29 browser_tests 0x00000001041da604 _ZN7content21ContentMainRunnerImpl3RunEv + 564 30 browser_tests 0x0000000106bef9b2 _ZN15service_manager4MainERKNS_10MainParamsE + 2482 31 browser_tests 0x00000001041d99a4 _ZN7content11ContentMainERKNS_17ContentMainParamsE + 68 32 browser_tests 0x0000000104b73b36 _ZN7content15BrowserTestBase5SetUpEv + 3078 33 browser_tests 0x00000001043fbf36 _ZN20InProcessBrowserTest5SetUpEv + 502 34 browser_tests 0x0000000101644761 _ZN7testing4Test3RunEv + 97 35 browser_tests 0x0000000101645350 _ZN7testing8TestInfo3RunEv + 288 36 browser_tests 0x0000000101645977 _ZN7testing8TestCase3RunEv + 263 37 browser_tests 0x000000010164cf07 _ZN7testing8internal12UnitTestImpl11RunAllTestsEv + 903 38 browser_tests 0x000000010164cb53 _ZN7testing8UnitTest3RunEv + 163 39 browser_tests 0x000000010441f277 _ZN4base9TestSuite3RunEv + 167 40 browser_tests 0x00000001042bfde5 _ZN21ChromeTestSuiteRunner12RunTestSuiteEiPPc + 37 41 browser_tests 0x0000000104bd5733 _ZN7content11LaunchTestsEPNS_20TestLauncherDelegateEmiPPc + 467 42 browser_tests 0x00000001042c0227 _Z17LaunchChromeTestsmPN7content20TestLauncherDelegateEiPPc + 135 43 browser_tests 0x00000001042bfd3e main + 94 44 libdyld.dylib 0x00007fff894025fd start + 1 ) Received signal 4 <unknown> 7fff8cff5e34 0 browser_tests 0x00000001042dd4fc base::debug::StackTrace::StackTrace(unsigned long) + 28 1 browser_tests 0x00000001042dd331 base::debug::(anonymous namespace)::StackDumpSignalHandler(int, __siginfo*, void*) + 2401 2 libsystem_platform.dylib 0x00007fff901085aa _sigtramp + 26 3 ??? 0x0000000000000000 0x0 + 0 4 AppKit 0x00007fff8cff628f -[NSApplication reportException:] + 134 5 CoreFoundation 0x00007fff94661702 __handleUncaughtException + 706 6 libobjc.A.dylib 0x00007fff89794304 _objc_terminate() + 94 7 libc++abi.dylib 0x00007fff958ee1d1 std::__terminate(void (*)()) + 8 8 libc++abi.dylib 0x00007fff958ede7b __cxa_rethrow + 109 9 libobjc.A.dylib 0x00007fff89794049 objc_exception_rethrow + 40 10 AppKit 0x00007fff8cee707c _NXCreateWindowWithStyleMask + 617 11 AppKit 0x00007fff8cee6d7d _NSCreateWindow + 64 12 AppKit 0x00007fff8cd9f72c -[NSWindow _commonAwake] + 2963 13 AppKit 0x00007fff8cc7b400 -[NSWindow _commonInitFrame:styleMask:backing:defer:] + 882 14 AppKit 0x00007fff8cc7a882 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 1054 15 AppKit 0x00007fff8cc7a458 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 45 16 browser_tests 0x00000001082792c8 StatusBubbleMac::Create() + 184 17 browser_tests 0x00000001082791ad StatusBubbleMac::StatusBubbleMac(NSWindow*, objc_object*) + 141 18 browser_tests 0x00000001081b6ec8 -[BrowserWindowController initWithBrowser:takeOwnership:] + 2248 19 browser_tests 0x00000001081c1ea2 BrowserWindow::CreateBrowserWindow(Browser*, bool) + 50 20 browser_tests 0x000000010802213b Browser::Browser(Browser::CreateParams const&) + 2363 21 browser_tests 0x000000010806b7aa StartupBrowserCreatorImpl::OpenTabsInBrowser(Browser*, bool, std::__1::vector<StartupTab, std::__1::allocator<StartupTab> > const&) + 186 22 browser_tests 0x000000010806c5c5 StartupBrowserCreatorImpl::RestoreOrCreateBrowser(std::__1::vector<StartupTab, std::__1::allocator<StartupTab> > const&, StartupBrowserCreatorImpl::BrowserOpenBehavior, unsigned int, bool, bool) + 629 23 browser_tests 0x000000010806b38d StartupBrowserCreatorImpl::DetermineURLsAndLaunch(bool, std::__1::vector<GURL, std::__1::allocator<GURL> > const&) + 589 24 browser_tests 0x000000010806ac22 StartupBrowserCreatorImpl::Launch(Profile*, std::__1::vector<GURL, std::__1::allocator<GURL> > const&, bool) + 1122 25 browser_tests 0x0000000108067c3a StartupBrowserCreator::LaunchBrowser(base::CommandLine const&, Profile*, base::FilePath const&, chrome::startup::IsProcessStartup, chrome::startup::IsFirstRun) + 410 26 browser_tests 0x00000001080679c2 StartupBrowserCreator::ProcessCmdLineImpl(base::CommandLine const&, base::FilePath const&, bool, Profile*, std::__1::vector<Profile*, std::__1::allocator<Profile*> > const&) + 2146 27 browser_tests 0x0000000108067017 StartupBrowserCreator::Start(base::CommandLine const&, base::FilePath const&, Profile*, std::__1::vector<Profile*, std::__1::allocator<Profile*> > const&) + 103 28 browser_tests 0x00000001044a730a ChromeBrowserMainParts::PreMainMessageLoopRunImpl() + 4522 29 browser_tests 0x00000001044a605e ChromeBrowserMainParts::PreMainMessageLoopRun() + 62 30 browser_tests 0x000000010241b143 content::BrowserMainLoop::PreMainMessageLoopRun() + 67 31 browser_tests 0x0000000102963e75 content::StartupTaskRunner::RunAllTasksNow() + 117 32 browser_tests 0x000000010241962f content::BrowserMainLoop::CreateStartupTasks() + 687 33 browser_tests 0x000000010241e57a content::BrowserMainRunnerImpl::Initialize(content::MainFunctionParams const&) + 650 34 browser_tests 0x00000001024175f2 content::BrowserMain(content::MainFunctionParams const&) + 210 35 browser_tests 0x00000001041da604 content::ContentMainRunnerImpl::Run() + 564 36 browser_tests 0x0000000106bef9b2 service_manager::Main(service_manager::MainParams const&) + 2482 37 browser_tests 0x00000001041d99a4 content::ContentMain(content::ContentMainParams const&) + 68 38 browser_tests 0x0000000104b73b36 content::BrowserTestBase::SetUp() + 3078 39 browser_tests 0x00000001043fbf36 InProcessBrowserTest::SetUp() + 502 40 browser_tests 0x0000000101644761 testing::Test::Run() + 97 41 browser_tests 0x0000000101645350 testing::TestInfo::Run() + 288 42 browser_tests 0x0000000101645977 testing::TestCase::Run() + 263 43 browser_tests 0x000000010164cf07 testing::internal::UnitTestImpl::RunAllTests() + 903 44 browser_tests 0x000000010164cb53 testing::UnitTest::Run() + 163 45 browser_tests 0x000000010441f277 base::TestSuite::Run() + 167 46 browser_tests 0x00000001042bfde5 ChromeTestSuiteRunner::RunTestSuite(int, char**) + 37 47 browser_tests 0x0000000104bd5733 content::LaunchTests(content::TestLauncherDelegate*, unsigned long, int, char**) + 467 48 browser_tests 0x00000001042c0227 LaunchChromeTests(unsigned long, content::TestLauncherDelegate*, int, char**) + 135 49 browser_tests 0x00000001042bfd3e main + 94 50 libdyld.dylib 0x00007fff894025fd start + 1 [end of stack trace] [623/788] WebViewTests/WebViewTest.ReloadAfterCrash/0 (CRASHED) --- CUT HERE --- Thanks!
,
Dec 20 2017
+adding mac experts. This is unlikely to be related to the tests that are crashing. The problem happens on window creation, which happens before any test code starts to run, so whatever happened, happened before this test started. From the logs I can see: [22504:771:1220/120820.400234:WARNING:mac_util.mm(157)] Couldn't get the main display's color space, using generic [22504:771:1220/120820.403990:ERROR:gl_initializer_mac.cc(59)] Error choosing pixel format. [22504:771:1220/120820.404020:ERROR:gl_initializer_mac.cc(138)] GLSurfaceCGL::InitializeOneOff failed. [22504:771:1220/120820.410730:WARNING:vt_video_decode_accelerator_mac.cc(178)] Failed to create VTDecompressionSession: Error Domain=NSOSStatusErrorDomain Code=-8973 "The operation couldn’t be completed. (OSStatus error -8973.)" (codecOpenErr) (-8973) [22504:771:1220/120820.411771:WARNING:vt_video_decode_accelerator_mac.cc(200)] Failed to create hardware VideoToolbox session [22504:771:1220/120820.466700:ERROR:gl_initializer_mac.cc(59)] Error choosing pixel format. [22504:771:1220/120820.466727:ERROR:gl_initializer_mac.cc(138)] GLSurfaceCGL::InitializeOneOff failed. [22504:771:1220/120820.472054:ERROR:vt_video_encode_accelerator_mac.cc(517)] VTCompressionSessionCreate failed: -12908 [22504:771:1220/120820.472205:ERROR:viz_main_impl.cc(196)] Exiting GPU process due to errors during initialization [22495:44803:1220/120820.493403:ERROR:browser_gpu_channel_host_factory.cc(120)] Failed to launch GPU process. 2017-12-20 12:08:20.787 browser_tests[22495:303] An uncaught exception was raised 2017-12-20 12:08:20.787 browser_tests[22495:303] Error (268435459) creating CGSWindow We probably need to figure out why we are failing to create the window (and possibly why the GPU process failed to launch).
,
Dec 20 2017
Reassigning as I won't have a chance to look at this before I'm OOO.
,
Dec 20 2017
+more magical Mac peeps.
,
Dec 20 2017
This seems to be a variant of, if not a dup of, issue 536195 .
,
Dec 20 2017
,
Dec 20 2017
Issue 536195 points to fixed Issue 515627 which used ScopedFakeNSWindowFullscreen in browser_window_controller_unittest.mm to avoid using "real" full-screen. Do we need to do the same thing in other test harnesses, possibly under a command-line flag?
,
Dec 20 2017
If we needed to fake out fullscreen in unit tests, I don't see why we wouldn't have to for browser tests.
,
Dec 20 2017
The window server reacts poorly to several factors [these are educated guesses] 1) running in a VM [w/o hardware acceleration] 2) too many parallel chrome instances 3) improper use of window server APIs/use of private APIs. We've observed and fixed several cases of (3). If the WebView tests are using "real" full-screen, I'd definitely recommend using ScopedFakeNSWindowFullscreen. [e.g. the window server probably reacts poorly to multiple windows trying to fullscreen at the same time]
,
Dec 20 2017
FTR, you can retry the task with the "Retry" button, map but not run it with the "Debug task" button (so you can ssh/vnc in and debug yourself) or fetch to run it locally with the relevant command in the help page. Since it's 10.9, "Debug task" avenue is probably the best avenue. Thanks
,
Dec 21 2017
And to clarify; no need to get out of vacations for this. I want this to be fixed as soon as reasonably possible, with dropping 10.9 support being one avenue *blink* *blink*, but this should not exceed normal working hours.
,
Dec 21 2017
Re#8: Webview only has real fullscreen tests on the interactive_ui_tests target, and they are disabled on Mac for causing issues, which seems to be likely what you are describing (see https://chromium.googlesource.com/chromium/src/+/db0b963d1d5fb33f56f2c410f036cbd93041b06e/chrome/browser/apps/guest_view/web_view_interactive_browsertest.cc#1374).
,
Jan 23 2018
Is there any way for the test code to ascertain, during startup, if it's 'safe' to proceed? If the flake rate is low, we could probably consider (in the short term) bailing individual test runs if we knew which ones ...
,
Jan 24 2018
,
Apr 5 2018
,
Apr 5 2018
,
Apr 5 2018
,
Apr 5 2018
Turns out that this is being mostly debugged under Issue 828031 .
,
Apr 5 2018
|
|||||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||||
Comment 1 by mar...@chromium.org
, Dec 20 2017