OSX low frequency test crashing that kills the host OS on Swarming |
|||
Issue description[cc'ing random people who may be interested in diagnosing since this is a latent bug, as grepped via chromium-checkins@, feel free to un-cc] This is causing a lot of transparent retries -> increasing the load on OSX. It happens often on: - browser_tests - unit_tests It happens on both OSX 10.9 and 10.10. Server query: https://chromium-swarm.appspot.com/user/tasks?state=bot_died&limit=100 Example tasks: https://chromium-swarm.appspot.com/user/task/2e29ebcd9e5ba111 https://chromium-swarm.appspot.com/user/task/2e29ebcd9e5ba112 https://chromium-swarm.appspot.com/user/task/2e29901bb1d54111 https://chromium-swarm.appspot.com/user/task/2e29901bb1d54112 https://chromium-swarm.appspot.com/user/task/2e29623e273de811 https://chromium-swarm.appspot.com/user/task/2e29623e273de812 One of the nice thing is that the retried tasks that failed twice will likely reproduce more easily so it's worth trying these binaries first to try to reproduce locally. Visit a task's page to see the command to retrieve the input files and reproduce on your workstation. Failure will look like this: 016-04-13 10:08:19.186 unit_tests[21279:507] An uncaught exception was raised 2016-04-13 10:08:19.186 unit_tests[21279:507] Error (268435459) creating CGSWindow on line 263 2016-04-13 10:08:19.186 unit_tests[21279:507] ( 0 CoreFoundation 0x00007fff907a825c __exceptionPreprocess + 172 1 unit_tests 0x000000010532d77b _ZN30chrome_browser_application_mac12_GLOBAL__N_121ExceptionPreprocessorEP11objc_object + 299 2 libobjc.A.dylib 0x00007fff858dae75 objc_exception_throw + 43 3 CoreFoundation 0x00007fff907a810c +[NSException raise:format:] + 204 4 AppKit 0x00007fff88ee7e95 _NSCreateWindowWithOpaqueShape2 + 1403 5 AppKit 0x00007fff88ee6a21 -[NSWindow _commonAwake] + 3720 6 AppKit 0x00007fff88ef4b63 -[NSWindow _reallyDoOrderWindow:relativeTo:findKey:forCounter:force:isModal:] + 1121 7 AppKit 0x00007fff88ef4460 -[NSWindow _doOrderWindow:relativeTo:findKey:forCounter:force:isModal:] + 786 8 AppKit 0x00007fff88ef40e0 -[NSWindow orderWindow:relativeTo:] + 162 9 unit_tests 0x000000010503c08b _ZN23FramedBrowserWindowTest5SetUpEv + 155 10 unit_tests 0x0000000105c17c7b _ZN7testing4Test3RunEv + 299 11 unit_tests 0x0000000105c188ba _ZN7testing8TestInfo3RunEv + 410 12 unit_tests 0x0000000105c18e83 _ZN7testing8TestCase3RunEv + 451 13 unit_tests 0x0000000105c1f499 _ZN7testing8internal12UnitTestImpl11RunAllTestsEv + 841 14 unit_tests 0x0000000105c1f11d _ZN7testing8UnitTest3RunEv + 269 15 unit_tests 0x0000000105888da6 _ZN4base9TestSuite3RunEv + 150 16 unit_tests 0x00000001058823a9 _ZN4base15LaunchUnitTestsEiPPcRKNS_8CallbackIFivELNS_8internal8CopyModeE1EEE + 137 17 unit_tests 0x0000000105c5604e main + 286 18 unit_tests 0x0000000103a6eef4 start + 52 ) 2016-04-13 10:08:19.187 unit_tests[21279:507] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Error (268435459) creating CGSWindow on line 263' *** First throw call stack: ( 0 CoreFoundation 0x00007fff907a825c __exceptionPreprocess + 172 1 unit_tests 0x000000010532d77b _ZN30chrome_browser_application_mac12_GLOBAL__N_121ExceptionPreprocessorEP11objc_object + 299 2 libobjc.A.dylib 0x00007fff858dae75 objc_exception_throw + 43 3 CoreFoundation 0x00007fff907a810c +[NSException raise:format:] + 204 4 AppKit 0x00007fff88ee7e95 _NSCreateWindowWithOpaqueShape2 + 1403 5 AppKit 0x00007fff88ee6a21 -[NSWindow _commonAwake] + 3720 6 AppKit 0x00007fff88ef4b63 -[NSWindow _reallyDoOrderWindow:relativeTo:findKey:forCounter:force:isModal:] + 1121 7 AppKit 0x00007fff88ef4460 -[NSWindow _doOrderWindow:relativeTo:findKey:forCounter:force:isModal:] + 786 8 AppKit 0x00007fff88ef40e0 -[NSWindow orderWindow:relativeTo:] + 162 9 unit_tests 0x000000010503c08b _ZN23FramedBrowserWindowTest5SetUpEv + 155 10 unit_tests 0x0000000105c17c7b _ZN7testing4Test3RunEv + 299 11 unit_tests 0x0000000105c188ba _ZN7testing8TestInfo3RunEv + 410 12 unit_tests 0x0000000105c18e83 _ZN7testing8TestCase3RunEv + 451 13 unit_tests 0x0000000105c1f499 _ZN7testing8internal12UnitTestImpl11RunAllTestsEv + 841 14 unit_tests 0x0000000105c1f11d _ZN7testing8UnitTest3RunEv + 269 15 unit_tests 0x0000000105888da6 _ZN4base9TestSuite3RunEv + 150 16 unit_tests 0x00000001058823a9 _ZN4base15LaunchUnitTestsEiPPcRKNS_8CallbackIFivELNS_8internal8CopyModeE1EEE + 137 17 unit_tests 0x0000000105c5604e main + 286 18 unit_tests 0x0000000103a6eef4 start + 52 ) libc++abi.dylib: terminating with uncaught exception of type NSException Received signal 6 [0x0001061128da] [0x7fff8c24f5aa] [0x000000000000] [0x7fff8816bb2e] [0x7fff91a11f31] [0x7fff91a37952] [0x7fff858db30d] [0x7fff91a351d1] [0x7fff91a34c5b] [0x7fff858dafa1] [0x7fff907a7e09] [0x7fff88ee6b6c] [0x7fff88ef4b63] [0x7fff88ef4460] [0x7fff88ef40e0] [0x00010503c08b] [0x000105c17c7b] [0x000105c188ba] [0x000105c18e83] [0x000105c1f499] [0x000105c1f11d] [0x000105888da6] [0x0001058823a9] [0x000105c5604e] [0x000103a6eef4] [end of stack trace]
,
Apr 13 2016
Is the error always hex(268435459) = '0x10000003'? That probably decodes to MACH_SEND_INVALID_DEST, i.e. the thing we're IPC'ing is either dead or invalid.
,
Apr 13 2016
... which I think I wrote already. Yup! https://bugs.chromium.org/p/chromium/issues/detail?id=536195#c14
,
Apr 13 2016
I would guess with high probability that this is the same as: https://bugs.chromium.org/p/chromium/issues/detail?id=515627
,
Apr 14 2016
,
Apr 14 2016
chromium-try-flakes is tracking this as "unit_test" step flakiness: https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyFQsSBUZsYWtlIgp1bml0X3Rlc3RzDA. It currently posts updates to issue 550016 , because it was marked as a duplicate of the issue 599911 , which in turn was marked as duplicate of the automatically created issue 600048 . You can the sequence of issues by looking "reported in XXXXXX" note for each detected flaky.
,
Apr 26 2016
,
May 6 2016
|
|||
►
Sign in to add a comment |
|||
Comment 1 by thakis@chromium.org
, Apr 13 2016