CrWinClangLLD tester bot failing on larger tests, probably due to LTO |
|||
Issue descriptionThe CrWinClangLLD builder bot finally went green after the inline asm issue ( bug 666994 ), so now we get to run the tests, and it turns out they fail. This is the first real test run since we enabled LTO (last green Nov 11), so I'm guessing LTO is the culprit. It looks like any test binary large enough to link in the browser parts of chrome is failing somewhere during browser startup. Because we don't have PDBs yet, the stack traces are pretty unhelpful. Example build: https://build.chromium.org/p/chromium.fyi/builders/CrWinClangLLD%20tester/builds/1250
,
Dec 1 2016
I can repro locally:
D:\src\chromium\src>out\lto\browser_tests --gtest_filter=BrowserTest.GetSizeForN
ewRenderView
IMPORTANT DEBUGGING NOTE: each test is run inside its own process.
For debugging a test inside a debugger, use the
--gtest_filter=<your_test_name> flag along with either
--single_process (to run the test in one launcher/browser process) or
--single-process (to do the above, and also run Chrome in single-process mode).
Using sharding settings from environment. This is shard 0/1
Using 1 parallel jobs.
Note: Google Test filter = BrowserTest.GetSizeForNewRenderView
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from BrowserTest, where TypeParam =
[ RUN ] BrowserTest.GetSizeForNewRenderView
[6496:9312:1201/141743.779:ERROR:configuration_policy_handler_list.cc(92)] Unkno
wn policy: EnableMemoryInfo
[6496:9312:1201/141743.780:ERROR:configuration_policy_handler_list.cc(92)] Unkno
wn policy: RemoteAccessClientFirewallTraversal
[6496:9312:1201/141743.781:ERROR:configuration_policy_handler_list.cc(92)] Unkno
wn policy: RemoteAccessHostMatchUsername
[6496:9312:1201/141743.878:ERROR:configuration_policy_handler_list.cc(92)] Unkno
wn policy: EnableMemoryInfo
[6496:9312:1201/141743.878:ERROR:configuration_policy_handler_list.cc(92)] Unkno
wn policy: RemoteAccessClientFirewallTraversal
[6496:9312:1201/141743.879:ERROR:configuration_policy_handler_list.cc(92)] Unkno
wn policy: RemoteAccessHostMatchUsername
Backtrace:
RtlInitUnicodeString [0x778CE43E+406]
RtlFreeHeap [0x778CE0A3+126]
HeapFree [0x758E14AD+20]
nacl_global_xlate_base [0x028CCD93+9949455]
nacl_global_xlate_base [0x028CCD3F+9949371]
nacl_global_xlate_base [0x028CCCB8+9949236]
InjectDumpForHangDebugging [0x05E2B94D+16707279]
nacl_global_xlate_base [0x03259744+19962560]
nacl_global_xlate_base [0x0325327C+19936760]
nacl_global_xlate_base [0x03252CFF+19935355]
nacl_global_xlate_base [0x03252CCE+19935306]
nacl_global_xlate_base [0x03257110+19952780]
InjectDumpForHangDebugging [0x050F41BC+2848574]
IsSandboxedProcess [0x033AAFF9+9421]
nacl_global_xlate_base [0x02C532D9+13645397]
IsSandboxedProcess [0x03ACF0B3+7497095]
IsSandboxedProcess [0x03ACF079+7497037]
nacl_global_xlate_base [0x02B6A119+12690581]
GetHandleVerifier [0x063DEF27+5502599]
InjectDumpForHangDebugging [0x05E2C95E+16711392]
InjectDumpForHangDebugging [0x05E2D1F6+16713592]
InjectDumpForHangDebugging [0x05E2C44B+16710093]
GetHandleVerifier [0x063E1469+5512137]
GetHandleVerifier [0x063E091C+5509244]
InjectDumpForHangDebugging [0x05E4D62E+16845744]
InjectDumpForHangDebugging [0x05E6661B+16948125]
nacl_global_xlate_base [0x02BD1BD3+13115215]
nacl_global_xlate_base [0x02BD1C6E+13115370]
InjectDumpForHangDebugging [0x05E66500+16947842]
InjectDumpForHangDebugging [0x05E5FF15+16921751]
BaseThreadInitThunk [0x758E336A+18]
RtlInitializeExceptionChain [0x778D9902+99]
RtlInitializeExceptionChain [0x778D98D5+54]
[1/1] BrowserTest.GetSizeForNewRenderView (2472 ms)
1 test failed:
BrowserTest.GetSizeForNewRenderView (../../chrome/browser/ui/browser_browser
test.cc:2564)
chrome.exe is also broken; it crashes when trying to navigate somewhere (it survives startup). This used to work, so it must be a regression.
,
Dec 1 2016
content_shell.exe also doesn't work, but there only the renderer crashes, not the main process. Trying chrome.exe --single-process http://www.google.com/ running under windbg, it seems to crash here: (29e0.2f5c): Access violation - code c0000005 (first chance) First chance exceptions are reported before any exception handling. This exception may be expected and handled. ntdll32!RtlpLowFragHeapFree+0x31: 778ce43e 8b4604 mov eax,dword ptr [esi+4] ds:002b:34fc7794=???????? 0:020:x86> k # ChildEBP RetAddr 00 067af32c 778ce0a3 ntdll32!RtlpLowFragHeapFree+0x31 01 067af344 758e14ad ntdll32!RtlFreeHeap+0x105 02 067af358 5f97dfff kernel32!HeapFree+0x14 WARNING: Stack unwind information not available. Following frames may be wrong. 03 067af370 5f97dfab chrome_5f130000!ChromeMain+0x2f7b 04 067af37c 5f97df24 chrome_5f130000!ChromeMain+0x2f27 05 067af38c 60f94ded chrome_5f130000!ChromeMain+0x2ea0 06 067af39c 5fccc153 chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0xaa2404 07 067af3a8 5fccc0dd chrome_5f130000!ChromeMain+0x3510cf 08 067af3b4 5fcc80bb chrome_5f130000!ChromeMain+0x351059 09 067af3d0 5fcc7fc7 chrome_5f130000!ChromeMain+0x34d037 0a 067af3dc 5fcbcd73 chrome_5f130000!ChromeMain+0x34cf43 0b 067af3fc 5fcc1ba7 chrome_5f130000!ChromeMain+0x341cef 0c 067af428 5fcbe36c chrome_5f130000!ChromeMain+0x346b23 0d 067af450 5fcbe175 chrome_5f130000!ChromeMain+0x3432e8 0e 067af4b4 5fcc7b41 chrome_5f130000!ChromeMain+0x3430f1 0f 067af500 5fcc71d6 chrome_5f130000!ChromeMain+0x34cabd 10 067af560 5fcc7575 chrome_5f130000!ChromeMain+0x34c152 11 067af5c4 5fcc741a chrome_5f130000!ChromeMain+0x34c4f1 12 067af60c 5fcc751d chrome_5f130000!ChromeMain+0x34c396 13 067af674 5fcb453e chrome_5f130000!ChromeMain+0x34c499 14 067af694 62012867 chrome_5f130000!ChromeMain+0x3394ba 15 067af720 61eaae6e chrome_5f130000!GetHandleVerifier+0xf8e27 16 067af7b4 61eab706 chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19b8485 17 067af7c8 61eaa96b chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19b8d1d 18 067af878 62015f29 chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19b7f82 19 067af8c0 620153dc chrome_5f130000!GetHandleVerifier+0xfc4e9 1a 067af8f4 61eca38e chrome_5f130000!GetHandleVerifier+0xfb99c 1b 067af960 61ee24db chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19d79a5 1c 067af968 5fbbad49 chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19efaf2 1d 067af97c 5fbbade4 chrome_5f130000!ChromeMain+0x23fcc5 1e 067af994 61ee23c0 chrome_5f130000!ChromeMain+0x23fd60 1f 067af9e0 61edc8bc chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19ef9d7 20 067afa04 758e336a chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19e9ed3 21 067afa10 778d9902 kernel32!BaseThreadInitThunk+0xe 22 067afa50 778d98d5 ntdll32!__RtlUserThreadStart+0x70 23 067afa68 00000000 ntdll32!_RtlUserThreadStart+0x1b
,
Dec 9 2016
Build chrome again, but this time saved the map file for chrome.dll to symbolize the stack manually. Below is the stack, where I've looked up the symbol for the return address from each frame in the map file: # ChildEBP RetAddr 00 0662ef04 778ce0a3 ntdll32!RtlpLowFragHeapFree+0x31 01 0662ef1c 758e14ad ntdll32!RtlFreeHeap+0x105 02 0662ef30 5f84241b kernel32!HeapFree+0x14 WARNING: Stack unwind information not available. Following frames may be wrong. 10842401 ?WinHeapFree@allocator@base@@YAXPAX@Z 03 0662ef48 5f8423c7 chrome_5f000000!ChromeMain+0x3397 108423bc ?DefaultWinHeapFreeImpl@?A@@YAXPBUAllocatorDispatch@allocator@base@@PAX@Z 04 0662ef54 5f842307 chrome_5f000000!ChromeMain+0x3343 108422f8 _free 05 0662ef64 5f861585 chrome_5f000000!ChromeMain+0x3283 10861542 ?_Tidy@?$vector@W4__MIDL___MIDL_itf_inputscope_0000_0000_0001@@V?$allocator@W4__MIDL___MIDL_itf_inputscope_0000_0000_0001@@@std@@@std@@IAEXXZ (lots of functions ICF'd to that address) 06 0662ef74 5fbb2b1b chrome_5f000000!ovly_debug_event+0x1e70d 10bb2ae2 ??1ThrottlingResourceHandler@content@@UAE@XZ 07 0662ef80 5fbb2aa5 chrome_5f000000!ovly_debug_event+0x36fca3 10bb2a9a ??_GThrottlingResourceHandler@content@@UAEPAXI@Z 08 0662ef8c 5fbae973 chrome_5f000000!ovly_debug_event+0x36fc2d 10bae908 ??1ResourceLoader@content@@UAE@XZ 09 0662efa8 5fbae87f chrome_5f000000!ovly_debug_event+0x36bafb 10bae874 ??_GResourceLoader@content@@UAEPAXI@Z 0a 0662efb4 5fba32e9 chrome_5f000000!ovly_debug_event+0x36ba07 10ba30b4 ?erase@?$_Tree@V?$_Tmap_traits@UGlobalRequestID@content@@V?$unique_ptr@VResourceLoader@content@@U?$default_delete@VResourceLoader@content@@@std@@@std@@U?$less@UGlobalRequestID@content@@@4@V?$allocator@U?$pair@$$CBUGlobalRequestID@content@@V?$unique_ptr@VResourceLoader@content@@U?$default_delete@VResourceLoader@content@@@std@@@std@@@std@@@4@$0A@@std@@@std@@QAE?AV?$_Tree_iterator@V?$_Tree_val@U?$_Tree_simple_types@U?$pair@$$CBUGlobalRequestID@content@@V?$unique_ptr@VResourceLoader@content@@U?$default_delete@VResourceLoader@content@@@std@@@std@@@std@@@std@@@std@@@2@V?$_Tree_const_iterator@V?$_Tree_val@U?$_Tree_simple_types@U?$pair@$$CBUGlobalRequestID@content@@V?$unique_ptr@VResourceLoader@content@@U?$default_delete@VResourceLoader@content@@@std@@@std@@@std@@@std@@@std@@@2@@Z 0b 0662efd4 5fba80d5 chrome_5f000000!ovly_debug_event+0x360471 10ba807a ?RemovePendingLoader@ResourceDispatcherHostImpl@content@@AAEXABV?$_Tree_iterator@V?$_Tree_val@U?$_Tree_simple_types@U?$pair@$$CBUGlobalRequestID@content@@V?$unique_ptr@VResourceLoader@content@@U?$default_delete@VResourceLoader@content@@@std@@@std@@@std@@@std@@@std@@@std@@@Z 0c 0662f000 5fba48e2 chrome_5f000000!ovly_debug_event+0x36525d 10ba48a4 ?RemovePendingRequest@ResourceDispatcherHostImpl@content@@QAEXHH@Z 0d 0662f028 5fba46eb chrome_5f000000!ovly_debug_event+0x361a6a 10ba3fa6 ?DidFinishLoading@ResourceDispatcherHostImpl@content@@EAEXPAVResourceLoader@2@@Z 0e 0662f08c 5fbae539 chrome_5f000000!ovly_debug_event+0x361873 10bae488 ?CallDidFinishLoading@ResourceLoader@content@@AAEXXZ 0f 0662f0d8 5fbadab6 chrome_5f000000!ovly_debug_event+0x36b6c1 10bad9a0 ?ResponseCompleted@ResourceLoader@content@@AAEXXZ 10 0662f138 5fbadf6d chrome_5f000000!ovly_debug_event+0x36ac3e 10bade24 ?OnReadCompleted@ResourceLoader@content@@EAEXPAVURLRequest@net@@H@Z 11 0662f1a0 5fbaddfe chrome_5f000000!ovly_debug_event+0x36b0f5 10badbf2 ?ReadMore@ResourceLoader@content@@AAEX_N@Z 12 0662f21c 5fbadf15 chrome_5f000000!ovly_debug_event+0x36af86 10bade24 ?OnReadCompleted@ResourceLoader@content@@EAEXPAVURLRequest@net@@H@Z 13 0662f284 5ff0d1bc chrome_5f000000!ovly_debug_event+0x36b09d 10f0d168 ?NotifyReadCompleted@URLRequest@net@@AAEXH@Z 14 0662f29c 5ff0e0f0 chrome_5f000000!ovly_debug_event+0x6ca344 10f0e058 ?SourceStreamReadComplete@URLRequestJob@net@@AAEX_NH@Z 15 0662f2c0 5ff0f0ec chrome_5f000000!ovly_debug_event+0x6cb278 10f0f0ae ?Run@?$Invoker@U?$BindState@P8URLRequestJob@net@@AEX_NH@ZV?$WeakPtr@VURLRequestJob@net@@@base@@_N@internal@base@@$$A6AXH@Z@internal@base@@SAXPAVBindStateBase@23@$$QAH@Z 16 0662f2dc 601ddd88 chrome_5f000000!ovly_debug_event+0x6cc274 111ddd34 ?OnIOComplete@FilterSourceStream@net@@AAEXH@Z 17 0662f304 5fa396e1 chrome_5f000000!IsSandboxedProcess+0x216898 10a396d0 ?Run@?$Invoker@U?$BindState@P8Writer@RedirectToFileResourceHandler@content@@AEXH@ZV?$UnretainedWrapper@VWriter@RedirectToFileResourceHandler@content@@@internal@base@@@internal@base@@$$A6AXH@Z@internal@base@@SAXPAVBindStateBase@23@$$QAH@Z (Lots of ICF'd functions) 18 0662f310 5ff0ee2c chrome_5f000000!ovly_debug_event+0x1f6869 10f0edc4 ?ReadRawDataComplete@URLRequestJob@net@@IAEXH@Z 19 0662f35c 605bccc5 chrome_5f000000!ovly_debug_event+0x6cbfb4 115bcbf0 ?OnReadCompleted@URLRequestHttpJob@net@@AAEXH@Z 1a 0662f3a4 5fa396e1 chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1e4568 10a396d0 ?Run@?$Invoker@U?$BindState@P8Writer@RedirectToFileResourceHandler@content@@AEXH@ZV?$UnretainedWrapper@VWriter@RedirectToFileResourceHandler@content@@@internal@base@@@internal@base@@$$A6AXH@Z@internal@base@@SAXPAVBindStateBase@23@$$QAH@Z (Lots of ICF'd functions) 1b 0662f3b0 601de759 chrome_5f000000!ovly_debug_event+0x1f6869 111de45a ?DoLoop@Transaction@HttpCache@net@@AAEHH@Z 1c 0662f3d8 5f9ce196 chrome_5f000000!IsSandboxedProcess+0x217269 109ce160 ?Run@?$Invoker@U?$BindState@P8ViewCacheHelper@net@@AEXH@ZV?$WeakPtr@VViewCacheHelper@net@@@base@@@internal@base@@$$A6AXH@Z@internal@base@@SAXPAVBindStateBase@23@$$QAH@Z (Lots of ICF'd functions) 1d 0662f3f0 6057cc0b chrome_5f000000!ovly_debug_event+0x18b31e 1157cb84 ?OnOperationComplete@InFlightBackendIO@disk_cache@@MAEXPAVBackgroundIO@2@_N@Z 1e 0662f41c 60586388 chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1a44ae 11586302 ?InvokeCallback@InFlightIO@disk_cache@@QAEXPAVBackgroundIO@2@_N@Z 1f 0662f440 605862de chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1adc2b 11586202 ?OnIOSignalled@BackgroundIO@disk_cache@@QAEXXZ 20 0662f4a4 61ec3f47 chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1adb81 12ec3d30 ?RunTask@TaskAnnotator@debug@base@@QAEXPBDPAUPendingTask@3@@Z 21 0662f530 61d4daae chrome_5f000000!GetHandleVerifier+0x105567 12d4d870 ?RunTask@MessageLoop@base@@QAEXPAUPendingTask@2@@Z 22 0662f5c4 61d4e346 chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1975351 12d4e320 ?DeferOrRunPendingTask@MessageLoop@base@@AAE_NUPendingTask@2@@Z 23 0662f5d8 61d4d5ab chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1975be9 12d4d350 ?DoWork@MessageLoop@base@@EAE_NXZ 24 0662f688 61ec7609 chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1974e4e 12ec74b0 ?DoRunLoop@MessagePumpForIO@base@@EAEXXZ 25 0662f6d0 61ec6abc chrome_5f000000!GetHandleVerifier+0x108c29 12ec69d0 ?CalculateDelayedRuntime@?A@internal@base@@YA?AVTimeTicks@2@VTimeDelta@2@@Z 26 0662f704 61d6ddee chrome_5f000000!GetHandleVerifier+0x1080dc 12d6daf0 ?Run@RunLoop@base@@QAEXXZ 27 0662f778 61d85fcb chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1995691 12d85fc0 ?Run@Thread@base@@MAEXPAVRunLoop@2@@Z 28 0662f780 5fa9e7ed chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19ad86e 10a9e7ce ?IOThreadRun@BrowserThreadImpl@content@@AAEXPAVRunLoop@base@@@Z 29 0662f794 5fa9e888 chrome_5f000000!ovly_debug_event+0x25b975 10a9e806 ?Run@BrowserThreadImpl@content@@MAEXPAVRunLoop@base@@@Z 2a 0662f7ac 61d85eb0 chrome_5f000000!ovly_debug_event+0x25ba10 12d85d40 ?ThreadMain@Thread@base@@EAEXXZ 2b 0662f7f8 61d8039c chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19ad753 12d802f0 ?ThreadFunc@?A@base@@YGKPAX@Z 2c 0662f81c 758e336a chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19a7c3f 2d 0662f828 778d9902 kernel32!BaseThreadInitThunk+0xe 2e 0662f868 778d98d5 ntdll32!__RtlUserThreadStart+0x70 2f 0662f880 00000000 ntdll32!_RtlUserThreadStart+0x1b This was an interesting exercise, but maybe not very helpful as it seems like we're crashing due to heap curruption :-/
,
Dec 14 2016
I thought about why I'm getting crashes while inglorion's builds seem to work fine, and realized I'm doing 32-bit and he's doing 64-bit builds. rnk pointed out that 32-bit only often implicates inalloca Looking at the stack, the first function with inalloca calls is ResourceDispatcherHostImpl::RemovePendingLoader When I commented out the body of that function, I appear to get further in loading a page, but then crash somewhere else. I didn't symbolize the full stack, but now we're reaching HeapFree via ?reset@?$unique_ptr@VWatcher@mojo@@U?$default_delete@VWatcher@mojo@@@std@@@std@@QAEXPAVWatcher@mojo@@@Z so this is from a whole different place :-/ And I don't see any inalloca calls. Hmm.
,
Sep 19 2017
Is this fixed now?
,
Sep 19 2017
It's very old, let's archive it. The bot seems to be red for other reasons these days. |
|||
►
Sign in to add a comment |
|||
Comment 1 by ajha@chromium.org
, Dec 1 2016