New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 669940 link

Starred by 2 users

Issue metadata

Status: Archived
Owner: ----
Closed: Sep 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

CrWinClangLLD tester bot failing on larger tests, probably due to LTO

Project Member Reported by r...@chromium.org, Nov 30 2016

Issue description

The CrWinClangLLD builder bot finally went green after the inline asm issue ( bug 666994 ), so now we get to run the tests, and it turns out they fail. This is the first real test run since we enabled LTO (last green Nov 11), so I'm guessing LTO is the culprit.

It looks like any test binary large enough to link in the browser parts of chrome is failing somewhere during browser startup. Because we don't have PDBs yet, the stack traces are pretty unhelpful.

Example build:
https://build.chromium.org/p/chromium.fyi/builders/CrWinClangLLD%20tester/builds/1250
 

Comment 1 by ajha@chromium.org, Dec 1 2016

Components: Build

Comment 2 by h...@chromium.org, Dec 1 2016

I can repro locally:

D:\src\chromium\src>out\lto\browser_tests --gtest_filter=BrowserTest.GetSizeForN
ewRenderView
IMPORTANT DEBUGGING NOTE: each test is run inside its own process.
For debugging a test inside a debugger, use the
--gtest_filter=<your_test_name> flag along with either
--single_process (to run the test in one launcher/browser process) or
--single-process (to do the above, and also run Chrome in single-process mode).
Using sharding settings from environment. This is shard 0/1
Using 1 parallel jobs.
Note: Google Test filter = BrowserTest.GetSizeForNewRenderView
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from BrowserTest, where TypeParam =
[ RUN      ] BrowserTest.GetSizeForNewRenderView
[6496:9312:1201/141743.779:ERROR:configuration_policy_handler_list.cc(92)] Unkno
wn policy: EnableMemoryInfo
[6496:9312:1201/141743.780:ERROR:configuration_policy_handler_list.cc(92)] Unkno
wn policy: RemoteAccessClientFirewallTraversal
[6496:9312:1201/141743.781:ERROR:configuration_policy_handler_list.cc(92)] Unkno
wn policy: RemoteAccessHostMatchUsername
[6496:9312:1201/141743.878:ERROR:configuration_policy_handler_list.cc(92)] Unkno
wn policy: EnableMemoryInfo
[6496:9312:1201/141743.878:ERROR:configuration_policy_handler_list.cc(92)] Unkno
wn policy: RemoteAccessClientFirewallTraversal
[6496:9312:1201/141743.879:ERROR:configuration_policy_handler_list.cc(92)] Unkno
wn policy: RemoteAccessHostMatchUsername
Backtrace:
        RtlInitUnicodeString [0x778CE43E+406]
        RtlFreeHeap [0x778CE0A3+126]
        HeapFree [0x758E14AD+20]
        nacl_global_xlate_base [0x028CCD93+9949455]
        nacl_global_xlate_base [0x028CCD3F+9949371]
        nacl_global_xlate_base [0x028CCCB8+9949236]
        InjectDumpForHangDebugging [0x05E2B94D+16707279]
        nacl_global_xlate_base [0x03259744+19962560]
        nacl_global_xlate_base [0x0325327C+19936760]
        nacl_global_xlate_base [0x03252CFF+19935355]
        nacl_global_xlate_base [0x03252CCE+19935306]
        nacl_global_xlate_base [0x03257110+19952780]
        InjectDumpForHangDebugging [0x050F41BC+2848574]
        IsSandboxedProcess [0x033AAFF9+9421]
        nacl_global_xlate_base [0x02C532D9+13645397]
        IsSandboxedProcess [0x03ACF0B3+7497095]
        IsSandboxedProcess [0x03ACF079+7497037]
        nacl_global_xlate_base [0x02B6A119+12690581]
        GetHandleVerifier [0x063DEF27+5502599]
        InjectDumpForHangDebugging [0x05E2C95E+16711392]
        InjectDumpForHangDebugging [0x05E2D1F6+16713592]
        InjectDumpForHangDebugging [0x05E2C44B+16710093]
        GetHandleVerifier [0x063E1469+5512137]
        GetHandleVerifier [0x063E091C+5509244]
        InjectDumpForHangDebugging [0x05E4D62E+16845744]
        InjectDumpForHangDebugging [0x05E6661B+16948125]
        nacl_global_xlate_base [0x02BD1BD3+13115215]
        nacl_global_xlate_base [0x02BD1C6E+13115370]
        InjectDumpForHangDebugging [0x05E66500+16947842]
        InjectDumpForHangDebugging [0x05E5FF15+16921751]
        BaseThreadInitThunk [0x758E336A+18]
        RtlInitializeExceptionChain [0x778D9902+99]
        RtlInitializeExceptionChain [0x778D98D5+54]
[1/1] BrowserTest.GetSizeForNewRenderView (2472 ms)
1 test failed:
    BrowserTest.GetSizeForNewRenderView (../../chrome/browser/ui/browser_browser
test.cc:2564)


chrome.exe is also broken; it crashes when trying to navigate somewhere (it survives startup). This used to work, so it must be a regression.

Comment 3 by h...@chromium.org, Dec 1 2016

Status: Available (was: Untriaged)
content_shell.exe also doesn't work, but there only the renderer crashes, not the main process.

Trying chrome.exe --single-process http://www.google.com/

running under windbg, it seems to crash here:

(29e0.2f5c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
ntdll32!RtlpLowFragHeapFree+0x31:
778ce43e 8b4604          mov     eax,dword ptr [esi+4] ds:002b:34fc7794=????????
0:020:x86> k
 # ChildEBP RetAddr  
00 067af32c 778ce0a3 ntdll32!RtlpLowFragHeapFree+0x31
01 067af344 758e14ad ntdll32!RtlFreeHeap+0x105
02 067af358 5f97dfff kernel32!HeapFree+0x14
WARNING: Stack unwind information not available. Following frames may be wrong.
03 067af370 5f97dfab chrome_5f130000!ChromeMain+0x2f7b
04 067af37c 5f97df24 chrome_5f130000!ChromeMain+0x2f27
05 067af38c 60f94ded chrome_5f130000!ChromeMain+0x2ea0
06 067af39c 5fccc153 chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0xaa2404
07 067af3a8 5fccc0dd chrome_5f130000!ChromeMain+0x3510cf
08 067af3b4 5fcc80bb chrome_5f130000!ChromeMain+0x351059
09 067af3d0 5fcc7fc7 chrome_5f130000!ChromeMain+0x34d037
0a 067af3dc 5fcbcd73 chrome_5f130000!ChromeMain+0x34cf43
0b 067af3fc 5fcc1ba7 chrome_5f130000!ChromeMain+0x341cef
0c 067af428 5fcbe36c chrome_5f130000!ChromeMain+0x346b23
0d 067af450 5fcbe175 chrome_5f130000!ChromeMain+0x3432e8
0e 067af4b4 5fcc7b41 chrome_5f130000!ChromeMain+0x3430f1
0f 067af500 5fcc71d6 chrome_5f130000!ChromeMain+0x34cabd
10 067af560 5fcc7575 chrome_5f130000!ChromeMain+0x34c152
11 067af5c4 5fcc741a chrome_5f130000!ChromeMain+0x34c4f1
12 067af60c 5fcc751d chrome_5f130000!ChromeMain+0x34c396
13 067af674 5fcb453e chrome_5f130000!ChromeMain+0x34c499
14 067af694 62012867 chrome_5f130000!ChromeMain+0x3394ba
15 067af720 61eaae6e chrome_5f130000!GetHandleVerifier+0xf8e27
16 067af7b4 61eab706 chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19b8485
17 067af7c8 61eaa96b chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19b8d1d
18 067af878 62015f29 chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19b7f82
19 067af8c0 620153dc chrome_5f130000!GetHandleVerifier+0xfc4e9
1a 067af8f4 61eca38e chrome_5f130000!GetHandleVerifier+0xfb99c
1b 067af960 61ee24db chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19d79a5
1c 067af968 5fbbad49 chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19efaf2
1d 067af97c 5fbbade4 chrome_5f130000!ChromeMain+0x23fcc5
1e 067af994 61ee23c0 chrome_5f130000!ChromeMain+0x23fd60
1f 067af9e0 61edc8bc chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19ef9d7
20 067afa04 758e336a chrome_5f130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19e9ed3
21 067afa10 778d9902 kernel32!BaseThreadInitThunk+0xe
22 067afa50 778d98d5 ntdll32!__RtlUserThreadStart+0x70
23 067afa68 00000000 ntdll32!_RtlUserThreadStart+0x1b

Comment 4 by h...@chromium.org, Dec 9 2016

Build chrome again, but this time saved the map file for chrome.dll to symbolize the stack manually.

Below is the stack, where I've looked up the symbol for the return address from each frame in the map file:

 # ChildEBP RetAddr  
00 0662ef04 778ce0a3 ntdll32!RtlpLowFragHeapFree+0x31
01 0662ef1c 758e14ad ntdll32!RtlFreeHeap+0x105

02 0662ef30 5f84241b kernel32!HeapFree+0x14
WARNING: Stack unwind information not available. Following frames may be wrong.
10842401 ?WinHeapFree@allocator@base@@YAXPAX@Z

03 0662ef48 5f8423c7 chrome_5f000000!ChromeMain+0x3397
108423bc ?DefaultWinHeapFreeImpl@?A@@YAXPBUAllocatorDispatch@allocator@base@@PAX@Z

04 0662ef54 5f842307 chrome_5f000000!ChromeMain+0x3343
108422f8 _free

05 0662ef64 5f861585 chrome_5f000000!ChromeMain+0x3283
10861542 ?_Tidy@?$vector@W4__MIDL___MIDL_itf_inputscope_0000_0000_0001@@V?$allocator@W4__MIDL___MIDL_itf_inputscope_0000_0000_0001@@@std@@@std@@IAEXXZ
(lots of functions ICF'd to that address)

06 0662ef74 5fbb2b1b chrome_5f000000!ovly_debug_event+0x1e70d
10bb2ae2 ??1ThrottlingResourceHandler@content@@UAE@XZ

07 0662ef80 5fbb2aa5 chrome_5f000000!ovly_debug_event+0x36fca3
10bb2a9a ??_GThrottlingResourceHandler@content@@UAEPAXI@Z

08 0662ef8c 5fbae973 chrome_5f000000!ovly_debug_event+0x36fc2d
10bae908 ??1ResourceLoader@content@@UAE@XZ

09 0662efa8 5fbae87f chrome_5f000000!ovly_debug_event+0x36bafb
10bae874 ??_GResourceLoader@content@@UAEPAXI@Z

0a 0662efb4 5fba32e9 chrome_5f000000!ovly_debug_event+0x36ba07
10ba30b4 ?erase@?$_Tree@V?$_Tmap_traits@UGlobalRequestID@content@@V?$unique_ptr@VResourceLoader@content@@U?$default_delete@VResourceLoader@content@@@std@@@std@@U?$less@UGlobalRequestID@content@@@4@V?$allocator@U?$pair@$$CBUGlobalRequestID@content@@V?$unique_ptr@VResourceLoader@content@@U?$default_delete@VResourceLoader@content@@@std@@@std@@@std@@@4@$0A@@std@@@std@@QAE?AV?$_Tree_iterator@V?$_Tree_val@U?$_Tree_simple_types@U?$pair@$$CBUGlobalRequestID@content@@V?$unique_ptr@VResourceLoader@content@@U?$default_delete@VResourceLoader@content@@@std@@@std@@@std@@@std@@@std@@@2@V?$_Tree_const_iterator@V?$_Tree_val@U?$_Tree_simple_types@U?$pair@$$CBUGlobalRequestID@content@@V?$unique_ptr@VResourceLoader@content@@U?$default_delete@VResourceLoader@content@@@std@@@std@@@std@@@std@@@std@@@2@@Z

0b 0662efd4 5fba80d5 chrome_5f000000!ovly_debug_event+0x360471
10ba807a ?RemovePendingLoader@ResourceDispatcherHostImpl@content@@AAEXABV?$_Tree_iterator@V?$_Tree_val@U?$_Tree_simple_types@U?$pair@$$CBUGlobalRequestID@content@@V?$unique_ptr@VResourceLoader@content@@U?$default_delete@VResourceLoader@content@@@std@@@std@@@std@@@std@@@std@@@std@@@Z

0c 0662f000 5fba48e2 chrome_5f000000!ovly_debug_event+0x36525d
10ba48a4 ?RemovePendingRequest@ResourceDispatcherHostImpl@content@@QAEXHH@Z

0d 0662f028 5fba46eb chrome_5f000000!ovly_debug_event+0x361a6a
10ba3fa6 ?DidFinishLoading@ResourceDispatcherHostImpl@content@@EAEXPAVResourceLoader@2@@Z

0e 0662f08c 5fbae539 chrome_5f000000!ovly_debug_event+0x361873
10bae488 ?CallDidFinishLoading@ResourceLoader@content@@AAEXXZ

0f 0662f0d8 5fbadab6 chrome_5f000000!ovly_debug_event+0x36b6c1
10bad9a0 ?ResponseCompleted@ResourceLoader@content@@AAEXXZ

10 0662f138 5fbadf6d chrome_5f000000!ovly_debug_event+0x36ac3e
10bade24 ?OnReadCompleted@ResourceLoader@content@@EAEXPAVURLRequest@net@@H@Z

11 0662f1a0 5fbaddfe chrome_5f000000!ovly_debug_event+0x36b0f5
10badbf2 ?ReadMore@ResourceLoader@content@@AAEX_N@Z

12 0662f21c 5fbadf15 chrome_5f000000!ovly_debug_event+0x36af86
10bade24 ?OnReadCompleted@ResourceLoader@content@@EAEXPAVURLRequest@net@@H@Z

13 0662f284 5ff0d1bc chrome_5f000000!ovly_debug_event+0x36b09d
10f0d168 ?NotifyReadCompleted@URLRequest@net@@AAEXH@Z

14 0662f29c 5ff0e0f0 chrome_5f000000!ovly_debug_event+0x6ca344
10f0e058 ?SourceStreamReadComplete@URLRequestJob@net@@AAEX_NH@Z

15 0662f2c0 5ff0f0ec chrome_5f000000!ovly_debug_event+0x6cb278
10f0f0ae ?Run@?$Invoker@U?$BindState@P8URLRequestJob@net@@AEX_NH@ZV?$WeakPtr@VURLRequestJob@net@@@base@@_N@internal@base@@$$A6AXH@Z@internal@base@@SAXPAVBindStateBase@23@$$QAH@Z

16 0662f2dc 601ddd88 chrome_5f000000!ovly_debug_event+0x6cc274
111ddd34 ?OnIOComplete@FilterSourceStream@net@@AAEXH@Z

17 0662f304 5fa396e1 chrome_5f000000!IsSandboxedProcess+0x216898
10a396d0 ?Run@?$Invoker@U?$BindState@P8Writer@RedirectToFileResourceHandler@content@@AEXH@ZV?$UnretainedWrapper@VWriter@RedirectToFileResourceHandler@content@@@internal@base@@@internal@base@@$$A6AXH@Z@internal@base@@SAXPAVBindStateBase@23@$$QAH@Z
(Lots of ICF'd functions)

18 0662f310 5ff0ee2c chrome_5f000000!ovly_debug_event+0x1f6869
10f0edc4 ?ReadRawDataComplete@URLRequestJob@net@@IAEXH@Z

19 0662f35c 605bccc5 chrome_5f000000!ovly_debug_event+0x6cbfb4
115bcbf0 ?OnReadCompleted@URLRequestHttpJob@net@@AAEXH@Z

1a 0662f3a4 5fa396e1 chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1e4568
10a396d0 ?Run@?$Invoker@U?$BindState@P8Writer@RedirectToFileResourceHandler@content@@AEXH@ZV?$UnretainedWrapper@VWriter@RedirectToFileResourceHandler@content@@@internal@base@@@internal@base@@$$A6AXH@Z@internal@base@@SAXPAVBindStateBase@23@$$QAH@Z
(Lots of ICF'd functions)

1b 0662f3b0 601de759 chrome_5f000000!ovly_debug_event+0x1f6869
111de45a ?DoLoop@Transaction@HttpCache@net@@AAEHH@Z

1c 0662f3d8 5f9ce196 chrome_5f000000!IsSandboxedProcess+0x217269
109ce160 ?Run@?$Invoker@U?$BindState@P8ViewCacheHelper@net@@AEXH@ZV?$WeakPtr@VViewCacheHelper@net@@@base@@@internal@base@@$$A6AXH@Z@internal@base@@SAXPAVBindStateBase@23@$$QAH@Z
(Lots of ICF'd functions)

1d 0662f3f0 6057cc0b chrome_5f000000!ovly_debug_event+0x18b31e
1157cb84 ?OnOperationComplete@InFlightBackendIO@disk_cache@@MAEXPAVBackgroundIO@2@_N@Z

1e 0662f41c 60586388 chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1a44ae
11586302 ?InvokeCallback@InFlightIO@disk_cache@@QAEXPAVBackgroundIO@2@_N@Z

1f 0662f440 605862de chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1adc2b
11586202 ?OnIOSignalled@BackgroundIO@disk_cache@@QAEXXZ

20 0662f4a4 61ec3f47 chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1adb81
12ec3d30 ?RunTask@TaskAnnotator@debug@base@@QAEXPBDPAUPendingTask@3@@Z

21 0662f530 61d4daae chrome_5f000000!GetHandleVerifier+0x105567
12d4d870 ?RunTask@MessageLoop@base@@QAEXPAUPendingTask@2@@Z

22 0662f5c4 61d4e346 chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1975351
12d4e320 ?DeferOrRunPendingTask@MessageLoop@base@@AAE_NUPendingTask@2@@Z

23 0662f5d8 61d4d5ab chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1975be9
12d4d350 ?DoWork@MessageLoop@base@@EAE_NXZ

24 0662f688 61ec7609 chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1974e4e
12ec74b0 ?DoRunLoop@MessagePumpForIO@base@@EAEXXZ

25 0662f6d0 61ec6abc chrome_5f000000!GetHandleVerifier+0x108c29
12ec69d0 ?CalculateDelayedRuntime@?A@internal@base@@YA?AVTimeTicks@2@VTimeDelta@2@@Z

26 0662f704 61d6ddee chrome_5f000000!GetHandleVerifier+0x1080dc
12d6daf0 ?Run@RunLoop@base@@QAEXXZ

27 0662f778 61d85fcb chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x1995691
12d85fc0 ?Run@Thread@base@@MAEXPAVRunLoop@2@@Z

28 0662f780 5fa9e7ed chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19ad86e
10a9e7ce ?IOThreadRun@BrowserThreadImpl@content@@AAEXPAVRunLoop@base@@@Z

29 0662f794 5fa9e888 chrome_5f000000!ovly_debug_event+0x25b975
10a9e806 ?Run@BrowserThreadImpl@content@@MAEXPAVRunLoop@base@@@Z

2a 0662f7ac 61d85eb0 chrome_5f000000!ovly_debug_event+0x25ba10
12d85d40 ?ThreadMain@Thread@base@@EAEXXZ

2b 0662f7f8 61d8039c chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19ad753
12d802f0 ?ThreadFunc@?A@base@@YGKPAX@Z

2c 0662f81c 758e336a chrome_5f000000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x19a7c3f
2d 0662f828 778d9902 kernel32!BaseThreadInitThunk+0xe
2e 0662f868 778d98d5 ntdll32!__RtlUserThreadStart+0x70
2f 0662f880 00000000 ntdll32!_RtlUserThreadStart+0x1b


This was an interesting exercise, but maybe not very helpful as it seems like we're crashing due to heap curruption :-/

Comment 5 by h...@chromium.org, Dec 14 2016

I thought about why I'm getting crashes while inglorion's builds seem to work fine, and realized I'm doing 32-bit and he's doing 64-bit builds.

rnk pointed out that 32-bit only often implicates inalloca

Looking at the stack, the first function with inalloca calls is ResourceDispatcherHostImpl::RemovePendingLoader

When I commented out the body of that function, I appear to get further in loading a page, but then crash somewhere else.

I didn't symbolize the full stack, but now we're reaching HeapFree via
?reset@?$unique_ptr@VWatcher@mojo@@U?$default_delete@VWatcher@mojo@@@std@@@std@@QAEXPAVWatcher@mojo@@@Z
so this is from a whole different place :-/ And I don't see any inalloca calls. Hmm.
Is this fixed now?

Comment 7 by h...@chromium.org, Sep 19 2017

Status: Archived (was: Available)
It's very old, let's archive it. The bot seems to be red for other reasons these days.

Sign in to add a comment