New issue
Advanced search Search tips

Issue 873112 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Aug 14
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocked on:
issue 873139



Sign in to add a comment

memory.desktop/Trivial* pages failing flakily on win-10-perf

Project Member Reported by nednguyen@chromium.org, Aug 10

Issue description

Thanks ned. Looks like the same stack trace for both:

"""
	.  0  Id: 158c.1af8 Suspend: 0 Teb: 00000057`230b3000 Unfrozen
	RetAddr           : Args to Child                                                           : Call Site
	00007ffa`61168e1a : 00000057`233fd798 00000000`00000028 00000198`8211d0e0 00000000`00000028 : ntdll!ZwDelayExecution+0x14
	*** WARNING: Unable to verify checksum for chrome_elf.dll
	00007ffa`4cb12f1f : 00000000`00000000 00000000`00000000 00000000`c00000bb 00000000`00000000 : KERNELBASE!SleepEx+0x9a
	00007ffa`611f95a0 : 00000000`00000000 00000057`233ffdc0 00000000`00000000 00007ffa`302ef75f : chrome_elf!crashpad::`anonymous namespace'::UnhandledExceptionHandler+0xdd
	00007ffa`64b48548 : 00007ff6`af7d69a0 00007ffa`64bde640 00000000`00000000 00000000`00000000 : KERNELBASE!UnhandledExceptionFilter+0x190
	00007ffa`64b30f96 : 00000000`00000000 00007ffa`64b0f13f 00000057`233fe6b0 00000000`00000000 : ntdll!memset+0x1b08
	00007ffa`64b44c7d : 00000000`00000000 00000057`233fdbb0 00000000`00000000 00000057`233fdbb0 : ntdll!_C_specific_handler+0x96
	00007ffa`64abd1d8 : 00000000`00000000 00000000`00000000 0000717f`c5b53500 00000198`00000002 : ntdll!_chkstk+0x11d
	00007ffa`64b43bae : 00000198`ffe1f930 00007ffa`3023bdd4 00007ffa`3364a4a8 00000057`233fe508 : ntdll!RtlWalkFrameChain+0x1518
	00007ffa`30f1682c : 00000000`00000000 00007ffa`30243a1c 00000000`00000000 00000005`1563bf16 : ntdll!KiUserExceptionDispatcher+0x2e
	00007ffa`30243a1c : 00000000`00000000 00000005`1563bf16 00000198`82210f70 efefefef`efefefef : chrome!content::protocol::SystemInfoHandlerGpuObserver::ObserverWatchdogCallback+0x4
	00007ffa`30243417 : 00000198`8227b6f0 00007ffa`302c49e1 00000000`00000000 00000000`000001a4 : chrome!base::debug::TaskAnnotator::RunTask+0x12c
	00007ffa`3023cdf7 : 00000000`00000000 00007ffa`333dbe88 00007ffa`64311a90 00007ffa`3145084f : chrome!base::MessageLoop::RunTask+0x247
	00007ffa`30336952 : 00000000`00000000 00000000`00000000 00000198`f96a0e00 00000057`233fee30 : chrome!base::MessageLoop::DoDelayedWork+0x177
	00007ffa`3023c77e : 00000198`f9688540 00007ffa`335f16d0 00000057`233feea8 00000000`000001f4 : chrome!base::MessagePumpForUI::DoRunLoop+0xc2
	00007ffa`3023c4e1 : 0000717f`c5b5015b 00000198`f969d4b0 0000717f`c5b5018b 00000198`f96b33e8 : chrome!base::MessagePumpWin::Run+0x4e
	00007ffa`305f23b0 : 00000000`00000692 00000198`803c51b0 00000057`233ff738 00007ffa`302326df : chrome!base::RunLoop::Run+0x31
	00007ffa`305f21c2 : 00000000`00000000 00000198`f92c0cc0 00000000`000003e8 00000000`00989680 : chrome!ChromeBrowserMainParts::MainMessageLoopRun+0x84
	00007ffa`305f216d : 00000198`f96b2e10 00000057`233ff210 00000000`00000003 00000057`233ff738 : chrome!content::BrowserMainLoop::RunMainMessageLoopParts+0x48
	00007ffa`30e9fe0e : 00000198`f96a8420 00007ffa`30221536 00000000`ffffffff 00007ffa`302387d8 : chrome!content::BrowserMainRunnerImpl::Run+0x11
	00007ffa`313e0b97 : 00000000`00000000 0000717f`c5b51e4b 00000198`f969dd20 00000198`f96b3240 : chrome!content::BrowserMain+0xc6
	00007ffa`313e0e9e : 0000717f`c5b51e1b 00000198`f97c8080 00000057`233ff418 00000057`233ff2d0 : chrome!content::RunBrowserProcessMain+0x6f
	00007ffa`30225978 : 00000000`00000004 00007ffa`32dad96b 00000198`f97c0300 00000198`f969d330 : chrome!content::ContentMainRunnerImpl::Run+0x25e
	00007ffa`30225578 : 00000057`233ff908 00007ffa`30222b6b 00000000`00004e1a 00007ffa`3022123b : chrome!service_manager::Main+0x333
	00007ffa`30221d6a : 00000057`233ff7c0 00007ff6`af6c0000 00000057`233ff8e0 00000057`233ff830 : chrome!content::ContentMain+0x41
	00007ff6`af6c376c : 00000198`f91e1100 00007ffa`30221c52 00000057`233ff908 00000198`f91e10f0 : chrome!ChromeMain+0x118
	00007ff6`af6c1699 : 00000000`00000000 ffffffff`fffffffe 00000000`00000000 00002ed2`df7823e1 : chrome_exe!MainDllLoader::Launch+0x26c
	00007ff6`af784f22 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : chrome_exe!wWinMain+0x699
	00007ffa`62ac1fe4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : chrome_exe!__scrt_common_main_seh+0x106
	00007ffa`64b0efc1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x14
	00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21
"""
Okay, there are a bunch of problems here:

catapult:
1) We're trying to make the browser 6 times. [even though retry = 3]. The first 5 crash, but we don't have symbolized crash dumps. The last one suffers a timeout, which triggers a CHECK(false), which is symbolized.

https://chrome-isolated.appspot.com/browse?namespace=default-gzip&digest=5d551e2c2df2ff3e61eab751bd20f5e10bdb019e&as=memory.desktop%5Cbenchmark_log.txt

Search for "Browser creation failed".

2) We're missing the browser logs. See https://www.chromium.org/for-testers/enable-logging for more details on how to get the logs.

chrome:
3) SystemInfoHandlerGpuObserver has a 5s timeout for GPU process startup/information gathering. This is too short. EVen successful callbacks are taking ~3s.

example:
"""
(INFO) 2018-08-09 21:07:10,072 browser._LogBrowserInfo:100  Browser started (pid=1120).
(INFO) 2018-08-09 21:07:10,072 browser._LogBrowserInfo:103  OS: win win10
(INFO) 2018-08-09 21:07:10,072 browser._LogBrowserInfo:106  Detailed OS version: 10.0.16299
(INFO) 2018-08-09 21:07:13,713 browser._LogBrowserInfo:112  Browser command line: [emitted for brevity]
(INFO) 2018-08-09 21:07:13,713 browser._LogBrowserInfo:115  GPU device 0: VENDOR = 0x8086 (Google Inc.), DEVICE = 0x5912 (ANGLE (Intel(R) HD Graphics 630 Direct3D11 vs_5_0 ps_5_0))
(INFO) 2018-08-09 21:07:13,713 browser._LogBrowserInfo:117  GPU Attributes:
...
"""

Notice the 3s delay before browser._LogBrowserInfo is able to log GPU info. This is because "system_info = self.GetSystemInfo()" takes 3s to run.
Blockedon: 873139
Filed issue 873139 about browser logging
Status: Started (was: Untriaged)
Project Member

Comment 6 by bugdroid1@chromium.org, Aug 10

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1c9cf51bc70b424ce8d9849d24c115a7f6930772

commit 1c9cf51bc70b424ce8d9849d24c115a7f6930772
Author: Erik Chen <erikchen@chromium.org>
Date: Fri Aug 10 22:55:52 2018

Increase DevTools SystemInfoGPU timeout for Windows.

Before this CL, DevTools would crash the browser if it did not get a response in
5 seconds. This timeout is too low for Windows. On Catapult perf tests on
Windows, it takes ~3 seconds for the GPU process to respond on average, and
sometimes it takes more than 5 seconds.

Bug:  873112 
Change-Id: I67ee82a1a75df0ba9e1b483a58c66c56a8bf2213
Reviewed-on: https://chromium-review.googlesource.com/1170584
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Andrey Kosyakov <caseq@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#582375}
[modify] https://crrev.com/1c9cf51bc70b424ce8d9849d24c115a7f6930772/content/browser/devtools/protocol/system_info_handler.cc

Status: Fixed (was: Started)
Is htis still a problem? Closing for now -- reopen if issue recurs.

Sign in to add a comment