New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 638384 link

Starred by 1 user
Status: Duplicate
Merged: issue 636489
Owner:
vmi...@chromium.org
Closed: Aug 2016
Cc:
█ kbr@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 2
Type: Bug



Sign in to add a comment

gpu_process_launch_tests flaky on Linux Debug (New Intel) gpu.fyi bot

Project Member Reported by ynovikov@chromium.org, Aug 16 2016 
https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Debug%20%28New%20Intel%29/builds/3439 - GpuProcess.css3d (*)
https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Debug%20%28New%20Intel%29/builds/3436 - GpuProcess.identify_active_gpu2
https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Debug%20%28New%20Intel%29/builds/3435 - GpuProcess.no_gpu_process
https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Debug%20%28New%20Intel%29/builds/3430 - GpuProcess.driver_bug_workarounds_upon_gl_renderer
https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Debug%20%28New%20Intel%29/builds/3429 - GpuProcess.canvas2d, GpuProcess.skip_gpu_process (*)
https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Debug%20%28New%20Intel%29/builds/3428 - GpuProcess.has_transparent_visuals_gpu_process (*)
https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Debug%20%28New%20Intel%29/builds/3426 - GpuProcess.webgl
https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Debug%20%28New%20Intel%29/builds/3421 - GpuProcess.css3d, GpuProcess.equal_bug_workarounds_in_browser_and_gpu_process
https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Debug%20%28New%20Intel%29/builds/3420 - GpuProcess.software_gpu_process
https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Debug%20%28New%20Intel%29/builds/3419 - GpuProcess.webgl

Typical error:
[ RUN      ] GpuProcess.identify_active_gpu2
(INFO) 2016-08-16 10:18:44,307 desktop_browser_backend.GetBrowserStartupArgs:250  Requested remote debugging port: 0
(INFO) 2016-08-16 10:18:44,307 desktop_browser_backend.Start:285  Starting Chrome ['/b/c/b/Linux_Debug__New_Intel_/iregKq65/out/Debug/chrome', '--gpu-testing-secondary-device-ids=0x0de1', '--gpu-testing-gl-vendor=Intel', '--enable-gpu-benchmarking', '--gpu-testing-secondary-vendor-ids=0x10de', '--js-flags=--expose-gc', '--gpu-testing-device-id=0x040a', '--enable-logging=stderr', '--gpu-testing-vendor-id=0x8086', '--enable-net-benchmarking', '--metrics-recording-only', '--no-default-browser-check', '--no-first-run', '--enable-gpu-benchmarking', '--disable-background-networking', '--no-proxy-server', '--disable-component-extensions-with-background-pages', '--disable-default-apps', '--remote-debugging-port=0', '--enable-crash-reporter-for-testing', '--window-size=1280,1024', '--user-data-dir=/b/c/b/Linux_Debug__New_Intel_/itmg7rOo/tmp0bvjLD', 'about:blank']
[1:1:0816/101845:ERROR:memory_mapped_file.cc(52)] Couldn't open /b/c/b/Linux_Debug__New_Intel_/iregKq65/out/Debug/chrome_200_percent.pak
[1:1:0816/101845:ERROR:data_pack.cc(79)] Failed to mmap datapack
[14502:14502:0816/101845:ERROR:memory_mapped_file.cc(52)] Couldn't open /b/c/b/Linux_Debug__New_Intel_/iregKq65/out/Debug/chrome_200_percent.pak
[14502:14502:0816/101845:ERROR:data_pack.cc(79)] Failed to mmap datapack
[14502:14502:0816/101845:WARNING:password_store_factory.cc(250)] Using basic (unencrypted) store for password storage. See https://chromium.googlesource.com/chromium/src/+/master/docs/linux_password_storage.md for more information about password storage options.
(INFO) 2016-08-16 10:18:45,928 desktop_browser_backend.HasBrowserFinishedLaunching:237  Discovered ephemeral port 41340
(INFO) 2016-08-16 10:18:57,103 desktop_browser_backend.HasBrowserFinishedLaunching:237  Discovered ephemeral port 41340
(INFO) 2016-08-16 10:19:09,397 desktop_browser_backend.HasBrowserFinishedLaunching:237  Discovered ephemeral port 41340
(INFO) 2016-08-16 10:19:22,922 desktop_browser_backend.HasBrowserFinishedLaunching:237  Discovered ephemeral port 41340
(INFO) 2016-08-16 10:19:37,800 desktop_browser_backend.HasBrowserFinishedLaunching:237  Discovered ephemeral port 41340
(WARNING) 2016-08-16 10:19:47,812 desktop_browser_backend._GetAllCrashpadMinidumps:349  No path to crashpad_database_util found
(INFO) 2016-08-16 10:19:47,812 desktop_browser_backend._GetMostRecentMinidump:417  No minidump found via crashpad_database_util
(WARNING) 2016-08-16 10:19:47,813 desktop_browser_backend._GetAllCrashpadMinidumps:349  No path to crashpad_database_util found
(INFO) 2016-08-16 10:19:47,813 desktop_browser_backend._GetMostRecentMinidump:417  No minidump found via crashpad_database_util
Can't get standard output with --show-stdout
(WARNING) 2016-08-16 10:19:52,898 desktop_browser_backend.Close:589  Failed to gracefully shutdown.
(WARNING) 2016-08-16 10:19:52,898 desktop_browser_backend.Close:593  Proceed to kill the browser.
(ERROR) 2016-08-16 10:19:52,901 browser.__init__:62  Failure while starting browser backend.
Traceback (most recent call last):
  File "/b/c/b/Linux_Debug__New_Intel_/iregKq65/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 55, in __init__
    self._browser_backend.Start()
  File "/b/c/b/Linux_Debug__New_Intel_/iregKq65/third_party/catapult/telemetry/telemetry/internal/backends/chrome/desktop_browser_backend.py", line 294, in Start
    self._WaitForBrowserToComeUp()
  File "/b/c/b/Linux_Debug__New_Intel_/iregKq65/third_party/catapult/telemetry/telemetry/internal/backends/chrome/chrome_browser_backend.py", line 174, in _WaitForBrowserToComeUp
    raise exceptions.BrowserConnectionGoneException(self.browser, e)
BrowserConnectionGoneException: Timed out while waiting 60s for HasBrowserFinishedLaunching.
Found Minidump: False
Stack Trace:
********************************************************************************
	No crash dump found.
********************************************************************************
Standard output:
********************************************************************************
********************************************************************************
(WARNING) 2016-08-16 10:19:52,902 shared_page_state.DumpStateUponFailure:136  Cannot dump browser state: No browser.
(WARNING) 2016-08-16 10:19:52,902 shared_page_state.DumpStateUponFailure:144  Taking screenshots upon failures disabled.
Traceback (most recent call last):
  File "/b/c/b/Linux_Debug__New_Intel_/iregKq65/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 79, in _RunStoryAndProcessErrorIfNeeded
    state.WillRunStory(story)
  File "/b/c/b/Linux_Debug__New_Intel_/iregKq65/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 218, in WillRunStory
    self._StartBrowser(page)
  File "/b/c/b/Linux_Debug__New_Intel_/iregKq65/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 178, in _StartBrowser
    self._browser = self._possible_browser.Create(self._finder_options)
  File "/b/c/b/Linux_Debug__New_Intel_/iregKq65/third_party/catapult/telemetry/telemetry/internal/backends/chrome/desktop_browser_finder.py", line 68, in Create
    browser_backend, self._platform_backend, self._credentials_path)
  File "/b/c/b/Linux_Debug__New_Intel_/iregKq65/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 55, in __init__
    self._browser_backend.Start()
  File "/b/c/b/Linux_Debug__New_Intel_/iregKq65/third_party/catapult/telemetry/telemetry/internal/backends/chrome/desktop_browser_backend.py", line 294, in Start
    self._WaitForBrowserToComeUp()
  File "/b/c/b/Linux_Debug__New_Intel_/iregKq65/third_party/catapult/telemetry/telemetry/internal/backends/chrome/chrome_browser_backend.py", line 174, in _WaitForBrowserToComeUp
    raise exceptions.BrowserConnectionGoneException(self.browser, e)
BrowserConnectionGoneException: Timed out while waiting 60s for HasBrowserFinishedLaunching.
Found Minidump: False
Stack Trace:
********************************************************************************
	No crash dump found.
********************************************************************************
Standard output:
********************************************************************************
********************************************************************************

[  FAILED  ] GpuProcess.identify_active_gpu2 (68598 ms)

Maybe this bot is just to slow to start the browser in 60 seconds?

On the other hand, the failures I've marked with (*) have this stack from tc_free:

	 1  libbase.so!MallocBlockQueueEntry::MallocBlockQueueEntry(MallocBlock*, unsigned long) + 0x5d
	    rbx = 0x0000350c44744090   rbp = 0x00007fff133ea330
	    rsp = 0x00007fff133ea310   r12 = 0x0000000000000002
	    r13 = 0x0000000000000010   r14 = 0x00007fff133eaac0
	    r15 = 0x00007fff133ea9b0   rip = 0x00007ff5add222bd
	    Found by: call frame info
	 2  libbase.so!MallocBlock::ProcessFreeQueue(MallocBlock*, unsigned long, int) + 0x8d
	    rbx = 0x0000350c44744090   rbp = 0x00007fff133ea780
	    rsp = 0x00007fff133ea340   r12 = 0x0000000000000002
	    r13 = 0x0000000000000010   r14 = 0x00007fff133eaac0
	    r15 = 0x00007fff133ea9b0   rip = 0x00007ff5add200fd
	    Found by: call frame info
	 3  libbase.so!MallocBlock::Deallocate(int) + 0x124
	    rbx = 0x0000350c44744090   rbp = 0x00007fff133ea7d0
	    rsp = 0x00007fff133ea790   r12 = 0x0000000000000002
	    r13 = 0x0000000000000010   r14 = 0x00007fff133eaac0
	    r15 = 0x00007fff133ea9b0   rip = 0x00007ff5add24364
	    Found by: call frame info
	 4  libbase.so!DebugDeallocate(void*, int) + 0xe5
	    rbx = 0x0000350c44744090   rbp = 0x00007fff133ea820
	    rsp = 0x00007fff133ea7e0   r12 = 0x0000000000000002
	    r13 = 0x0000000000000010   r14 = 0x00007fff133eaac0
	    r15 = 0x00007fff133ea9b0   rip = 0x00007ff5add1cc25
	    Found by: call frame info
	 5  libbase.so!tc_free + 0x23
	    rbx = 0x0000350c44744090   rbp = 0x00007fff133ea840
	    rsp = 0x00007fff133ea830   r12 = 0x0000000000000002
	    r13 = 0x0000000000000010   r14 = 0x00007fff133eaac0
	    r15 = 0x00007fff133ea9b0   rip = 0x00007ff5add4b9a3
	    Found by: call frame info

I couldn't reproduce this on my workstation (maybe because it has NVIDIA card).
Ken, can you please triage?

Comment 1 by kbr@chromium.org, Aug 17 2016

Cc: kbr@chromium.org
Components: Internals>GPU>Testing Blink>JavaScript
Labels: Hotlist-PixelWrangler
Owner: vmi...@chromium.org
Status: Assigned (was: Untriaged)
Victor's been trying to reproduce these failures under a different bug ID.

He told me offline that it's possible, though very difficult, to reproduce. It seems to reproduce the first time after a build. Could it be a race condition that happens most often in cold start scenarios?

Comment 2 by vmi...@chromium.org, Aug 18 2016 

This seems the same as  Issue 636489 .  Unfortunately on my workstation I've only been able to reproduce this crash 3 times out of hundreds of runs.

On the bots it's failing an average of 1 in 10, so bisecting via the bots may help.

Comment 3 by kbr@chromium.org, Aug 18 2016

Mergedinto: 636489
Status: Duplicate (was: Assigned)
Thanks for triaging this Victor. Duplicating into the other bug.
►

Sign in to add a comment