New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 649904 link

Starred by 3 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug

Blocked on:
issue 467809



Sign in to add a comment

Flakiness of gpu tests on Win7 Debug (New Intel) GPU.FYI bot, exception because of DCHECK

Project Member Reported by ynovikov@chromium.org, Sep 24 2016

Issue description

Different tests fail: trace_test, gpu_process_launch_tests, context_lost_tests. Log contains:
[176:5200:0917/001455:FATAL:handle_closer_agent.cc(89)] Check failed: dup_dummy == closed_handle. 

Started in https://build.chromium.org/p/chromium.gpu.fyi/builders/Win7%20Debug%20%28New%20Intel%29/builds/1248,
as far as I can tell.

 crbug.com/648369  could be related.
 

Comment 1 by kbr@chromium.org, Sep 24 2016

Components: Security Internals>Sandbox
Labels: -OS-Linux -Pri-1 Pri-2
Owner: wfh@chromium.org
Status: Assigned (was: Unconfirmed)
Can't be related; handle_closer_agent.cc is Windows-only.

For anyone investigating, it's this bot:

https://build.chromium.org/p/chromium.gpu.fyi/builders/Win7%20Debug%20%28New%20Intel%29?numbuilds=200

Here's the full stack trace.

[1916:4036:0916/170146:FATAL:handle_closer_agent.cc(89)] Check failed: dup_dummy == closed_handle. 
Backtrace:
	base::debug::StackTrace::StackTrace [0x1007B9A7+23]
	logging::LogMessage::~LogMessage [0x100CB92B+59]
	sandbox::HandleCloserAgent::AttemptToStuffHandleSlot [0x0052A164+612]
	sandbox::HandleCloserAgent::CloseHandles [0x0052A4B5+629]
	sandbox::CountedBuffer::Buffer [0x005196BE+78]
	sandbox::TargetServicesBase::LowerToken [0x00519C69+185]
	content::RendererMainPlatformDelegate::EnableSandbox [0x131C5297+55]
	content::RendererMain [0x131C4CD4+580]
	content::RunNamedProcessTypeMain [0x135B4457+135]
	content::ContentMainRunnerImpl::Run [0x135B4318+488]
	content::ContentMain [0x135B2184+100]
	ChromeMain [0x04AC2016+198]
	MainDllLoader::Launch [0x0042E0D4+916]
	wWinMain [0x00428EDD+541]
	invoke_main [0x0056931E+30] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:118)
	__scrt_common_main_seh [0x00569180+336] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253)
	__scrt_common_main [0x0056901D+13] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:296)
	wWinMainCRTStartup [0x00569338+8] (f:\dd\vctools\crt\vcstartup\src\startup\exe_wwinmain.cpp:17)
	BaseThreadInitThunk [0x74CE38F4+36]
	RtlUnicodeStringToInteger [0x77A45DE3+595]
	RtlUnicodeStringToInteger [0x77A45DAE+542]

Will, may I assign this to you to triage? I'm downgrading this to P2 because this is an FYI-only debug bot -- though we'd like it to be reliably green.

Comment 2 by kbr@chromium.org, Sep 24 2016

Blockedon: 467809
Labels: -Pri-2 Pri-1
Actually, I'm upgrading this to P1 again so it gets triaged quickly.  Issue 467809  was the previous incidence of this problem. Thanks Yuly for raising the problem.

Comment 3 by wfh@chromium.org, Sep 25 2016

This test has indeed been flaky for a while, on all OS.

Comment 4 by kbr@chromium.org, Sep 25 2016

wfh@: could you please help us with the handle_closer_agent.cc problem here? That will address flakiness of these tests on Windows.


Comment 6 by wfh@chromium.org, Oct 26 2016

Cc: ynovikov@chromium.org
I think the best option for the short term would be to remove the DCHECK - it should not be happening but it depends on the state of the handle table while the process is launching, so something that warrants deeper investigation (and might not be achievable in the short term)

If this is blocking you, I would lgtm a CL to remove the DCHECK with a TODO(wfh) to investigate why exactly this is failing on certain platform configurations.

Comment 7 by kbr@chromium.org, Oct 26 2016

Yuly: would it be possible for you to put together that CL? We really should get this bot reliably green.

Looking more at the failures, I suspect now that "Check failed: dup_dummy == closed_handle" happens later, when the test tries to recover after a previous problem.

In https://build.chromium.org/p/chromium.gpu.fyi/builders/Win7%20Debug%20%28New%20Intel%29/builds/1492
I see
err: rx::Renderer11::initializeD3DDevice(748): Failed creating Debug D3D11 device - falling back to release runtime.

In
https://build.chromium.org/p/chromium.gpu.fyi/builders/Win7%20Debug%20%28New%20Intel%29/builds/1509
there is
[5300:5920:1026/205536:ERROR:scoped_com_initializer.h(58)] Multiple CoInitialize() calls for thread 5920

In
https://build.chromium.org/p/chromium.gpu.fyi/builders/Win7%20Debug%20%28New%20Intel%29/builds/1510
there is
DevtoolsTargetCrashException: Devtools target crashed

I imagine what happens is that browser gets stuck in unkillable state, new browser gets launched, and then DCHECK fires.
Probably the same thing happens on all "New Intel" Windows and Linux bots, just on Win Debug we get more messages.

Ultimately, I suspect the source of the problem is what was mentioned in  issue 648369  - hardware.
Still, it would be nice if we were able to recover cleanly from problems even on slow hardware.

Comment 9 by kbr@chromium.org, Oct 28 2016

Thanks for your help diagnosing this Yuly.

Project Member

Comment 10 by bugdroid1@chromium.org, Nov 9 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/da7006ebeb66503c5988482e3631d43d44f5c74c

commit da7006ebeb66503c5988482e3631d43d44f5c74c
Author: jbauman <jbauman@chromium.org>
Date: Wed Nov 09 22:12:14 2016

Remove DCHECK when handle stuffing fails.

This seems to be causing a lot of flakes on the Win7 Intel GPU FYI bot.

BUG=649904
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win10_chromium_x64_rel_ng

Review-Url: https://codereview.chromium.org/2484813005
Cr-Commit-Position: refs/heads/master@{#431049}

[modify] https://crrev.com/da7006ebeb66503c5988482e3631d43d44f5c74c/sandbox/win/src/handle_closer_agent.cc

Sign in to add a comment