The earliest flake I could find was https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29/1657 which dates back to July 2nd, though it's quite possible the culprit is introduced before that. In many tests I see stack traces such as the following:

[ RUN      ] CookiesEnabled.RunAsyncTest
[5288:7732:0716/070455.355:4821062:ERROR:registration_protocol_win.cc(84)] TransactNamedPipe: The pipe has been ended. (0x6D)
[5288:7732:0716/070455.355:4821062:WARNING:resource_bundle.cc(358)] locale_file_path.empty() for locale
[5288:7732:0716/070455.402:4821109:ERROR:gpu_process_transport_factory.cc(642)] Switching to software compositing.
[5288:7732:0716/070455.402:4821109:ERROR:gpu_process_transport_factory.cc(1007)] Lost UI shared context.
[0716/070455.558:ERROR:registration_protocol_win.cc(56)] CreateFile: The system cannot find the file specified. (0x2)
../../headless/test/headless_browser_test.cc(251): error: Failed
Abnormal renderer termination
Stack trace:
Backtrace:
	StackTraceGetter::CurrentStackTrace [0x00007FF66E2A9680+80]
	testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop [0x00007FF66E2C2F7A+90]
	testing::internal::AssertHelper::operator= [0x00007FF66E2C2A7A+90]
	headless::HeadlessAsyncDevTooledBrowserTest::RenderProcessExited [0x00007FF66DEB58D0+192]
	headless::HeadlessWebContentsImpl::RenderProcessExited [0x00007FFDC02928A5+421]
	content::RenderProcessHostImpl::ProcessDied [0x00007FFDB9DE7EE6+1222]
	content::RenderProcessHostImpl::OnChannelError [0x00007FFDB9DE9D83+179]
	IPC::ChannelProxy::Context::OnDispatchError [0x00007FFDC6ED0E6D+45]
	??$Invoke@P8Context@ChannelProxy@IPC@@EAAXXZAEBV?$scoped_refptr@VContext@ChannelProxy@IPC@@@@$$V@?$FunctorTraits@P8Context@ChannelProxy@IPC@@EAAXXZX@internal@base@@SAXP8Context@ChannelProxy@IPC@@EAAXXZAEBV?$scoped_refptr@VContext@ChannelProxy@IPC@@@@@Z [0x00007FFDC6ED7CF3+67]
	base::internal::InvokeHelper<0,void>::MakeItSo<void (__cdecl IPC::ChannelProxy::Context::*const & __ptr64)(void) __ptr64,scoped_refptr<IPC::ChannelProxy::Context> const & __ptr64> [0x00007FFDC6ED7C66+86]
	base::internal::Invoker<base::internal::BindState<void (__cdecl IPC::ChannelProxy::Context::*)(void) __ptr64,scoped_refptr<IPC::ChannelProxy::Context> >,void __cdecl(void)>::RunImpl<void (__cdecl IPC::ChannelProxy::Context::*const & __ptr64)(void) __ptr64 [0x00007FFDC6ED7C09+73]
	base::internal::Invoker<base::internal::BindState<void (__cdecl IPC::ChannelProxy::Context::*)(void) __ptr64,scoped_refptr<IPC::ChannelProxy::Context> >,void __cdecl(void)>::Run [0x00007FFDC6ED7B3C+60]
	base::OnceCallback<void __cdecl(void)>::Run [0x00007FFDADEBC301+97]
	base::debug::TaskAnnotator::RunTask [0x00007FFDADF23D13+915]
	base::internal::IncomingTaskQueue::RunTask [0x00007FFDADFBCDB4+212]
	base::MessageLoop::RunTask [0x00007FFDADFC73F2+946]
	base::MessageLoop::DeferOrRunPendingTask [0x00007FFDADFC7C13+83]
	base::MessageLoop::DoWork [0x00007FFDADFC8019+793]
	base::MessagePumpForUI::DoRunLoop [0x00007FFDADFD4B5D+77]
	base::MessagePumpWin::Run [0x00007FFDADFD409C+140]
	base::MessageLoop::Run [0x00007FFDADFC6C7C+524]
	base::RunLoop::Run [0x00007FFDAE0A270A+506]
	headless::HeadlessBrowserTest::RunAsynchronousTest [0x00007FF66DEB51C0+432]
	headless::HeadlessAsyncDevTooledBrowserTest::RunTest [0x00007FF66DEB5B87+583]
	headless::CookiesEnabled_RunAsyncTest_Test::RunTestOnMainThread [0x00007FF66DE6C563+19]
	content::BrowserTestBase::ProxyRunTestOnMainThreadLoop [0x00007FF66F31EA7B+811]
	??$Invoke@P8BrowserTestBase@content@@EAAXXZPEAV12@$$V@?$FunctorTraits@P8BrowserTestBase@content@@EAAXXZX@internal@base@@SAXP8BrowserTestBase@content@@EAAXXZ$$QEAPEAV34@@Z [0x00007FF66F32151A+26]
	base::internal::InvokeHelper<0,void>::MakeItSo<void (__cdecl content::BrowserTestBase::*const & __ptr64)(void) __ptr64,content::BrowserTestBase * __ptr64> [0x00007FF66F321494+52]
	base::internal::Invoker<base::internal::BindState<void (__cdecl content::BrowserTestBase::*)(void) __ptr64,base::internal::UnretainedWrapper<content::BrowserTestBase> >,void __cdecl(void)>::RunImpl<void (__cdecl content::BrowserTestBase::*const & __ptr64) [0x00007FF66F321448+88]
	base::internal::Invoker<base::internal::BindState<void (__cdecl content::BrowserTestBase::*)(void) __ptr64,base::internal::UnretainedWrapper<content::BrowserTestBase> >,void __cdecl(void)>::Run [0x00007FF66F32136C+60]
	base::OnceCallback<void __cdecl(void)>::Run [0x00007FFDADEBC301+97]
	base::debug::TaskAnnotator::RunTask [0x00007FFDADF23D13+915]
	base::internal::IncomingTaskQueue::RunTask [0x00007FFDADFBCDB4+212]
	base::MessageLoop::RunTask [0x00007FFDADFC73F2+946]
	base::MessageLoop::DeferOrRunPendingTask [0x00007FFDADFC7C13+83]
	base::MessageLoop::DoWork [0x00007FFDADFC8019+793]
	base::MessagePumpForUI::DoRunLoop [0x00007FFDADFD4B5D+77]
	base::MessagePumpWin::Run [0x00007FFDADFD409C+140]
	base::MessageLoop::Run [0x00007FFDADFC6C7C+524]
	base::RunLoop::Run [0x00007FFDAE0A270A+506]
	content::BrowserMainLoop::MainMessageLoopRun [0x00007FFDB8ED4196+518]
	content::BrowserMainLoop::RunMainMessageLoopParts [0x00007FFDB8ED3E25+517]
[5288:6768:0716/070455.745:4821453:WARNING:discardable_shared_memory_manager.cc(431)] Some MojoDiscardableSharedMemoryManagerImpls are still alive. They will be leaked.
[  FAILED  ] CookiesEnabled.RunAsyncTest, where TypeParam =  and GetParam() =  (460 ms)

components_browsertests started to failure on
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29/1891

Too many failing tests (21), skipping retries.
2 tests failed:
    SecurityStateContentUtilsBrowserTest.VisibleSecurityStateInsecureFieldEdit (../../components/security_state/content/content_utils_browsertest.cc:99)
    SecurityStateContentUtilsBrowserTest.VisibleSecurityStateNonsecureFormInputs (../../components/security_state/content/content_utils_browsertest.cc:54)
17 tests timed out:
    DistillerPageWebContentsTest.BasicDistillationWorks (../../components/dom_distiller/content/browser/distiller_page_web_contents_browsertest.cc:185)
    DistillerPageWebContentsTest.HandlesRelativeImages (../../components/dom_distiller/content/browser/distiller_page_web_contents_browsertest.cc:223)
    DistillerPageWebContentsTest.HandlesRelativeLinks (../../components/dom_distiller/content/browser/distiller_page_web_contents_browsertest.cc:205)
    DistillerPageWebContentsTest.HandlesRelativeVideos (../../components/dom_distiller/content/browser/distiller_page_web_contents_browsertest.cc:241)
    DistillerPageWebContentsTest.MarkupInfo (../../components/dom_distiller/content/browser/distiller_page_web_contents_browsertest.cc:419)
    DistillerPageWebContentsTest.PageDestroyedBeforeFinishDistillation (../../components/dom_distiller/content/browser/distiller_page_web_contents_browsertest.cc:379)
    DistillerPageWebContentsTest.TestPinch (../../components/dom_distiller/content/browser/distiller_page_web_contents_browsertest.cc:492)
    DistillerPageWebContentsTest.UsingCurrentWebContentsNoMainFrameObserver (../../components/dom_distiller/content/browser/distiller_page_web_contents_browsertest.cc:304)
    DistillerPageWebContentsTest.UsingCurrentWebContentsNotFinishedLoadingYet (../../components/dom_distiller/content/browser/distiller_page_web_contents_browsertest.cc:316)
    DistillerPageWebContentsTest.UsingCurrentWebContentsReadyForDistillation (../../components/dom_distiller/content/browser/distiller_page_web_contents_browsertest.cc:328)
    DistillerPageWebContentsTest.UsingCurrentWebContentsWrongUrl (../../components/dom_distiller/content/browser/distiller_page_web_contents_browsertest.cc:292)
    DistillerPageWebContentsTest.VisibilityDetection (../../components/dom_distiller/content/browser/distiller_page_web_contents_browsertest.cc:264)
    DomDistillerDistillablePageUtilsTest.TestIsDistillablePage (../../components/dom_distiller/content/browser/distillable_page_utils_browsertest.cc:115)
    DomDistillerDistillablePageUtilsTest.TestIsNotDistillablePage (../../components/dom_distiller/content/browser/distillable_page_utils_browsertest.cc:140)
    DomDistillerJsTest.RunJsTests (../../components/dom_distiller/content/browser/test/dom_distiller_js_browsertest.cc:110)
    PageRenovatorBrowserTest.CorrectRenovationsRun (../../components/offline_pages/content/renovations/test/page_renovator_browsertest.cc:198)
    PageRenovatorBrowserTest.WikipediaRenovationRuns (../../components/offline_pages/content/renovations/test/page_renovator_browsertest.cc:223)
2 tests crashed:
    SourceUrlRecorderWebContentsObserverBrowserTest.Basic (../../components/ukm/content/source_url_recorder_browsertest.cc:85)
    SourceUrlRecorderWebContentsObserverBrowserTest.IgnoreUrlInSubframe (../../components/ukm/content/source_url_recorder_browsertest.cc:111)

This bug has an owner, thus, it's been triaged. Changing status to "assigned".

Hi, is there any progress here? These tests are still flooding the Win10 builders with errors:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29/2544
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29/2545
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29/2549

Since headless_browsertests sometimes passed tests and other tests are too.
It seems this is bot's flakiness instead of tests themselves.

Assigned to martiniss@ as he worked on deflaking Win10 IIRC, please reassign if needed.

I haven't worked on deflaking Win10 bots in a long time. 

is there a particular reason we suspect this is an infra issue? it's not clear to me from #8.

+yosin: can you elaborate a bit more?

jbudorick@, since changes are not related to the failure and failed tests are passed after failure found without related changes.

Could you replace or add another machine for "Win10 Tests x64(dbg)"?
I'm not sure actual cause, it can be machine slowness, configuration issue, etc.

Adding a machine with same configuration causes same behavior, it may be configuration issue. If there are no failures, this is caused by hardware.

This issue makes heavy load to build sheriff, change something on bot may help to identify root cause.

This is still an issue, and the stack trace in #c3 is still seen. 

Flakiness Dashboard: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#showAllRuns=true&testType=headless_browsertests&builder=chromium.win%3AWin10%20Tests%20x64%20(dbg)

#13: that just sounds like flaky tests, possible more flaky in this configuration than in others. Tests on this bot run across a large swarming pool, not a single machine.

This is still an issue.

https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8936249688140632160/+/steps/headless_browsertests_on_Windows-10-15063/0/logs/HeadlessProtocolBrowserTest.VirtualTimeLocalStorage/0

The stacktrace doesn't have any information why renderer process crashed...

 Issue 882512  has been merged into this issue.

 Issue 894985  has been merged into this issue.

Still reproduce but stack looks different to #3.

C:/src/build/src/chrome3/src/out_gn/Debug>
$ headless_browsertests.exe
IMPORTANT DEBUGGING NOTE: each test is run inside its own process.
For debugging a test inside a debugger, use the
--gtest_filter=<your_test_name> flag along with either
--single_process (to run the test in one launcher/browser process) or
--single-process (to do the above, and also run Chrome in single-process mode).
Using sharding settings from environment. This is shard 0/1
Using 28 parallel jobs.
Still waiting for the following processes to finish:
        headless_browsertests.exe --disable-gpu-process-for-dx12-vulkan-info-collection --gtest_also_run_disabled_tests --gtest_filter=HeadlessBrowserTest.CreateAndDestroyBrowserContext --single_process --test-launcher-output="C:\src\Temp\scoped_dir23284_10030\results23284_22183\test_results.xml"  
/// ...
        headless_browsertests.exe --disable-gpu-process-for-dx12-vulkan-info-collection --gtest_also_run_disabled_tests --gtest_filter=HeadlessDevToolsClientEvalTest.RunAsyncTest --single_process --test-launcher-output="C:\src\Temp\scoped_dir23284_10030\results23284_23778\test_results.xml"
[1/114] HeadlessBrowserTest.CreateAndDestroyWebContents (28029 ms)
[2/114] HeadlessBrowserTest.CreateAndDoNotDestroyWebContents (29143 ms)
[3/114] HeadlessBrowserTest.CreateWithBadURL (30906 ms)
[4/114] HeadlessBrowserTest.TraceUsingBrowserDevToolsTarget (31793 ms)
[5/114] HeadlessBrowserTest.CreateAndDestroyBrowserContext (31980 ms)
[6/114] HeadlessBrowserTest.WebContentsAreDestroyedWithContext (33983 ms)
[7/114] HeadlessBrowserTest.DestroyAndCreateTwoWebContents (35801 ms)
[8/114] HeadlessBrowserTest.CreateAndDoNotDestroyBrowserContext (36764 ms)
[9/114] HeadlessBrowserTest.ClipboardCopyPasteText (39042 ms)
[10/114] HeadlessBrowserTest.PermissionManagerAlwaysASK (43107 ms)
[ RUN      ] HeadlessBrowserTest.DefaultSizes
[40416:64748:1017/163605.382:525878968:ERROR:registration_protocol_win.cc(31)] debug:SendTo \\.\pipe\crashpad_40416_TBHAKXZSRNADWQNB
Backtrace:
        base::debug::StackTrace::StackTrace [0x00007FFF396C2925+101] (c:\src\build\src\chrome3\src\base\debug\stack_trace_win.cc:289)
        base::debug::StackTrace::StackTrace [0x00007FFF396C18ED+29] (c:\src\build\src\chrome3\src\base\debug\stack_trace.cc:203)
        crashpad::SendToCrashHandlerServer [0x00007FFF37E3EB9E+238] (c:\src\build\src\chrome3\src\third_party\crashpad\crashpad\util\win\registration_protocol_win.cc:33)
        crashpad::`anonymous namespace'::StartHandlerProcess [0x00007FFF37DF6ABA+5114] (c:\src\build\src\chrome3\src\third_party\crashpad\crashpad\client\crashpad_client_win.cc:512)
        crashpad::CrashpadClient::StartHandler [0x00007FFF37DF4E10+1248] (c:\src\build\src\chrome3\src\third_party\crashpad\crashpad\client\crashpad_client_win.cc:635)
        crash_reporter::internal::PlatformCrashpadInitialization [0x00007FFF37E11647+2455] (c:\src\build\src\chrome3\src\components\crash\content\app\crashpad_win.cc:152)
        crash_reporter::`anonymous namespace'::InitializeCrashpadImpl [0x00007FFF37E12206+758] (c:\src\build\src\chrome3\src\components\crash\content\app\crashpad.cc:156)
        crash_reporter::InitializeCrashpadWithEmbeddedHandler [0x00007FFF37E123CD+61] (c:\src\build\src\chrome3\src\components\crash\content\app\crashpad.cc:214)
        headless::HeadlessContentMainDelegate::InitCrashReporter [0x00007FFF375451E6+310] (c:\src\build\src\chrome3\src\headless\lib\headless_content_main_delegate.cc:240)
        headless::HeadlessContentMainDelegate::PreSandboxStartup [0x00007FFF375452EC+60] (c:\src\build\src\chrome3\src\headless\lib\headless_content_main_delegate.cc:260)
        content::ContentMainRunnerImpl::Initialize [0x00007FFF3148480D+1437] (c:\src\build\src\chrome3\src\content\app\content_main_runner_impl.cc:804)
        content::ContentServiceManagerMainDelegate::Initialize [0x00007FFF314812C1+81] (c:\src\build\src\chrome3\src\content\app\content_service_manager_main_delegate.cc:38)
        service_manager::Main [0x00007FFF187232EA+938] (c:\src\build\src\chrome3\src\services\service_manager\embedder\main.cc:416)
        content::ContentMain [0x00007FFF31483C6F+95] (c:\src\build\src\chrome3\src\content\app\content_main.cc:20)
        content::BrowserTestBase::SetUp [0x00007FF7289643D2+2946] (c:\src\build\src\chrome3\src\content\public\test\browser_test_base.cc:340)
        headless::HeadlessBrowserTest::SetUp [0x00007FF726DE4340+144] (c:\src\build\src\chrome3\src\headless\test\headless_browser_test.cc:150)
        testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,void> [0x00007FF7272DBA3F+79] (c:\src\build\src\chrome3\src\third_party\googletest\src\googletest\src\gtest.cc:2508)
        testing::Test::Run [0x00007FF7272DB967+103] (c:\src\build\src\chrome3\src\third_party\googletest\src\googletest\src\gtest.cc:2520)
        testing::TestInfo::Run [0x00007FF7272DC69A+218] (c:\src\build\src\chrome3\src\third_party\googletest\src\googletest\src\gtest.cc:2702)
        testing::TestCase::Run [0x00007FF7272DD303+243] (c:\src\build\src\chrome3\src\third_party\googletest\src\googletest\src\gtest.cc:2815)
        testing::internal::UnitTestImpl::RunAllTests [0x00007FF7272E6F7D+765] (c:\src\build\src\chrome3\src\third_party\googletest\src\googletest\src\gtest.cc:5181)
        testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool>
[0x00007FF7272E6C65+85] (c:\src\build\src\chrome3\src\third_party\googletest\src\googletest\src\gtest.cc:2506)
        testing::UnitTest::Run [0x00007FF7272E6AB3+307] (c:\src\build\src\chrome3\src\third_party\googletest\src\googletest\src\gtest.cc:4791)
        RUN_ALL_TESTS [0x00007FF7289FF231+17] (c:\src\build\src\chrome3\src\third_party\googletest\src\googletest\include\gtest\gtest.h:2333)
        base::TestSuite::Run [0x00007FF7289FE7C3+163] (c:\src\build\src\chrome3\src\base\test\test_suite.cc:294)
        headless::`anonymous namespace'::HeadlessTestLauncherDelegate::RunTestSuite [0x00007FF726DFEEF3+83] (c:\src\build\src\chrome3\src\headless\test\headless_test_launcher.cc:47)
        content::LaunchTests [0x00007FF7289A4496+806] (c:\src\build\src\chrome3\src\content\public\test\test_launcher.cc:647)
        main [0x00007FF726DFEAFD+397] (c:\src\build\src\chrome3\src\headless\test\headless_test_launcher.cc:98)
        invoke_main [0x00007FF72910C314+52] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:79)
        __scrt_common_main_seh [0x00007FF72910C444+292] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283)
        __scrt_common_main [0x00007FF72910C4CE+14] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:326)
        mainCRTStartup [0x00007FF72910C4E9+9] (f:\dd\vctools\crt\vcstartup\src\startup\exe_main.cpp:17)
        BaseThreadInitThunk [0x00007FFF98351FE4+20]
        RtlUserThreadStart [0x00007FFF9860CB31+33]
[40416:64748:1017/163628.831:525902421:ERROR:registration_protocol_win.cc(60)] CreateFile: The system cannot find the file specified. (0x2)
[40416:64748:1017/163628.843:525902437:WARNING:resource_bundle.cc(358)] locale_file_path.empty() for
locale
[40416:64748:1017/163628.915:525902500:ERROR:gpu_process_transport_factory.cc(637)] Switching to software compositing.
,,,

Looking at this during my sheriff shift.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/746af72c69f18007eb6f53036dd174ceffef0611

commit 746af72c69f18007eb6f53036dd174ceffef0611
Author: Bret Sepulveda <bsep@chromium.org>
Date: Tue Oct 23 20:04:31 2018

Speculatively disable HeadlessBrowserTest.WebGLSupported on Windows.

This might be causing cascading failures in the suite, which I'm unable
to reproduce locally.

TBR=skyostil@chromium.org

Bug: 867447
Change-Id: Ia34f8a01386f21df0550f146c5cac544fba96c90
Reviewed-on: https://chromium-review.googlesource.com/c/1297180
Reviewed-by: Bret Sepulveda <bsep@chromium.org>
Commit-Queue: Bret Sepulveda <bsep@chromium.org>
Cr-Commit-Position: refs/heads/master@{#602067}
[modify] https://crrev.com/746af72c69f18007eb6f53036dd174ceffef0611/headless/lib/headless_browser_browsertest.cc

End of shift update: headless_browsertests haven't failed since I disabled the test, so I'm leaving it. alancutter@ floated the hypothesis that the longer runtime is causing it to time out, the test runner is killing the process, and the renderer (or something) is tearing down uncleanly.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ada36c860c33071f931923ecea68de379441769d

commit ada36c860c33071f931923ecea68de379441769d
Author: Bret Sepulveda <bsep@chromium.org>
Date: Wed Oct 24 01:23:28 2018

Disable HeadlessBrowserTest.WebGLSupported on Debug only.

Follow up to http://crrev.com/c/1297180. Test is only flaky on Win dbg.

TBR=skyostil@chromium.org

Bug: 867447
Change-Id: If2bc19e39bfa59519fca38424e27be08459a9958
Reviewed-on: https://chromium-review.googlesource.com/c/1297642
Reviewed-by: Bret Sepulveda <bsep@chromium.org>
Commit-Queue: Bret Sepulveda <bsep@chromium.org>
Cr-Commit-Position: refs/heads/master@{#602208}
[modify] https://crrev.com/ada36c860c33071f931923ecea68de379441769d/headless/lib/headless_browser_browsertest.cc

Still appears to be happening every several builds.
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29/2922

It would be very helpful to see the stack trace of the crashed renderer process. I'm not sure why that's not already in the logs.

+dvallet because headless::HeadlessContentMainDelegate::InitCrashReporter() appeared in #20.
Could this be what the renderer is crashing on and we don't get a stack trace because the crash reporting system is crashing?