It's disappointing that this hasn't been triaged yet, but  Issue 859366  has just been filed about the same failure mode.

Robert, do you know who should take a look at this one?

hajimehoshi@, does this happen randomly with the same command line? When it doesn't crash, do you still see "System dependencies check failed", or is that unrelated?

 Issue 859366  has been merged into this issue.

kbr@ reported the same failed check + crash in  issue 859366 , where the crash logs agree with this issue.

For comparison, here's a working run, and shard 9/12 of that:
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win7_chromium_rel_ng/40845
https://chromium-swarm.appspot.com/task?id=3ec3befda12cc410&refresh=10&show_raw=1

The "Checking system dependencies" bit didn't fail there, and notably was followed by "Starting WPTServe" which didn't happen in the failure case.

Searching for webkit_layout_tests failures in https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win7_chromium_rel_ng I'm unable to find other cases where this has happened, but I'm not sure exactly how to look for it. I see some cases where a shard has failed, but how do things appear if it's failed in this way? kbr@, do you know?

For prosperity, here's the failing shard from the other bug:
https://chromium-swarm.appspot.com/task?id=3e6bdf4d4cca0410&refresh=10&show_raw=1

And here's a retry of the Swarming task above, which succeeded:
https://chromium-swarm.appspot.com/task?id=3ec6d792a10b3a10&refresh=10&show_raw=1

The root error seems to be:
[12492:12264:1011/182132.222:2643695890:ERROR:browser_gpu_channel_host_factory.cc(107)] Failed to launch GPU process.

(Re #8, the lack of "Starting WPTServe" is expected because we only start wptserve if the system check passes.)

The error would usually go away after retrying the exact same isolated input.

Perhaps GPU folks would have a clue? (+Internals>GPU, although I don't know if it's the correct component.)

Layout tests run on virtual machines that don't have GPUs. It's expected that they will fail to launch the GPU process. I'm not sure why this error isn't showing up in all Swarming shards but the root cause appears to be the failure of the system dependencies check. Here's another recent failure:

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win7_chromium_rel_ng/41519

and failing shard:

https://chromium-swarm.appspot.com/task?id=3ec72793317f3f10&refresh=10&show_raw=1

Failure log excerpt:

10:07:05.309 5160 Checking system dependencies ...
10:07:06.214 5160 "e:\b\s\w\ir\out\Release\content_shell.exe --check-layout-test-sys-deps" took 0.90s
10:07:06.214 5160 System dependencies check failed.
10:07:06.214 5160 To override, invoke with --nocheck-sys-deps
10:07:06.214 5160 
10:07:06.214 5160 
DevTools listening on ws://127.0.0.1:49675/devtools/browser/736a427f-7d20-458e-9d24-6a28cbd686bb
[5528:5328:0718/100705.777:ERROR:gpu_process_host.cc(589)] BrowserMainRunner::ExitedMainMessageLoop()
[5528:5328:0718/100705.777:WARNING:discardable_shared_memory_manager.cc(431)] Some MojoDiscardableSharedMemoryManagerImpls are still alive. They will be leaked.
[5528:4368:0718/100705.840:WARNING:scoped_temp_dir.cc(23)] Could not delete temp dir in dtor.
[5528:5204:0718/100705.840:FATAL:at_exit.cc(63)] Check failed: !g_top_manager->processing_callbacks_. 
Backtrace:
	base::debug::StackTrace::StackTrace [0x01CA8B80+32]
	base::debug::StackTrace::StackTrace [0x01CA833D+13]
	logging::LogMessage::~LogMessage [0x01CC06E3+83]
	base::AtExitManager::RegisterTask [0x01CA1CDA+122]
	base::AtExitManager::RegisterCallback [0x01CA1C48+184]
	base::internal::CompleteLazyInstance [0x01CBFCAF+31]
	disk_cache::File::Init [0x0310C9A3+499]
	disk_cache::File::Init [0x0310C879+201]
	disk_cache::MappedFile::Init [0x0310D4E6+102]
	disk_cache::BlockFiles::OpenBlockFile [0x03105A1A+202]
	disk_cache::BlockFiles::Init [0x031057B4+164]
	disk_cache::BackendImpl::SyncInit [0x0260D14E+430]
	disk_cache::BackendIO::ExecuteBackendOperation [0x031037CA+58]
	base::debug::TaskAnnotator::RunTask [0x02547082+306]
	base::MessageLoop::RunTask [0x01CC9B23+467]
	base::MessageLoop::DeferOrRunPendingTask [0x01CC9E7D+157]
	base::MessageLoop::DoWork [0x01CCA0F7+599]
	base::MessagePumpForIO::DoRunLoop [0x01CCC7B2+18]
	base::MessagePumpWin::Run [0x01CCBBA1+65]
	base::MessageLoop::Run [0x01CC9677+119]
	base::RunLoop::Run [0x01CE52DC+204]
	base::Thread::Run [0x01D0C414+164]
	base::Thread::ThreadMain [0x01D0C69D+621]
	base::PlatformThread::SetCurrentThreadPriority [0x01D0A865+533]
	BaseThreadInitThunk [0x7590336A+18]
	RtlInitializeExceptionChain [0x77979902+99]
	RtlInitializeExceptionChain [0x779798D5+54]

Removing Internals>GPU as I think this is a red herring.

kbr@, how did you find that shard? Looking at https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win7_chromium_rel_ng/41519 I wouldn't have suspected that shard 0 failed in this way, since other shards failed too and nothing about the overall result looks special. Can you search all the logs somehow?

Anyway, seems like we need more output in the "System dependencies check failed" case, since no clues are printed now. It looks like in https://cs.chromium.org/chromium/src/third_party/blink/tools/blinkpy/web_tests/port/base.py?l=411&rcl=177edef71c816a41c87da3ebc4bdb2eb41c26ffb, `output` was the empty string, not sure if `script_error.output` could have been anything different in the `error_handler` callback.

foolip@: I just got lucky. It was the first shard I looked at. However it doesn't take long to scan through the failing shards and scroll down in the logs to see if any tests ran at all. If they didn't then you know it's the system dependencies check failing.

nednguyen@, this P1 has been languishing for a while, might you be able to take a look?

Looking at the code, it seems to me that place that checking system deps is the check_sys_deps() method in Port subclasses (https://cs.chromium.org/chromium/src/third_party/blink/tools/blinkpy/web_tests/controllers/manager.py?rcl=f96c6f6cf723d115d72229c0c784ae5078891703&l=379). 


Win port doesn't override the default check_sys_deps() (which just returns OK_EXIT_STATUS), so it should be a no-op. 

Digging a bit more, I found Erik's https://chromium-review.googlesource.com/c/chromium/src/+/1229240/ which also make this type of crash recoverable. From his CL description:

> run_web_tests.py would exit with SYS_DEPS_EXIT_STATUS upon encountering a
  crash in kCheckLayoutTestSysDeps. This failure was not recoverable. Now,
  flaky crashes are caught using the normal layout test runner mechanism, and
  are appropriately retried.


So in some sense, Erik has fixed this issue :-) 

I find some minor clean up that can be made following Erik's fixed & will make the CL

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c91ef9689e59c6c83aa408367a68599556b9086f

commit c91ef9689e59c6c83aa408367a68599556b9086f
Author: Ned Nguyen <nednguyen@google.com>
Date: Tue Nov 13 18:51:20 2018

Remove needs_http parameter in check_sys_deps method

This also remove check_sys_deps from base.Port's subclasses that are
the same as the base class method.

Note: this CL removes an unused parameter and is a refactor with no intended behavior change.

This clean up is a follow-up of https://chromium-review.googlesource.com/c/chromium/src/+/1229240/

BUG:  chromium:773617 , chromium:884776 
Change-Id: I82edf733a696c4e0e6de20a8bac954e244d51365
Reviewed-on: https://chromium-review.googlesource.com/c/1333893
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Robert Ma <robertma@chromium.org>
Reviewed-by: Erik Chen <erikchen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#607676}
[modify] https://crrev.com/c91ef9689e59c6c83aa408367a68599556b9086f/third_party/blink/tools/blinkpy/web_tests/controllers/manager.py
[modify] https://crrev.com/c91ef9689e59c6c83aa408367a68599556b9086f/third_party/blink/tools/blinkpy/web_tests/port/android.py
[modify] https://crrev.com/c91ef9689e59c6c83aa408367a68599556b9086f/third_party/blink/tools/blinkpy/web_tests/port/base.py
[modify] https://crrev.com/c91ef9689e59c6c83aa408367a68599556b9086f/third_party/blink/tools/blinkpy/web_tests/port/browser_test.py
[modify] https://crrev.com/c91ef9689e59c6c83aa408367a68599556b9086f/third_party/blink/tools/blinkpy/web_tests/port/fuchsia.py
[modify] https://crrev.com/c91ef9689e59c6c83aa408367a68599556b9086f/third_party/blink/tools/blinkpy/web_tests/port/mock_drt.py
[modify] https://crrev.com/c91ef9689e59c6c83aa408367a68599556b9086f/third_party/blink/tools/blinkpy/web_tests/port/test.py