Sample task:
https://chromium-swarm.appspot.com/task?id=35a37d0d4caad510

bot:
gce-trusty-52ae995e-2cqk

so it's likely the MP base image that needs more packages.

I'm not seeing what dependencies are needed. Why doesn't --check-layout-test-sys-deps actually say what's missing? Please let me know which dependencies are needed so we can determine what should be added to the Trusty image we use, and what should be added in the isolate for the task.

I think one of the following might be missing;
 * xdpyinfo
 * xvfb

Might be missing?

I'm trying to figure out how to get content-shell to be more verbose about what it can't find.

Actually, it might be missing fonts...

I tried running on this machine again and it seemed to succeed. Maybe it is an issue related to starting xvfb?

I'm trying to get more logging on what is causing the dep fail.

The problem seems to be the "00:59:58.617 25521 [25548:25548:0420/005958.614482:52637425855:ERROR:browser_main_loop.cc(1616)] Unable to open X display."

Sure, I agree that part looks odd, but I have no idea why that happened or what's required to open the X display. If a second task on this same machine succeeded maybe it flakily failed to start something?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dcdb807ced40e620460ccf56362945e6860db1ab

commit dcdb807ced40e620460ccf56362945e6860db1ab
Author: tansell <tansell@chromium.org>
Date: Wed Apr 26 06:30:03 2017

webkitpy: Check xvfb with xdpyinfo after starting.

This makes sure that xvfb actually started correctly and you can connect
to it. We don't use -display argument as we want to make sure the
environment is correct.

BUG= 713597 

Review-Url: https://codereview.chromium.org/2837773002
Cr-Commit-Position: refs/heads/master@{#467253}

[modify] https://crrev.com/dcdb807ced40e620460ccf56362945e6860db1ab/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/linux.py
[modify] https://crrev.com/dcdb807ced40e620460ccf56362945e6860db1ab/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/linux_unittest.py

So, are these GCE instances faster than the others? 

With the patch above, normally you get;
-----------------------
03:15:27.819 14884 Clobbering old results in /b/s/w/ioQs4QUM
03:15:27.825 14884 "xdpyinfo -display :99" took 0.01s
03:15:27.829 14884 "xdpyinfo -display :100" took 0.00s
03:15:27.830 14884 Starting Xvfb with display ":100".
03:15:27.867 14884 "xdpyinfo" took 0.03s
03:15:27.867 14884 Successfully started Xvfb with display ":100".
03:15:27.868 14884 Checking system dependencies ...
-----------------------

But I have now seen the following a couple of times;
-----------------------
03:15:34.677 1854 "/usr/sbin/apache2 -v" took 0.09s
03:15:34.678 1854 Clobbering old results in /b/s/w/io04NUWW
03:15:34.729 1854 "xdpyinfo -display :99" took 0.05s
03:15:34.734 1854 "xdpyinfo -display :100" took 0.01s
03:15:34.735 1854 Starting Xvfb with display ":100".
03:15:34.743 1854 "xdpyinfo" took 0.00s
03:15:34.743 1854 xdpyinfo check failed with exit code 1 while starting Xvfb on ":100".
03:15:34.849 1854 "xdpyinfo" took 0.01s
03:15:34.849 1854 Successfully started Xvfb with display ":100".
03:15:34.850 1854 Checking system dependencies ...
03:15:35.100 1854 "/b/s/w/ir/out/Release/content_shell --check-layout-test-sys-deps" took 0.25s
03:15:35.100 1854 Starting WPTServe ...
03:15:35.101 1854 all ports are available
-----------------------


If we had run the sys-dep check when the following line ran, it would have failed with a sys-dep error;
   03:15:34.743 1854 xdpyinfo check failed with exit code 1 while starting Xvfb on ":100".


Lets leave this to bake for the rest of the week and make sure we don't get any failures, then declare the problem fixed.

Oh, I bumped the shard number back up to 18 to try and increase the likelihood of failures as it should cycle through more machines.

So, I haven't seen any failures in the last 14 hours.

I'll leave this open until Monday next week and if nothing has failed, we can close it as verified fixed.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c2fe004639a9a21fff08b6de088f052a0c5060b8

commit c2fe004639a9a21fff08b6de088f052a0c5060b8
Author: tansell <tansell@chromium.org>
Date: Thu May 04 05:23:00 2017

webkitpy: Poll xvfb while checking with xdpyinfo.

If xvfb dies or fails to start correctly while checking xdpyinfo, we
should exit straight away rather than continuing to try and probe with
xdpyinfo and waiting to time out.

BUG=708681, 713597 

Review-Url: https://codereview.chromium.org/2848833002
Cr-Commit-Position: refs/heads/master@{#469274}

[modify] https://crrev.com/c2fe004639a9a21fff08b6de088f052a0c5060b8/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/linux.py