Note: this is still happening. Please investigate the behavior on build73-b1 first:
https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Debug%20(New%20Intel)

build72-b1 appears to be misconfigured per Issue 607541.

Note: Issue 607541 was a red herring. Debugging these problems either on build72-b1 or build73-b1 should be fine.

After more thought -- these bots aren't using Swarming.

There must be something going wrong in the internals of the recipe engine. Here is one example of a step that failed:

https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Release%20%28New%20Intel%29/builds/1438

[       OK ] ScreenshotSync.WithCanvas (14158 ms)
[ RUN      ] ScreenshotSync.WithDivs
[       OK ] ScreenshotSync.WithDivs (12999 ms)
(WARNING) 2016-05-05 10:19:51,640 desktop_browser_backend.Close:527  Failed to gracefully shutdown.
(WARNING) 2016-05-05 10:19:51,640 desktop_browser_backend.Close:531  Proceed to kill the browser.
[  PASSED  ] 2 tests.

(INFO) 2016-05-05 10:19:51,644 atexit_with_log._wrapped_function:10  Try running <function _ListAllSubprocesses at 0x7f015f3d1758>
(ERROR) 2016-05-05 10:19:51,667 ps_util._ListAllSubprocesses:89  Telemetry leaks these processes: chrome (21785) - [''], python (21887) - ['']
(INFO) 2016-05-05 10:19:51,667 atexit_with_log._wrapped_function:12  Did run <function _ListAllSubprocesses at 0x7f015f3d1758>
Running ['/usr/bin/python', '../../content/test/gpu/run_gpu_test.py', 'screenshot_sync', '--show-stdout', '--browser=release', '-v', '--extra-browser-args=--enable-logging=stderr --js-flags=--expose-gc', '--output-dir', '/tmp/isolated_tmpCTprv_/tmpCDCsBwtelemetry', '--output-format=json']
Command ['/usr/bin/python', '../../content/test/gpu/run_gpu_test.py', 'screenshot_sync', '--show-stdout', '--browser=release', '-v', '--extra-browser-args=--enable-logging=stderr --js-flags=--expose-gc', '--output-dir', '/tmp/isolated_tmpCTprv_/tmpCDCsBwtelemetry', '--output-format=json'] returned exit code 0
Additional test environment:
    CHROME_DEVEL_SANDBOX=/opt/chromium/chrome_sandbox
    LANG=en_US.UTF-8
Command: /usr/bin/python ../../testing/scripts/run_telemetry_benchmark_as_googletest.py ../../content/test/gpu/run_gpu_test.py screenshot_sync --show-stdout --browser=release -v --extra-browser-args=--enable-logging=stderr --js-flags=--expose-gc --isolated-script-test-output /tmp/tmp1Lv7eB.json

21599 2016-05-05 17:19:51.709 I: Waiting for proces exit
21599 2016-05-05 17:19:51.709 I: Profiling: Section RunTest took 40.477 seconds
21599 2016-05-05 17:19:51.709 I: Command finished with exit code 0 (0x0)
21599 2016-05-05 17:19:51.709 I: rmtree(/tmp/isolated_run51Y9J1)
21599 2016-05-05 17:19:51.974 I: rmtree(/tmp/isolated_tmpCTprv_)
21599 2016-05-05 17:19:51.975 I: rmtree(/tmp/isolated_outxJxVtZ)
21599 2016-05-05 17:19:51.984 I: Result:
{"exit_code":0,"had_hard_timeout":false,"internal_failure":null,"outputs_ref":null,"version":2}
21599 2016-05-05 17:19:51.984 I: Waiting for all threads to die...
21599 2016-05-05 17:19:51.984 I: Done.


--------------------------------------------------------------------------------
started: Thu May  5 10:19:09 2016
ended: Thu May  5 10:59:52 2016
duration: 40 mins, 42 secs
status: FAILURE
status reason: return code was -1.


Stephen, could you please help? These bots were stable when they were running Wily and are now unstable running Xenial. The upgrade was needed in order to run the latest version of Intel's graphics driver.

I'll take a look.

cc-ing friedman who's been doing some work on xenial lately

friedman says he doesn't see anything obvious. 

Just to confirm, the error

[1:1:0505/092241:ERROR:resource_bundle.cc(754)] Failed to load /tmp/isolated_runl3R7ze/out/Release/chrome_material_100_percent.pak


is probably not related, right?

It's a good question, and ideally the warning would be fixed by fixing Chromium's isolates, but it's not related. This job:

https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Release%20%28NVIDIA%29/builds/39665/steps/screenshot_sync_tests%20on%20NVIDIA%20GPU%20on%20Linux%20on%20Linux/logs/stdio

has the same warning but the exit code is 0. (That job's also swarmed, but this shouldn't be related, I assume.)

I caught it in a timeout about an hour ago, and got some data. I wasn't able pinpoint exactly what was wrong, but I did capture some data about it.

Here's the output of strace while it's running:
sudo strace -p 3180
strace: Process 3180 attached
futex(0x1ce7460, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff^Cstrace: Process 3180 detached
 <detached ...>

I've also attached the output of 'ps auxf' while the process was running.

We've run into something similar ( crbug.com/522396 ) to this on windows layout_tests, which we recently discovered to be a problem with the recipe engine and how it launches some threads. cc-ing relevant people.

Deleted: martiniss_psauxf_4 24.7 KB

martiniss_psauxf_4 24.7 KB View Download

cc-ing iannucci as well

I'm going to wait until the bot hangs again, and try diagnosing more then.

I've confirmed that this is the same thing as  bug 522396 . Merging into that bug.

I'm not willing to say that this bug and 522396 are the same issue, so I'm unmerging this bug.

The test behaviour of https://codereview.chromium.org/1959563002 does not seem to be the same behaviour as seen with the Windows issue.

The fix however might end up being the same.

Comments 184 till 232 in https://bugs.chromium.org/p/chromium/issues/detail?id=522396 actually refer to this bug and not the Windows issue.

Why don't you think these are the same issue? It looked like it was the same thing to me; I was able to produce stack traces similar to what you produced in  bug 522396  on the linux machines, so I figured it was the same thing.

Ah ok, just read the update on the other bug.

These bots are running reliably now. Closing as fixed.

Rather, dup'ing into  Issue 522396 .