vm test 1 failed with:
https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/trybot-samus-pre-cq/R51-8064.0.0-b22216/vm_test_results_1/test_harness/all/SimpleTestVerify/1_autotest_tests/results-09-logging_UserCrash/?debugUI=CLOUD
user space crash handling was not started during last boot

vm test 2 failed on the same test with a bit different error:
https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/trybot-samus-pre-cq/R51-8064.0.0-b22216/vm_test_results_2/test_harness/all/SimpleTestVerify/1_autotest_tests/results-09-logging_UserCrash

Autotest client terminated unexpectedly: DUT rebooted during the test run.

It does seem to be some legit failure.

There are no obvious failure candidates in the kernel repo (Mar 11 changes are all for a different CPU architecture):

https://chromium-review.googlesource.com/#/q/status:merged+project:chromiumos/third_party/kernel+branch:chromeos-3.14

There are no relevant changes in the samus overlay / private overlay either.

How is it possible that these tests are failing consistently pre-cq yet passing consistently on samus-paladin?

https://chromegw.corp.google.com/i/chromeos/builders/samus-paladin?numbuilds=200

samus-paladin builder does not run vmtest, samus-release does, and vmtest is failing there too:

https://chromegw.corp.google.com/i/chromeos/builders/samus-release

Here's the diff between the last ok vmtest on samus-release and the first failing vmtest:

https://crosland.corp.google.com/log/8042.0.0..8045.0.0

Let me take a closer look at the kernel changes, there may be one that is relevant for x86.

The VMTest stages are timing out, probably because chrome is crashing and generating the crash dump files slows things down (we are seeing similar problems on the x86-alex pfq builder).

Here are some crashes from a recent samus-release run:

https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/samus-release/R51-8064.0.0/vm_test_results_1/test_harness/all/SimpleTestVerify/1_autotest_tests/results-04-login_OwnershipTaken/login_OwnershipTaken/sysinfo/iteration.1/var/spool/crash/?debugUI=CLOUD

This is the failed build page:
https://uberchromegw.corp.google.com/i/chromeos/builders/samus-release/builds/3015

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/073521df1afd098121f6694469be9e0f135e21c5

commit 073521df1afd098121f6694469be9e0f135e21c5
Author: Stéphane Marchesin <marcheu@chromium.org>
Date: Tue Mar 15 07:15:39 2016

media-libs/mesa: Fix prime mapping code

If we already have a pointer to the buffer, don't overwrite it with
NULL.

BUG= chromium:594699 
TEST=run a samus image in a VM

Change-Id: If63efb3e970e7e6a4e168a60f8cf1936d70a85f8
Reviewed-on: https://chromium-review.googlesource.com/332436
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
Commit-Queue: Stéphane Marchesin <marcheu@chromium.org>
Tested-by: Stéphane Marchesin <marcheu@chromium.org>

[rename] https://crrev.com/073521df1afd098121f6694469be9e0f135e21c5/media-libs/mesa/mesa-11.3.0-r2.ebuild
[modify] https://crrev.com/073521df1afd098121f6694469be9e0f135e21c5/media-libs/mesa/files/10.3-dri-add-swrast-support-on-top-of-prime-imported.patch

Well, the vmtest stage is fixed now, but samus breaks somewhere else :)