A symptom appears to be a large number of "New Crash Dump" messages in test_that.DEBUG:

https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/x86-alex-tot-chrome-pfq-informational/R51-8043.0.0-b19352/vm_test_results_1/test_harness/all/SimpleTestUpdateAndVerify/2_autotest_tests/debug/

The crash info is not especially helpful. I suspect we are crashing on shutdown:

https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/x86-alex-tot-chrome-pfq-informational/R51-8043.0.0-b19352/vm_test_results_1/test_harness/all/SimpleTestUpdateAndVerify/2_autotest_tests/results-01-login_LogoutProcessCleanup/login_LogoutProcessCleanup/sysinfo/iteration.1/var/spool/crash/

The crashes all appear to be in swrast_dri - is that Mesa? This might be a red herring, but did we make any recent changes to the mesa drivers or the VMTests that might be causing crashes on shutdown?

(And by recent I mean specifically around Wed March 9th)

Yes, I think that is Mesa. Would be nice if there were symbols. Maybe removed a patch to aggressively. I'll dig some more, but that should be for Stephane.

Crash reason:  SIGSEGV
Crash address: 0x0
Process uptime: not available

Thread 0 (crashed)
 0  swrast_dri.so + 0x5cc116
    eip = 0x6e36e116   esp = 0x6d46dfe0   ebp = 0x6d46e178   ebx = 0x6f017a54
    esi = 0x7badd57c   edi = 0x7a3b0708   eax = 0x7bae3760   ecx = 0x00000000
    edx = 0x00000000   efl = 0x00210206
    Found by: given as instruction pointer in context
 1  swrast_dri.so + 0x5cb325
    eip = 0x6e36d325   esp = 0x6d46e180   ebp = 0x6d46e1e8
    Found by: previous frame's frame pointer
 2  swrast_dri.so + 0x5cad3b
    eip = 0x6e36cd3b   esp = 0x6d46e1f0   ebp = 0x6d46e218
    Found by: previous frame's frame pointer

Well, it still has the swrast patch
https://chromium-review.googlesource.com/#/c/331365/
Digging for symbols.

Nope, no symbolisation anywhere.

Many but not all login related tests crashing. I think logging in as guest may repro. Sample tests
login_LogoutProcessCleanup
login_CryptohomeIncognito

I was able to reproduce the crashes running:

./bin/cros_run_vm_test --board=x86-alex --results_dir=/tmp/vm_results_f1dcaac login_LogoutProcessCleanup

at both:

d4dcbd342dd54f55383daf8bc44b2c9d97fe0d0b - the last revsion in build 19352
f1dcaac3801ed8d0a8abe62f1496e61fbaf639a7 - the last revision in build 19351

Since 19352* was slow and had crashes, and 19351 was fast and did not have crashes, I can only surmise that this is related to Mesa or the VM, but I don't know how to verify whether something changed there? Both runs appear to be using Chrome OS R51-8043.0.0.


* https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/x86-alex-tot-chrome-pfq-informational/builds/19352

https://chromium-review.googlesource.com/#/c/332681/1 fixes it

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/0b1086ca2be31dccd9d05faa547e2063b269a5d0

commit 0b1086ca2be31dccd9d05faa547e2063b269a5d0
Author: Stéphane Marchesin <marcheu@chromium.org>
Date: Tue Mar 15 01:34:31 2016

media-libs/mesa: Fix llvmpipe alignment restrictions

The code assumes that a struct on the stack is aligned to a multiple
of 16, which isn't the case on 32 bit platforms. So relax the
alignment constraints on the generated assembly.

BUG= chromium:594659 
TEST=run a 32 bit image in a VM

Change-Id: I5dde3c6488e6f8722a10ff1d9da7d52bce0d17ca
Reviewed-on: https://chromium-review.googlesource.com/332681
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Tested-by: Stéphane Marchesin <marcheu@chromium.org>
Commit-Queue: Stéphane Marchesin <marcheu@chromium.org>

[modify] https://crrev.com/0b1086ca2be31dccd9d05faa547e2063b269a5d0/media-libs/mesa/mesa-11.3.0.ebuild
[modify] https://crrev.com/0b1086ca2be31dccd9d05faa547e2063b269a5d0/media-libs/mesa/mesa-9999.ebuild
[add] https://crrev.com/0b1086ca2be31dccd9d05faa547e2063b269a5d0/media-libs/mesa/files/10.5-lp_rast-Remove-alignment-constraints.patch
[add] https://crrev.com/0b1086ca2be31dccd9d05faa547e2063b269a5d0/media-libs/mesa/mesa-11.3.0-r1.ebuild

32-bit x86 VM's seem happy now:
https://uberchromegw.corp.google.com/i/chromeos/builders/x86-alex-chrome-pfq