Looking, but in general, please file these as trooper bugs.

Looks like we're hitting space limits on the VM images. Prob a few crashed tests that didn't cleanup after themselves leaving junk in the VMs. Looks like that happened to a few bad apples:
https://chromium-swarm.appspot.com/bot?id=gce-trusty-02248837-us-west1-c-p63n
https://chromium-swarm.appspot.com/bot?id=gce-trusty-02248837-us-west1-b-vcq5
https://chromium-swarm.appspot.com/bot?id=gce-trusty-02248837-us-east1-b-6bf9

Compared to a bot w/ space available:
https://chromium-swarm.appspot.com/bot?id=gce-trusty-02248837-us-west1-b-j785

I'll cleanup the ones that are running out of space.

This looks like it may be an issue with a subset of VMs rather than a suite, e.g.
 https://chromium-swarm.appspot.com/bot?id=gce-trusty-02248837-us-west1-b-vcq5
 https://chromium-swarm.appspot.com/bot?id=gce-trusty-02248837-us-east1-b-6j5x

The retry bit is expected. The test emits nothing and the recipe doesn't know what to retry.

CI bot exhibited similar failures: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel

-> bpastene given #3

The broken bots have been respawned, and the success rate is climbing. The CQ should no longer be blocked. Lowering pri.

I grabbed one of the busted VM images before deleting the bots. I'll work on figuring out the exact nature of the disk space problems we were seeing.

It looks like "/var/spool/crash" is the culprit. On the broken VM, that contains a few GBs worth of chrome crash dumps. Since we test chrome in the CQ on these bots, a change that causes the browser to crash on CrOS would generate a dump in that dir for every test run. Add retries and they'd start accumulating until the VM runs out of disk space.

I filed bug 878526 a bit ago to improve how we cleanup the VM after each test. I'll add crash dumps to the list.

After rdfc6b1dc264ece4083ecfaa7a454e2e604000fd6, we're now clearing persistent crash and log dirs in every test. So we shouldn't run into this again. Closing this out.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dfc6b1dc264ece4083ecfaa7a454e2e604000fd6

commit dfc6b1dc264ece4083ecfaa7a454e2e604000fd6
Author: Ben Pastene <bpastene@chromium.org>
Date: Wed Sep 05 21:36:12 2018

Delete crash dumps and logs that persist from tests on cros VMs.

The following two dirs can accumulate cruft over a VM's life time:
- /var/log/chrome/
- /var/spool/crash/

This clears them before each test. Should become unnecessary when
crbug.com/878526 is closed out.

Bug:  880878 
Change-Id: I5414cb65016294938e9e68db495946bbb05cad48
Reviewed-on: https://chromium-review.googlesource.com/1208254
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#589007}
[modify] https://crrev.com/dfc6b1dc264ece4083ecfaa7a454e2e604000fd6/build/chromeos/run_vm_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/35a2d44eb0b1b201f7ec2616f6bc4539d9873a23

commit 35a2d44eb0b1b201f7ec2616f6bc4539d9873a23
Author: Ben Pastene <bpastene@chromium.org>
Date: Thu Sep 06 22:23:23 2018

Cleanup cros VM crash logs after test runs and not before.

This script that runs in the VM only runs after all the test binaries &
deps have been pushed. Consequently if a test generates crash dumps, it
won't get cleaned up until the next test pushes all its deps (which
might not fit in the VM since there's left-over crash dumps.)

This fixes that by ensuring that the test that generates the dumps also
cleans them up.

Bug:  880878 
Change-Id: Icfe1559d244053998cace31bfc19f5c1606770b9
Reviewed-on: https://chromium-review.googlesource.com/1211706
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#589336}
[modify] https://crrev.com/35a2d44eb0b1b201f7ec2616f6bc4539d9873a23/build/chromeos/run_vm_test.py

 Issue 880119  has been merged into this issue.

 Issue 884105  has been merged into this issue.