It looks like the VM ran out of disk space in /usr/local:

2018/12/15 04:56:41 Command line: /usr/bin/tast run -build=false -resultsdir=/tmp/cbuildbot_FsLlA/tast_vm_paladin/out.613427146/meta.RunTests/subtest_results -keyfile=/home/chrome-bot/trunk/chromite/ssh_keys/testing_rsa -keydir=/home/chrome-bot/.ssh -remoterunner=/usr/bin/remote_test_runner -remotebundledir=/usr/libexec/tast/bundles/remote -remotedatadir=/usr/share/tast/data 127.0.0.1:9222 meta.LocalFiles meta.LocalPanic meta.RemoteFiles
2018/12/15 04:56:41 Writing results to /tmp/cbuildbot_FsLlA/tast_vm_paladin/out.613427146/meta.RunTests/subtest_results
2018/12/15 04:56:41 Connecting to 127.0.0.1:9222
2018/12/15 04:56:41 [06:56:40.474] Devserver status: using pseudo client
2018/12/15 04:56:41 [06:56:40.484] Found 1 external linked data file(s), need to download 1
2018/12/15 04:56:41 [06:56:40.485] Downloading gs://chromiumos-test-assets-public/tast/cros/meta/local_files_external_20180811.txt
2018/12/15 04:56:41 [06:56:40.696] Failed to download gs://chromiumos-test-assets-public/tast/cros/meta/local_files_external_20180811.txt: write /usr/local/share/tast/data/.external-download.554690545: no space left on device
2018/12/15 04:56:41 [06:56:40.697] Failed to download some external data files, but continuing anyway; corresponding tests will fail
...

Achuith, is this something you've seen recently? I'm not sure much headroom we usually have in amd64-generic-paladin.

These crashes probably aren't helping:

2018/12/15 04:56:44 Collecting system information
2018/12/15 04:56:44 /home/chronos/crash/f00ddd83-dfd2-41dc-436821af-f33294f3.dmp: skipping; too many files
2018/12/15 04:56:44 /var/spool/crash/cros_camera_service.20181215.065429.7091.dmp: skipping; too many files
2018/12/15 04:56:44 /var/spool/crash/cros_camera_service.20181215.065456.7348.dmp: skipping; too many files
2018/12/15 04:56:44 /var/spool/crash/cros_camera_service.20181215.065128.5274.dmp: skipping; too many files
2018/12/15 04:56:44 /var/spool/crash/cros_camera_service.20181215.065608.9308.dmp: skipping; too many files
2018/12/15 04:56:44 /var/spool/crash/cros_camera_service.20181215.065422.6315.dmp: skipping; too many files
2018/12/15 04:56:44 /home/chronos/crash/db0b05fc-19ab-4acf-10c9db91-ed98fb19.dmp: skipping; too many files
2018/12/15 04:56:44 /var/spool/crash/cros_camera_service.20181215.045001.1896.dmp: skipping; too many files
2018/12/15 04:56:44 /var/spool/crash/cros_camera_service.20181215.065137.5647.dmp: skipping; too many files
2018/12/15 04:56:44 /var/spool/crash/cros_camera_service.20181215.045013.3262.dmp: skipping; too many files

cros_camera_service is probably  issue 914110 . What's the status of fixing that?

I think we've been running into these disk space issues elsewhere. I'm planning to look at expanding the disk space available in the VM. Of course, if we're creating large dump files and crashing a lot, we'll still run out.

It looks like amd64-generic-paladin hit the same issue during TastVmTest in the current CQ run: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8926861204312504128

Is there anything we can do short-term to work around this?

Can we disable the camera tests?

> Can we disable the camera tests?

I don't think that any particular test is triggering the cros_camera_service crashes. Per  issue 914110 , it sounds like it just happens when the UI job gets stopped on a VM.

The fix for  issue 914110  landed, so we should be OK for the short term - I'll keep an eye out for any new occurrences.

Expanding the VM disk space still seems like a good idea.

I'm going to WontFix this since it doesn't seem to be an issue in the test itself.

Achuith, is there a bug tracking increasing the stateful partition size for VM images, assuming that that's the limit that we're hitting here?

I'm using crbug.com/913153 to track the VM size issue.