https://uberchromegw.corp.google.com/i/chromeos/builders/x86-mario-paladin/builds/10798
https://uberchromegw.corp.google.com/i/chromeos/builders/rambi-paladin/builds/10481

12:59:43: INFO: Running VM test pfq_suite.
12:59:43: INFO: RunCommand: bin/ctest '--board=x86-mario' '--type=vm' --no_graphics '--target_image=/b/cbuild/internal_master/src/build/images/x86-mario/latest-cbuildbot/chromiumos_test_image.bin' '--test_results_root=/b/cbuild/internal_master/chroot/tmp/cbuildbotg6E48x/test_harness' --quick_update --whitelist_chrome_crashes '--ssh_private_key=/b/cbuild/internal_master/src/build/images/x86-mario/latest-cbuildbot/id_rsa' in /b/cbuild/internal_master/src/scripts
2016/07/14 12:59:45 - ctest - INFO    : Could not find a latest image to use. Using target instead.
2016/07/14 12:59:45 - cros_build_lib.py - INFO    : RunCommand: ../platform/crostestutils/generate_test_payloads/cros_generate_test_payloads.py '--target=/b/cbuild/internal_master/src/build/images/x86-mario/latest-cbuildbot/chromiumos_test_image.bin' '--base=/b/cbuild/internal_master/src/build/images/x86-mario/latest-cbuildbot/chromiumos_test_image.bin' '--board=x86-mario' '--jobs=16' --basic_suite in /b/cbuild/internal_master/src/scripts
2016/07/14 12:59:47 - cros_generate_test_payloads.py - INFO    : Generating updates required for this test suite in parallel.
2016/07/14 12:59:47 - cros_generate_test_payloads.py - INFO    : Generating an unsigned delta payload from /b/cbuild/internal_master/src/build/images/x86-mario/latest-cbuildbot/chromiumos_qemu_image.bin to /b/cbuild/internal_master/src/build/images/x86-mario/latest-cbuildbot/chromiumos_qemu_image.bin and not patching the kernel.
2016/07/14 13:00:39 - cros_generate_test_payloads.py - INFO    : Dumping /b/cbuild/internal_master/src/build/images/x86-mario/latest-cbuildbot/update.cache
2016/07/14 13:00:39 - cros_build_lib.py - INFO    : RunCommand: /b/cbuild/internal_master/src/platform/crostestutils/au_test_harness/cros_au_test_harness.py '--base_image=/b/cbuild/internal_master/src/build/images/x86-mario/latest-cbuildbot/chromiumos_test_image.bin' '--target_image=/b/cbuild/internal_master/src/build/images/x86-mario/latest-cbuildbot/chromiumos_test_image.bin' '--board=x86-mario' '--type=vm' '--remote=0.0.0.0' --verbose '--jobs=16' '--ssh_private_key=/b/cbuild/internal_master/src/build/images/x86-mario/latest-cbuildbot/id_rsa' '--test_prefix=SimpleTestUpdateAndVerify' '--test_results_root=/b/cbuild/internal_master/chroot/tmp/cbuildbotg6E48x/test_harness' --no_graphics --whitelist_chrome_crashes in /b/cbuild/internal_master/src/scripts
2016/07/14 13:00:41 - cros_au_test_harness.py - INFO    : Loading update cache from /b/cbuild/internal_master/src/build/images/x86-mario/latest-cbuildbot/update.cache
2016/07/14 13:00:41 - dev_server_wrapper.py - DEBUG   : Retrieving http://127.0.0.1:8080/check_health
2016/07/14 13:00:41 - cros_build_lib.py - DEBUG   : RunCommand: /b/cbuild/internal_master/chromite/bin/cros_sdk --no-ns-pid -- sudo 'CROS_CACHEDIR=/b/cbuild/internal_master/.cache' 'CROS_SUDO_KEEP_ALIVE=unknown' -- start_devserver --pidfile /tmp/cbuildbotg6E48x/test_harness/devserver_wrapperIXUswg --logfile /tmp/cbuildbotg6E48x/test_harness/dev_server.log '--port=8080' --critical_update in /b/cbuild/internal_master
2016/07/14 13:00:46 - dev_server_wrapper.py - DEBUG   : Retrieving http://127.0.0.1:8080/check_health
2016/07/14 13:00:51 - dev_server_wrapper.py - DEBUG   : Retrieving http://127.0.0.1:8080/check_health
2016/07/14 13:00:51 - cros_au_test_harness.py - INFO    : Running tests in test suite in parallel.
2016/07/14 13:01:12 - au_worker.py - INFO    : SimpleTestUpdateAndVerify: Copied shared disk image /b/cbuild/internal_master/src/build/images/x86-mario/latest-cbuildbot/chromiumos_qemu_image.bin to /tmp/cbuildbot-tmprgbTfl/chromiumos_qemu_disk.bin.ZAGCIN.
2016/07/14 13:01:12 - au_worker.py - INFO    : SimpleTestUpdateAndVerify: Performing a delta update from /b/cbuild/internal_master/src/build/images/x86-mario/latest-cbuildbot/chromiumos_qemu_image.bin to /b/cbuild/internal_master/src/build/images/x86-mario/latest-cbuildbot/chromiumos_qemu_image.bin in a VM
2016/07/14 13:02:13 - au_worker.py - WARNING : None
Stopping the KVM instance
Saving memory snapshot to /b/cbuild/internal_master/chroot/tmp/cbuildbotg6E48x/test_harness/failed/SimpleTestUpdateAndVerify/1_update/chromiumos_qemu_mem.bin.6nyCT_...
    freezing VM...
    saving memory, piping through cat...
    flushing disk buffers...
    asking KVM to quit...
    done.
E
======================================================================
ERROR: SimpleTestUpdateAndVerify (crostestutils.au_test_harness.au_test.AUTest)
Test that updates to itself.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/b/cbuild/internal_master/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_test.py", line 221, in SimpleTestUpdateAndVerify
    self.worker.PerformUpdate(target_image_path, target_image_path)
  File "/b/cbuild/internal_master/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_worker.py", line 153, in PerformUpdate
    key_to_use)
  File "/b/cbuild/internal_master/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/vm_au_worker.py", line 110, in UpdateImage
    self.RunUpdateCmd(cmd, log_directory)
  File "/b/cbuild/internal_master/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_worker.py", line 238, in RunUpdateCmd
    raise update_exception.UpdateException(result.returncode, 'Update failed')
UpdateException: Update failed

----------------------------------------------------------------------
Ran 1 test in 106.084s

FAILED (errors=1)
2016/07/14 13:02:37 - cros_build_lib.py - ERROR   : <class 'chromite.lib.parallel.BackgroundFailure'>: Test harness failed. See logs for details.
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 440, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/cbuild/internal_master/src/platform/crostestutils/au_test_harness/cros_au_test_harness.py", line 77, in run
    raise parallel.BackgroundFailure(msg)
BackgroundFailure: Test harness failed. See logs for details.

2016/07/14 13:02:37 - dev_server_wrapper.py - DEBUG   : Stopping devserver instance with pid 3130
2016/07/14 13:02:37 - cros_build_lib.py - DEBUG   : RunCommand: sudo 'CROS_CACHEDIR=/b/cbuild/internal_master/.cache' 'CROS_SUDO_KEEP_ALIVE=unknown' -- kill 3130
13:02:38: ERROR: 
!!!VMTests failed!!!

Logs are uploaded in the corresponding vm_test_results_2. This can be found
by clicking on the artifacts link in the "Report" Stage. Specifically look
for the test_harness/failed for the failing tests. For more
particulars, please refer to which test failed i.e. above see the
individual test that failed -- or if an update failed, check the
corresponding update directory.

13:02:38: INFO: RunCommand: lsof /b/cbuild/internal_master/chroot/tmp/cbuildbotg6E48x/test_harness/failed/SimpleTestUpdateAndVerify/1_update/chromiumos_qemu_disk.bin.ZAGCIN in /b/cbuild/internal_master/chroot/tmp/cbuildbotg6E48x/test_harness/failed/SimpleTestUpdateAndVerify/1_update
13:03:26: INFO: RunCommand: lsof /b/cbuild/internal_master/chroot/tmp/cbuildbotg6E48x/test_harness/failed/SimpleTestUpdateAndVerify/1_update/chromiumos_qemu_mem.bin.6nyCT_ in /b/cbuild/internal_master/chroot/tmp/cbuildbotg6E48x/test_harness/failed/SimpleTestUpdateAndVerify/1_update
13:03:33: INFO: Uploading artifacts to Google Storage...

@@@STEP_LINK@vm_disk: failed_SimpleTestUpdateAndVerify_1_update_chromiumos_qemu_disk.bin.ZAGCIN.tar@https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/x86-mario-paladin/R54-8588.0.0-rc1/failed_SimpleTestUpdateAndVerify_1_update_chromiumos_qemu_disk.bin.ZAGCIN.tar@@@

@@@STEP_LINK@vm_memory: failed_SimpleTestUpdateAndVerify_1_update_chromiumos_qemu_mem.bin.6nyCT_.tar@https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/x86-mario-paladin/R54-8588.0.0-rc1/failed_SimpleTestUpdateAndVerify_1_update_chromiumos_qemu_mem.bin.6nyCT_.tar@@@
13:03:33: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/x86-mario-paladin/R54-8588.0.0-rc1/failed_SimpleTestUpdateAndVerify_1_update_chromiumos_qemu_disk.bin.ZAGCIN.tar gs://chromeos-image-archive/x86-mario-paladin/R54-8588.0.0-rc1/failed_SimpleTestUpdateAndVerify_1_update_chromiumos_qemu_disk.bin.ZAGCIN.tar
13:03:47: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/x86-mario-paladin/R54-8588.0.0-rc1/UPLOADED gs://chromeos-image-archive/x86-mario-paladin/R54-8588.0.0-rc1/UPLOADED
13:03:33: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/x86-mario-paladin/R54-8588.0.0-rc1/failed_SimpleTestUpdateAndVerify_1_update_chromiumos_qemu_mem.bin.6nyCT_.tar gs://chromeos-image-archive/x86-mario-paladin/R54-8588.0.0-rc1/failed_SimpleTestUpdateAndVerify_1_update_chromiumos_qemu_mem.bin.6nyCT_.tar
13:03:39: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/x86-mario-paladin/R54-8588.0.0-rc1/UPLOADED gs://chromeos-image-archive/x86-mario-paladin/R54-8588.0.0-rc1/UPLOADED
13:03:48: INFO: RunCommand: lsof +D /b/cbuild/internal_master/buildbot_archive/x86-mario-paladin/R54-8588.0.0-rc1/vm_test_results_2/. in /b/cbuild/internal_master/buildbot_archive/x86-mario-paladin/R54-8588.0.0-rc1/vm_test_results_2
13:03:48: INFO: Waiting up to forever for breakpad symbols ...
13:03:48: INFO: Uploading artifacts to Google Storage...

@@@STEP_LINK@vm_test_results_2@https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/x86-mario-paladin/R54-8588.0.0-rc1/vm_test_results_2@@@

@@@STEP_LINK@vm_test_results_2.tgz@https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/x86-mario-paladin/R54-8588.0.0-rc1/vm_test_results_2.tgz@@@
13:03:48: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -r -e -- /b/cbuild/internal_master/buildbot_archive/x86-mario-paladin/R54-8588.0.0-rc1/vm_test_results_2 gs://chromeos-image-archive/x86-mario-paladin/R54-8588.0.0-rc1/vm_test_results_2
13:03:49: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/x86-mario-paladin/R54-8588.0.0-rc1/UPLOADED gs://chromeos-image-archive/x86-mario-paladin/R54-8588.0.0-rc1/UPLOADED
13:03:48: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/x86-mario-paladin/R54-8588.0.0-rc1/vm_test_results_2.tgz gs://chromeos-image-archive/x86-mario-paladin/R54-8588.0.0-rc1/vm_test_results_2.tgz
13:03:49: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -