VMTest SimpleTestUpdateAndVerify failed on amd64-generic-chromium-pfq |
||||
Issue descriptionVMTests have been failing for awhile on amd64-generic-chromium-pfq (see Issue 822007 ), but starting with https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-generic-chromium-pfq/builds/11749, the cause changed from failing login_* tests to failing SimpleTestUpdateAndVerify. This looks (at least superficially) an awful lot like Issue 805517 . +some of the same folks as on that one, and assigning to sheriff to help find proper owner. Logs to follow...
,
Mar 16 2018
From amd64-generic-chromium-pfq/R67-10492.0.0-rc1/vm_test_results_2/pfq_suite/test_harness/failed/SimpleTestUpdateAndVerify/1_update/update.log:
flags:WARN flag name ([no]ssh_port) already defined
INFO : QEMU binary: /b/c/cbuild/repository/chroot/usr/bin/qemu-system-x86_64
INFO : QEMU version: QEMU emulator version 2.6.0, Copyright (c) 2003-2008 Fabrice Bellard
Starting a KVM instance
INFO : Set kvm_smp to min(8, 40).
INFO : Launching: /b/c/cbuild/repository/chroot/usr/bin/qemu-system-x86_64 -enable-kvm -m 8G -smp 8 -cpu SandyBridge,-invpcid,-tsc-deadline,check -vga virtio -pidfile /tmp/kvm.9227 -chardev pipe,id=control_pipe,path=/tmp/kvm.9227.monitor -serial file:/tmp/kvm.9227.serial -mon chardev=control_pipe -daemonize -net nic,model=virtio,vlan=9227 -display none -net user,hostfwd=tcp:127.0.0.1:9227-:22,vlan=9227 -drive file=/tmp/cbuildbot-tmpDrPFyM/chromiumos_qemu_disk.bin.vTIcLA,index=0,media=disk,cache=unsafe
WARNING: Image format was not specified for '/tmp/cbuildbot-tmpDrPFyM/chromiumos_qemu_disk.bin.vTIcLA' and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.
INFO : KVM started with pid stored in /tmp/kvm.9227
INFO : Serial output, if available, can be found here in /tmp/kvm.9227.serial
!!! You are using a deprecated script !!!
Please use 'cros flash' in the future. See 'cros flash -h' for the details.
More information available in the link below.
https://sites.google.com/a/chromium.org/dev/chromium-os/build/cros-flash
Iteration: 1 of 1
INFO : Omaha URL: --omaha_url=http://192.168.120.66:8080/update/cache/c432dc59b46eb9a746d261f5c492ed36_c432dc59b46eb9a746d261f5c492ed36
INFO : Forcing update independent of the current version
INFO : Waiting to initiate contact with the update_engine.
INFO : Starting update using args --update --omaha_url=http://192.168.120.66:8080/update/cache/c432dc59b46eb9a746d261f5c492ed36_c432dc59b46eb9a746d261f5c492ed36
INFO : Devserver handling ping. Check dev_server.log for more info.
[0315/182255:INFO:update_engine_client.cc(486)] Forcing an update by setting app_version to ForcedUpdate.
[0315/182255:INFO:update_engine_client.cc(488)] Initiating update check and install.
[0315/182255:INFO:update_engine_client.cc(517)] Waiting for update to complete.
INFO : State of updater has changed to: UPDATE_STATUS_UPDATE_AVAILABLE
INFO : State of updater has changed to: UPDATE_STATUS_FINALIZING
.....[0315/182344:ERROR:update_engine_client.cc(232)] Update failed, current operation is UPDATE_STATUS_IDLE, last error code is ErrorCode::kPostinstallRunnerError(5)
WARNING : Update failed. Dumping update_engine.log for debugging and/or bug reporting.
Failed to add the host to the list of known hosts (/tmp/image_to_live.jOGS/known_hosts).
no such identity: /tmp/image_to_live.jOGS/private_key: No such file or directory
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
Permission denied (publickey,keyboard-interactive).
Failed to add the host to the list of known hosts (/tmp/image_to_live.jOGS/known_hosts).
no such identity: /tmp/image_to_live.jOGS/private_key: No such file or directory
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
Permission denied (publickey,keyboard-interactive).
INFO : QEMU binary: /b/c/cbuild/repository/chroot/usr/bin/qemu-system-x86_64
INFO : QEMU version: QEMU emulator version 2.6.0, Copyright (c) 2003-2008 Fabrice Bellard
Persist requested. Use --ssh_port 9227 --ssh_private_key /b/c/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/id_rsa --kvm_pid /tmp/kvm.9227 to re-connect to it.
,
Mar 16 2018
,
Mar 16 2018
Same problem also consistently appearing on betty-chrome-pfq, starting with https://uberchromegw.corp.google.com/i/chromeos/builders/betty-chrome-pfq/builds/1315
,
Mar 16 2018
And on amd64-generic-tot-chromium-pfq-informational, starting with https://build.chromium.org/p/chromiumos.chromium/builders/amd64-generic-tot-chromium-pfq-informational/builds/15696
,
Mar 19 2018
https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-generic-chromium-pfq/builds/11757 was the last observed instance of this on amd64-generic-chromium-pfq. That was 2+ days and 18 builds ago. Similarly for other two builders. Problem seems to have cleared up.
,
Mar 19 2018
|
||||
►
Sign in to add a comment |
||||
Comment 1 by glevin@chromium.org
, Mar 16 2018From stdout: 2018/03/15 18:20:16 - au_worker.py - INFO : SimpleTestUpdateAndVerify: Copied shared disk image /b/c/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/chromiumos_qemu_image.bin to /tmp/cbuildbot-tmpDrPFyM/chromiumos_qemu_disk.bin.vTIcLA. 2018/03/15 18:20:16 - au_worker.py - INFO : SimpleTestUpdateAndVerify: Performing a delta update from /b/c/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/chromiumos_qemu_image.bin to /b/c/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/chromiumos_qemu_image.bin in a VM 18:23:01: INFO: Refreshing due to a 401 (attempt 1/2) 18:23:01: INFO: Refreshing access_token 2018/03/15 18:23:54 - au_worker.py - WARNING : None INFO : QEMU binary: /b/c/cbuild/repository/chroot/usr/bin/qemu-system-x86_64 INFO : QEMU version: QEMU emulator version 2.6.0, Copyright (c) 2003-2008 Fabrice Bellard Stopping the KVM instance Saving memory snapshot to /b/c/cbuild/repository/chroot/tmp/cbuildbotGuaQDx/pfq_suite/test_harness/failed/SimpleTestUpdateAndVerify/1_update/chromiumos_qemu_mem.bin.EavCGa... freezing VM... saving memory, piping through cat... flushing disk buffers... asking KVM to quit... done. E ====================================================================== ERROR: SimpleTestUpdateAndVerify (crostestutils.au_test_harness.au_test.AUTest) Test that updates to itself. ---------------------------------------------------------------------- Traceback (most recent call last): File "/b/c/cbuild/repository/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_test.py", line 221, in SimpleTestUpdateAndVerify self.worker.PerformUpdate(target_image_path, target_image_path) File "/b/c/cbuild/repository/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_worker.py", line 153, in PerformUpdate key_to_use) File "/b/c/cbuild/repository/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/vm_au_worker.py", line 115, in UpdateImage self.RunUpdateCmd(cmd, log_directory) File "/b/c/cbuild/repository/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_worker.py", line 238, in RunUpdateCmd raise update_exception.UpdateException(result.returncode, 'Update failed') UpdateException: Update failed ---------------------------------------------------------------------- Ran 1 test in 250.106s FAILED (errors=1)