New issue
Advanced search Search tips

Issue 822789 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner: ----
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

VMTest SimpleTestUpdateAndVerify failed on amd64-generic-chromium-pfq

Project Member Reported by glevin@chromium.org, Mar 16 2018

Issue description

VMTests have been failing for awhile on amd64-generic-chromium-pfq (see  Issue 822007 ), but starting with https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-generic-chromium-pfq/builds/11749, the cause changed from failing login_* tests to failing SimpleTestUpdateAndVerify.

This looks (at least superficially) an awful lot like  Issue 805517 . +some of the same folks as on that one, and assigning to sheriff to help find proper owner.

Logs to follow...


 

Comment 1 by glevin@chromium.org, Mar 16 2018

From stdout:

2018/03/15 18:20:16 - au_worker.py - INFO    : SimpleTestUpdateAndVerify: Copied shared disk image /b/c/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/chromiumos_qemu_image.bin to /tmp/cbuildbot-tmpDrPFyM/chromiumos_qemu_disk.bin.vTIcLA.
2018/03/15 18:20:16 - au_worker.py - INFO    : SimpleTestUpdateAndVerify: Performing a delta update from /b/c/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/chromiumos_qemu_image.bin to /b/c/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/chromiumos_qemu_image.bin in a VM
18:23:01: INFO: Refreshing due to a 401 (attempt 1/2)
18:23:01: INFO: Refreshing access_token
2018/03/15 18:23:54 - au_worker.py - WARNING : None
INFO    : QEMU binary: /b/c/cbuild/repository/chroot/usr/bin/qemu-system-x86_64
INFO    : QEMU version: QEMU emulator version 2.6.0, Copyright (c) 2003-2008 Fabrice Bellard
Stopping the KVM instance
Saving memory snapshot to /b/c/cbuild/repository/chroot/tmp/cbuildbotGuaQDx/pfq_suite/test_harness/failed/SimpleTestUpdateAndVerify/1_update/chromiumos_qemu_mem.bin.EavCGa...
    freezing VM...
    saving memory, piping through cat...
    flushing disk buffers...
    asking KVM to quit...
    done.
E
======================================================================
ERROR: SimpleTestUpdateAndVerify (crostestutils.au_test_harness.au_test.AUTest)
Test that updates to itself.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/b/c/cbuild/repository/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_test.py", line 221, in SimpleTestUpdateAndVerify
    self.worker.PerformUpdate(target_image_path, target_image_path)
  File "/b/c/cbuild/repository/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_worker.py", line 153, in PerformUpdate
    key_to_use)
  File "/b/c/cbuild/repository/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/vm_au_worker.py", line 115, in UpdateImage
    self.RunUpdateCmd(cmd, log_directory)
  File "/b/c/cbuild/repository/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_worker.py", line 238, in RunUpdateCmd
    raise update_exception.UpdateException(result.returncode, 'Update failed')
UpdateException: Update failed
----------------------------------------------------------------------
Ran 1 test in 250.106s
FAILED (errors=1)

Comment 2 by glevin@chromium.org, Mar 16 2018

From amd64-generic-chromium-pfq/R67-10492.0.0-rc1/vm_test_results_2/pfq_suite/test_harness/failed/SimpleTestUpdateAndVerify/1_update/update.log:

flags:WARN flag name ([no]ssh_port) already defined
INFO    : QEMU binary: /b/c/cbuild/repository/chroot/usr/bin/qemu-system-x86_64
INFO    : QEMU version: QEMU emulator version 2.6.0, Copyright (c) 2003-2008 Fabrice Bellard
Starting a KVM instance
INFO    : Set kvm_smp to min(8, 40).
INFO    : Launching: /b/c/cbuild/repository/chroot/usr/bin/qemu-system-x86_64 -enable-kvm -m 8G -smp 8 -cpu SandyBridge,-invpcid,-tsc-deadline,check -vga virtio -pidfile /tmp/kvm.9227 -chardev pipe,id=control_pipe,path=/tmp/kvm.9227.monitor -serial file:/tmp/kvm.9227.serial -mon chardev=control_pipe -daemonize -net nic,model=virtio,vlan=9227 -display none -net user,hostfwd=tcp:127.0.0.1:9227-:22,vlan=9227 -drive file=/tmp/cbuildbot-tmpDrPFyM/chromiumos_qemu_disk.bin.vTIcLA,index=0,media=disk,cache=unsafe
WARNING: Image format was not specified for '/tmp/cbuildbot-tmpDrPFyM/chromiumos_qemu_disk.bin.vTIcLA' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
INFO    : KVM started with pid stored in /tmp/kvm.9227
INFO    : Serial output, if available, can be found here in /tmp/kvm.9227.serial

!!! You are using a deprecated script !!!

Please use 'cros flash' in the future. See 'cros flash -h' for the details.
More information available in the link below.
https://sites.google.com/a/chromium.org/dev/chromium-os/build/cros-flash

Iteration:  1 of 1
INFO    : Omaha URL:  --omaha_url=http://192.168.120.66:8080/update/cache/c432dc59b46eb9a746d261f5c492ed36_c432dc59b46eb9a746d261f5c492ed36
INFO    : Forcing update independent of the current version
INFO    : Waiting to initiate contact with the update_engine.
INFO    : Starting update using args --update --omaha_url=http://192.168.120.66:8080/update/cache/c432dc59b46eb9a746d261f5c492ed36_c432dc59b46eb9a746d261f5c492ed36
INFO    : Devserver handling ping.  Check dev_server.log for more info.
[0315/182255:INFO:update_engine_client.cc(486)] Forcing an update by setting app_version to ForcedUpdate.
[0315/182255:INFO:update_engine_client.cc(488)] Initiating update check and install.
[0315/182255:INFO:update_engine_client.cc(517)] Waiting for update to complete.
INFO    : State of updater has changed to: UPDATE_STATUS_UPDATE_AVAILABLE
INFO    : State of updater has changed to: UPDATE_STATUS_FINALIZING
.....[0315/182344:ERROR:update_engine_client.cc(232)] Update failed, current operation is UPDATE_STATUS_IDLE, last error code is ErrorCode::kPostinstallRunnerError(5)
WARNING : Update failed.  Dumping update_engine.log for debugging and/or bug reporting.
Failed to add the host to the list of known hosts (/tmp/image_to_live.jOGS/known_hosts).
no such identity: /tmp/image_to_live.jOGS/private_key: No such file or directory
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
Permission denied (publickey,keyboard-interactive).
Failed to add the host to the list of known hosts (/tmp/image_to_live.jOGS/known_hosts).
no such identity: /tmp/image_to_live.jOGS/private_key: No such file or directory
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
Permission denied (publickey,keyboard-interactive).
INFO    : QEMU binary: /b/c/cbuild/repository/chroot/usr/bin/qemu-system-x86_64
INFO    : QEMU version: QEMU emulator version 2.6.0, Copyright (c) 2003-2008 Fabrice Bellard
Persist requested.  Use --ssh_port 9227  --ssh_private_key /b/c/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/id_rsa  --kvm_pid /tmp/kvm.9227 to re-connect to it.
Labels: Hotlist-Deputy

Comment 4 by glevin@chromium.org, Mar 16 2018

Same problem also consistently appearing on betty-chrome-pfq, starting with
https://uberchromegw.corp.google.com/i/chromeos/builders/betty-chrome-pfq/builds/1315

Comment 5 by glevin@chromium.org, Mar 16 2018

And on amd64-generic-tot-chromium-pfq-informational, starting with
https://build.chromium.org/p/chromiumos.chromium/builders/amd64-generic-tot-chromium-pfq-informational/builds/15696

Comment 6 by glevin@chromium.org, Mar 19 2018

Status: WontFix (was: Assigned)
https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-generic-chromium-pfq/builds/11757
was the last observed instance of this on amd64-generic-chromium-pfq.  That was 2+ days and 18 builds ago.  Similarly for other two builders.  Problem seems to have cleared up.

Comment 7 by za...@chromium.org, Mar 19 2018

Owner: ----

Sign in to add a comment