New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 848476 link

Starred by 2 users

Issue metadata

Status: WontFix
Owner: ----
Closed: Jul 13
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug
Build-Toolchain



Sign in to add a comment

VMTest stage fails on amd64-generic-asan builder

Project Member Reported by rahulchaudhry@chromium.org, May 31 2018

Issue description

See example build: https://ci.chromium.org/p/chromeos/builds/b8945033862851900736

The VMTest stage fails with:

======================================================================
FAIL: SimpleTestVerify (crostestutils.au_test_harness.au_test.AUTest)
Test that only verifies the target image.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/b/swarming/w/ir/cache/cbuild/repository/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_test.py", line 232, in SimpleTestVerify
    self.assertTrue(self.worker.VerifyImage(self))
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 358.730s
FAILED (failures=1)
2018/05/31 01:11:31 - cros_build_lib.py - ERROR   : Test harness failed.
2018/05/31 01:11:32 - cros_build_lib.py - ERROR   : /b/swarming/w/ir/cache/cbuild/repository/src/platform/crostestutils/au_test_harness/cros_au_test_harness.py --base_image=/b/swarming/w/ir/cache/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/chromiumos_test_image.bin --target_image=/b/swarming/w/ir/cache/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/chromiumos_test_image.bin --board=amd64-generic --type=vm --remote=0.0.0.0 --verbose --jobs=7 --ssh_private_key=/b/swarming/w/ir/cache/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/id_rsa --ssh_port=9228 --verify_suite_name=smoke --test_prefix=SimpleTestVerify --test_results_root=/b/swarming/w/ir/cache/cbuild/repository/chroot/tmp/cbuildbotAeWnok/smoke/test_harness --no_graphics --whitelist_chrome_crashes exited with code 1: None
01:11:33: ERROR: 
!!!VMTests failed!!!

 
Components: Tests
Cc: wonderfly@google.com pwang@chromium.org sosa@chromium.org
Need help finding an owner for this test.
Cc: akes...@chromium.org
This is the entry point for all VMTest tests. From the logs it looks like the builder was not able to connect to the KVM instance. I don't know if there are more logs but you could run the failed command locally and see if it's reproducible. Also, adding Aviv who might know the better person to ask for help.

`bin/ctest --board=amd64-generic --type=vm --no_graphics --verbose --target_image=/b/swarming/w/ir/cache/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/chromiumos_test_image.bin --test_results_root=/b/swarming/w/ir/cache/cbuild/repository/chroot/tmp/cbuildbotAeWnok/smoke/test_harness --ssh_port=9228 --only_verify --suite=smoke --whitelist_chrome_crashes --ssh_private_key=/b/swarming/w/ir/cache/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/id_rsa`


2018/05/31 01:06:56 - cros_build_lib.py - INFO    : RunCommand: ./bin/cros_run_vm_test '--board=amd64-generic' '--image_path=/b/swarming/w/ir/tmp/cbuildbot-tmpx3iz1_/chromiumos_qemu_disk.bin.PkWiRz' --persist '--kvm_pid=/tmp/kvm.9228' '--ssh_port=9228' '--results_dir_root=/tmp/cbuildbotAeWnok/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests' '--verbose=0' suite:smoke --no_graphics --whitelist_chrome_crashes '--ssh_private_key=/b/swarming/w/ir/cache/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/id_rsa' in /b/swarming/w/ir/cache/cbuild/repository/src/scripts
INFO    : QEMU binary: /b/swarming/w/ir/cache/cbuild/repository/chroot/usr/bin/qemu-system-x86_64
INFO    : QEMU version: QEMU emulator version 2.6.0, Copyright (c) 2003-2008 Fabrice Bellard
Starting a KVM instance
INFO    : Set kvm_smp to min(8, 32).
INFO    : Launching: /b/swarming/w/ir/cache/cbuild/repository/chroot/usr/bin/qemu-system-x86_64 -enable-kvm -m 8G -smp 8 -cpu SandyBridge,-invpcid,-tsc-deadline,check -vga virtio -pidfile /tmp/kvm.9228 -chardev pipe,id=control_pipe,path=/tmp/kvm.9228.monitor -serial file:/tmp/kvm.9228.serial -mon chardev=control_pipe -daemonize -device virtio-net,netdev=eth0 -display none -netdev user,id=eth0,net=10.0.2.0/27,hostfwd=tcp:127.0.0.1:9228-:22 -drive file=/b/swarming/w/ir/tmp/cbuildbot-tmpx3iz1_/chromiumos_qemu_disk.bin.PkWiRz,index=0,media=disk,cache=unsafe
WARNING: Image format was not specified for '/b/swarming/w/ir/tmp/cbuildbot-tmpx3iz1_/chromiumos_qemu_disk.bin.PkWiRz' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
INFO    : KVM started with pid stored in /tmp/kvm.9228
INFO    : Serial output, if available, can be found in /tmp/kvm.9228.serial
Could not initiate first contact with remote host
Connection timed out during banner exchange
Connection timed out during banner exchange
Failed to connect to virtual machine, retrying ... 
INFO    : QEMU binary: /b/swarming/w/ir/cache/cbuild/repository/chroot/usr/bin/qemu-system-x86_64
INFO    : QEMU version: QEMU emulator version 2.6.0, Copyright (c) 2003-2008 Fabrice Bellard
Persist requested.  Use --ssh_port 9228  --ssh_private_key /b/swarming/w/ir/cache/cbuild/repository/src/build/images/amd64-generic/latest-cbuildbot/id_rsa  --kvm_pid /tmp/kvm.9228 to re-connect to it.
INFO    : QEMU binary: /b/swarming/w/ir/cache/cbuild/repository/chroot/usr/bin/qemu-system-x86_64
INFO    : QEMU version: QEMU emulator version 2.6.0, Copyright (c) 2003-2008 Fabrice Bellard
Using a pre-created KVM instance specified by /tmp/kvm.9228.
Could not initiate first contact with remote host
Connection timed out during banner exchange
Connection timed out during banner exchange
Failed to connect to virtual machine, retrying ... 

Comment 5 by pwang@chromium.org, Jun 1 2018

Yes, as wonderfly@ said, the machine has problem connecting to its KVM instance.
It reminds me of a past issue crbug.com/808045 that it has network ip address collision once builder to GCE instance. Maybe it is related, but at current stage, we should run the failed command locally and check if it's reproducible.

Sign in to add a comment