New issue
Advanced search Search tips

Issue 884105 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 880878
Owner: ----
Closed: Sep 18
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

"cros_vm_sanity_test" is flaky

Project Member Reported by chromium...@appspot.gserviceaccount.com, Sep 14

Issue description

"cros_vm_sanity_test" is flaky.

This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is infrastructure-related, please add Infra-Troopers label and change issue status to Untriaged. When done, please remove the issue from Sheriff Bug Queue by removing the Sheriff-Chromium label.

We have detected 14 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHgsSBUZsYWtlIhNjcm9zX3ZtX3Nhbml0eV90ZXN0DA.

Flaky tests should be disabled within 30 minutes unless culprit CL is found and reverted. Please see more details here: https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/sheriffing-bug-queues#triaging-auto-filed-flakiness-bugs

This flaky test/step was previously tracked in  issue 880878 .
 
Test logs look like:

+-------------------------------------------------------------------------------------+
| Shard 0  https://chromium-swarm.appspot.com/user/task/3fed0d695b4f5c10              |
+-------------------------------------------------------------------------------------+
Additional test environment:
    CHROME_DEVEL_SANDBOX=/opt/chromium/chrome_sandbox
    CHROME_HEADLESS=1
    LANG=en_US.UTF-8
Command: bin/run_cros_vm_sanity_test --vm-logs-dir=/b/s/w/iofZ1Yiq --test-launcher-summary-output=/b/s/w/iofZ1Yiq/output.json
INFO:root:Running the following command on the host:
INFO:root:/b/s/w/ir/third_party/chromite/bin/cros_run_vm_test --start --board amd64-generic --cache-dir /b/s/w/ir/build/cros_cache --results-src /var/log/ --results-dest-dir /b/s/w/iofZ1Yiq --deploy --build-dir out_amd64-generic/Release
INFO:root:########################################
INFO:root:Test attempt #0
INFO:root:########################################
11:40:37: INFO: /b/s/w/itEiIyhu/cros_vm_9222/kvm.pid does not exist.
11:40:37: DEBUG: Stop VM
11:40:37: INFO: /b/s/w/itEiIyhu/cros_vm_9222/kvm.pid does not exist.
11:40:37: DEBUG: Start VM
11:40:37: DEBUG: QEMU path: /b/s/w/ir/build/cros_cache/chrome-sdk/tarballs/amd64-generic+11042.0.0+qemu/usr/bin/qemu-system-x86_64
11:40:37: INFO: RunCommand: /b/s/w/ir/build/cros_cache/chrome-sdk/tarballs/amd64-generic+11042.0.0+qemu/usr/bin/qemu-system-x86_64 --version
11:40:37: INFO: QEMU version 2.6.0
11:40:37: DEBUG: VM image path: /b/s/w/ir/build/cros_cache/chrome-sdk/tarballs/amd64-generic+11042.0.0+chromiumos_qemu_image.tar.xz/chromiumos_qemu_image.bin
11:40:37: INFO: Pid file: /b/s/w/itEiIyhu/cros_vm_9222/kvm.pid
11:40:37: INFO: RunCommand: /b/s/w/ir/build/cros_cache/chrome-sdk/tarballs/amd64-generic+11042.0.0+qemu/usr/bin/qemu-system-x86_64 -m 8G -smp 8 -vga virtio -daemonize -usbdevice tablet -pidfile /b/s/w/itEiIyhu/cros_vm_9222/kvm.pid -chardev 'pipe,id=control_pipe,path=/b/s/w/itEiIyhu/cros_vm_9222/kvm.monitor' -serial file:/b/s/w/itEiIyhu/cros_vm_9222/kvm.monitor.serial -mon 'chardev=control_pipe' -cpu SandyBridge,-invpcid,-tsc-deadline,check -device 'virtio-net,netdev=eth0' -netdev 'user,id=eth0,net=10.0.2.0/27,hostfwd=tcp:127.0.0.1:9222-:22' -drive 'file=/b/s/w/ir/build/cros_cache/chrome-sdk/tarballs/amd64-generic+11042.0.0+chromiumos_qemu_image.tar.xz/chromiumos_qemu_image.bin,index=0,media=disk,cache=unsafe,format=raw' -enable-kvm
VNC server running on '127.0.0.1;5900'
11:40:38: INFO: RunCommand: ssh -p 9222 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /b/s/w/itEiIyhu/ssh-tmpqZKoW9/testing_rsa root@localhost -- echo
11:41:08: INFO: (stdout):
Connection timed out during banner exchange
11:41:08: ERROR: Error connecting to device localhost
11:41:08: DEBUG: <class 'chromite.lib.remote_access.SSHConnectionError'>(None)
11:41:08: DEBUG: Retrying in 5.000000 (5.000000 + jitter 0.000000) seconds ...
11:41:13: INFO: RunCommand: ssh -p 9222 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /b/s/w/itEiIyhu/ssh-tmpqZKoW9/testing_rsa root@localhost -- echo
11:41:13: INFO: (stdout):
Warning: Permanently added '[localhost]:9222' (ED25519) to the list of known hosts.
Warning: Permanently added '[localhost]:9222' (ED25519) to the list of known hosts.
11:41:13: INFO: RunCommand: deploy_chrome --force --build-dir out_amd64-generic/Release --process-timeout 180 --to localhost --port 9222 --board amd64-generic --cache-dir /b/s/w/ir/build/cros_cache
11:41:13: INFO: Preparing device
11:41:13: INFO: Testing connection to the device...
11:41:13: INFO: Shutting down Chrome...
11:41:17: INFO: SDK: Fetching gs://chromeos-image-archive/amd64-generic-full/R71-11042.0.0-rc2/environment_chromeos-base_chromeos-chrome.tar.xz
11:41:23: INFO: Copying Chrome to /opt/google/chrome on device...
rsync is not found on the device.
Run dev_install on the device to get rsync installed
cros_run_vm_test: Unhandled exception:
Traceback (most recent call last):
  File "/b/s/w/ir/third_party/chromite/bin/cros_run_vm_test", line 169, in <module>
    DoMain()
  File "/b/s/w/ir/third_party/chromite/bin/cros_run_vm_test", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/s/w/ir/third_party/chromite/lib/commandline.py", line 912, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/s/w/ir/third_party/chromite/scripts/cros_run_vm_test.py", line 407, in main
    return VMTest(ParseCommandLine(argv)).Run()
  File "/b/s/w/ir/third_party/chromite/scripts/cros_run_vm_test.py", line 72, in Run
    self._Deploy()
  File "/b/s/w/ir/third_party/chromite/scripts/cros_run_vm_test.py", line 124, in _Deploy
    cros_build_lib.RunCommand(deploy_cmd)
  File "/b/s/w/ir/third_party/chromite/lib/cros_build_lib.py", line 647, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: deploy_chrome --force --build-dir out_amd64-generic/Release --process-timeout 180 --to localhost --port 9222 --board amd64-generic --cache-dir /b/s/w/ir/build/cros_cache
cmd=['deploy_chrome', '--force', '--build-dir', 'out_amd64-generic/Release', '--process-timeout', '180', '--to', 'localhost', '--port', '9222', '--board', 'amd64-generic', '--cache-dir', '/b/s/w/ir/build/cros_cache']
11:41:24: DEBUG: Stop VM
11:41:24: INFO: Killing 2754.
11:41:24: INFO: RunCommand: kill -9 2754
11:41:24: INFO: Time elapsed 0:00:46.710863.
INFO:root:Test exitted with 1.
Components: Infra
Labels: Infa-Troopers
No new flakes for 3 days. Not quite sure why the error is happening. Looks like a VM/Infra issue to me though.
Mergedinto: 880878
Status: Duplicate (was: Untriaged)
This was a short reoccurrence of  bug 880878 . The problem (no space left on VM) should be fixed for good w/  bug 881638 .

Sign in to add a comment