New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 260036 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Last visit > 30 days ago
Closed: Aug 2013
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

x86 alex canary - vmtests - failing devinstall_test - often

Reported by mtennant@chromium.org, Jul 12 2013

Issue description

OS: Chrome

What steps will reproduce the problem?
More often than not the x86 alex canary builder is failing the devinstall_test test within vmtests.  For example:

https://uberchromegw.corp.google.com/i/chromeos/builders/x86-alex%20canary/builds/3812/steps/VMTest%20%5Bx86-alex%5D/logs/stdio

4 of the last 6 builders failed this way.  This closes our tree.

Snippet:

OK
12:19:04: INFO: RunCommand: '/b/cbuild/x86-alex-canary-master/src/platform/crostestutils/devmode-test/devinstall_test.py' 'x86-alex' '/b/cbuild/x86-alex-canary-master/src/build/images/x86-alex/latest-cbuildbot/chromiumos_test_image.bin'
2013/07/12 12:19:04 - devinstall_test.py - INFO    : Setting up the image /b/cbuild/x86-alex-canary-master/src/build/images/x86-alex/R30-4391.0.0/chromiumos_test_image.bin for vm testing.
2013/07/12 12:19:04 - devinstall_test.py - INFO    : Making copy of the vm image /b/cbuild/x86-alex-canary-master/src/build/images/x86-alex/R30-4391.0.0/chromiumos_qemu_image.bin to manipulate.
2013/07/12 12:19:26 - devinstall_test.py - INFO    : Wiping /usr/local/bin from the image.
2013/07/12 12:19:38 - devinstall_test.py - INFO    : Starting the vm on port 58550.
2013/07/12 12:20:10 - devinstall_test.py - WARNING : Failed to connect to VM
2013/07/12 12:20:44 - devinstall_test.py - WARNING : Failed to connect to VM
2013/07/12 12:21:18 - devinstall_test.py - WARNING : Failed to connect to VM
2013/07/12 12:21:21 - devinstall_test.py - WARNING : Received error during cleanup
Traceback (most recent call last):
  File "/b/cbuild/x86-alex-canary-master/src/platform/crostestutils/devmode-test/devinstall_test.py", line 78, in Cleanup
    self._StopVM()
  File "/b/cbuild/x86-alex-canary-master/src/platform/crostestutils/devmode-test/devinstall_test.py", line 147, in _StopVM
    cros_build_lib.RunCommand(cmd, debug_level=logging.DEBUG)
  File "/b/cbuild/x86-alex-canary-master/chromite/lib/cros_build_lib.py", line 458, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1
Stopping the KVM instance
cat: /tmp/cbuildbot-tmpWLDJ_h/tmppC4w19DevModeTest/kvm_pid: No such file or directory
No kvm pid found to stop.

Failed command "'/b/cbuild/x86-alex-canary-master/src/scripts/bin/cros_stop_vm' '--kvm_pid' '/tmp/cbuildbot-tmpWLDJ_h/tmppC4w19DevModeTest/kvm_pid'", cwd=None, extra env=None
Traceback (most recent call last):
  File "/b/cbuild/x86-alex-canary-master/src/platform/crostestutils/devmode-test/devinstall_test.py", line 242, in <module>
    main()
  File "/b/cbuild/x86-alex-canary-master/src/platform/crostestutils/devmode-test/devinstall_test.py", line 233, in main
    test.PrepareTest()
  File "/b/cbuild/x86-alex-canary-master/src/platform/crostestutils/devmode-test/devinstall_test.py", line 167, in PrepareTest
    self._RobustlyStartVMWithSSH()
  File "/b/cbuild/x86-alex-canary-master/src/platform/crostestutils/devmode-test/devinstall_test.py", line 141, in _RobustlyStartVMWithSSH
    raise TestError('Max attempts to connect to VM exceeded')
__main__.TestError: Max attempts to connect to VM exceeded
12:21:21: ERROR: 
!!!VMTests failed!!!

I understand that Sosa has made recent changes to the logic of devinstall_test.  I am putting him as owner to triage.


 
Project Member

Comment 2 by bugdroid1@chromium.org, Jul 17 2013

Project: chromiumos/chromite
Branch : master
Author : Chris Sosa <sosa@chromium.org>
Commit : c296af87087008a98cba95e154c47fefee4ae492

Code Review +2: Matt Tennant, Ryan Cui
Verified    +1: Chris Sosa
Change-Id     : I70b390799079cc1e8dbb444e7e8068c041016b8d
Reviewed-at   : https://gerrit.chromium.org/gerrit/62354

Run devinstall_test verbosely.

When this test fails, there's not enough info to go by when their
is flakiness. Turn on all logging until we are confident all flakiness
is gone.

BUG= chromium:260036 
TEST=Ran it.

Commit-Queue: Chris Sosa <sosa@chromium.org>

M  buildbot/cbuildbot_commands.py

Comment 3 by sosa@chromium.org, Jul 17 2013

Labels: Iteration-86
Status: Started
Project Member

Comment 4 by bugdroid1@chromium.org, Jul 17 2013

Project: chromiumos/platform/crostestutils
Branch : master
Author : Chris Sosa <sosa@chromium.org>
Commit : 9eb333def4417ed220984a2e1224ef4db8590dbb

Code Review +2: Chris Sosa
Verified    +1: Chris Sosa
Change-Id     : Id1b08673eb28f700e43c50a5260358f093ddf333
Reviewed-at   : https://gerrit.chromium.org/gerrit/62351

Add more protection against flakiness of VM tests on builders.

BUG= chromium:260036 
TEST=Ran it, pyflakes, pylint.

Commit-Queue: Chris Sosa <sosa@chromium.org>

M  devmode-test/devinstall_test.py

Comment 5 by sosa@chromium.org, Jul 25 2013

Labels: Iteration-87

Comment 6 by sosa@chromium.org, Jul 25 2013

Labels: -Iteration-86
Punting to 87
Labels: -Iteration-87 Iteration-88 M-31
Seeing this or similar errors on all platforms in video tests, please see:
https://wmatrix.googleplex.com/video

Example:
http://chromegw/i/chromeos/builders/link%20canary/builds/2573/steps/VMTest/logs/stdio

Comment 9 by sosa@chromium.org, Aug 12 2013

#8 -- please file a different bug -- that's unrelated to this bug which deals with timing timeouts -- what you are pointing to an is likely an error in likely construction of image used on those builders.
Status: Fixed
Status: Verified

Sign in to add a comment