New issue
Advanced search Search tips

Issue 660894 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Failed to acquire lock on virtualenv in UnitTest stage

Project Member Reported by nxia@chromium.org, Oct 31 2016

Issue description

Saw this lock error a couple times.

https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/4123/steps/UnitTest/logs/stdio


chromite-0.0.2-r2648: 
chromite-0.0.2-r2648: 14:58:18: ERROR: ### LOG: scripts/prodmon_unittest
chromite-0.0.2-r2648: Traceback (most recent call last):
chromite-0.0.2-r2648:   File "scripts/prodmon_unittest", line 38, in <module>
chromite-0.0.2-r2648:     cros_build_lib.RunCommand([create_cmd])
chromite-0.0.2-r2648:   File "/mnt/host/source/chromite/lib/cros_build_lib.py", line 624, in RunCommand
chromite-0.0.2-r2648:     raise RunCommandError(msg, cmd_result)
chromite-0.0.2-r2648: chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: /mnt/host/source/chromite/venv/create_env.sh
chromite-0.0.2-r2648: Failed to acquire lock on virtualenv.
chromite-0.0.2-r2648: 
chromite-0.0.2-r2648: cwd=None
chromite-0.0.2-r2648: 
chromite-0.0.2-r2648: 
chromite-0.0.2-r2648: 14:58:18: ERROR: ### LOG: scripts/virtualenv_wrapper_unittest
chromite-0.0.2-r2648: Traceback (most recent call last):
chromite-0.0.2-r2648:   File "scripts/virtualenv_wrapper_unittest", line 38, in <module>
chromite-0.0.2-r2648:     cros_build_lib.RunCommand([create_cmd])
chromite-0.0.2-r2648:   File "/mnt/host/source/chromite/lib/cros_build_lib.py", line 624, in RunCommand
chromite-0.0.2-r2648:     raise RunCommandError(msg, cmd_result)
chromite-0.0.2-r2648: chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: /mnt/host/source/chromite/venv/create_env.sh
chromite-0.0.2-r2648: Failed to acquire lock on virtualenv.
chromite-0.0.2-r2648: 
chromite-0.0.2-r2648: cwd=None
chromite-0.0.2-r2648: 
chromite-0.0.2-r2648: 14:58:18: ERROR: The following 2 tests failed:
chromite-0.0.2-r2648:   scripts/prodmon_unittest
chromite-0.0.2-r2648:   scripts/virtualenv_wrapper_unittest
chromite-0.0.2-r2648:  * ERROR: chromeos-base/chromite-0.0.2-r2648::chromiumos failed (test phase):
chromite-0.0.2-r2648:  *   (no error message)
chromite-0.0.2-r2648:  * 
chromite-0.0.2-r2648:  * Call stack:
chromite-0.0.2-r2648:  *     ebuild.sh, line  93:  Called src_test
chromite-0.0.2-r2648:  *   environment, line 5870:  Called die
chromite-0.0.2-r2648:  * The specific snippet of code:
chromite-0.0.2-r2648:  *       cd "${CHROMITE_DIR}/cbuildbot" && sudo -u "${PORTAGE_USERNAME}" PATH="${CROS_WORKON_SRCROOT}/../depot_tools:${PATH}" ./run_tests || die
chromite-0.0.2-r2648:  * 
chromite-0.0.2-r2648:  * If you need support, post the output of `emerge --info '=chromeos-base/chromite-0.0.2-r2648::chromiumos'`,
chromite-0.0.2-r2648:  * the complete build log and the output of `emerge -pqv '=chromeos-base/chromite-0.0.2-r2648::chromiumos'`.
chromite-0.0.2-r2648:  * The complete build log is located at '/build/guado_moblab/tmp/portage/logs/chromeos-base:chromite-0.0.2-r2648:20161028-215527.log'.
chromite-0.0.2-r2648:  * For convenience, a symlink to the build log is located at '/build/guado_moblab/tmp/portage/chromeos-base/chromite-0.0.2-r2648/temp/build.log'.
chromite-0.0.2-r2648:  * The ebuild environment file is located at '/build/guado_moblab/tmp/portage/chromeos-base/chromite-0.0.2-r2648/temp/environment'.
chromite-0.0.2-r2648:  * Working directory: '/mnt/host/source/chromite/cbuildbot'
chromite-0.0.2-r2648:  * S: '/mnt/host/source/src/platform/../../chromite'
chromite-0.0.2-r2648: >>> Failed to emerge chromeos-base/chromite-0.0.2-r2648 for /build/guado_moblab/, Log file:
chromite-0.0.2-r2648: >>>  '/build/guado_moblab/tmp/portage/logs/chromeos-base:chromite-0.0.2-r2648:20161028-215527.log'
chromite-0.0.2-r2648: 
chromite-0.0.2-r2648:  * Messages for package chromeos-base/chromite-0.0.2-r2648 merged to /build/guado_moblab/:
chromite-0.0.2-r2648: 
chromite-0.0.2-r2648:  * ERROR: chromeos-base/chromite-0.0.2-r2648::chromiumos failed (test phase):
chromite-0.0.2-r2648:  *   (no error message)
chromite-0.0.2-r2648:  * 
chromite-0.0.2-r2648:  * Call stack:
chromite-0.0.2-r2648:  *     ebuild.sh, line  93:  Called src_test
chromite-0.0.2-r2648:  *   environment, line 5870:  Called die
chromite-0.0.2-r2648:  * The specific snippet of code:
chromite-0.0.2-r2648:  *       cd "${CHROMITE_DIR}/cbuildbot" && sudo -u "${PORTAGE_USERNAME}" PATH="${CROS_WORKON_SRCROOT}/../depot_tools:${PATH}" ./run_tests || die
chromite-0.0.2-r2648:  * 
chromite-0.0.2-r2648:  * If you need support, post the output of `emerge --info '=chromeos-base/chromite-0.0.2-r2648::chromiumos'`,
chromite-0.0.2-r2648:  * the complete build log and the output of `emerge -pqv '=chromeos-base/chromite-0.0.2-r2648::chromiumos'`.
chromite-0.0.2-r2648:  * The complete build log is located at '/build/guado_moblab/tmp/portage/logs/chromeos-base:chromite-0.0.2-r2648:20161028-215527.log'.
chromite-0.0.2-r2648:  * For convenience, a symlink to the build log is located at '/build/guado_moblab/tmp/portage/chromeos-base/chromite-0.0.2-r2648/temp/build.log'.
chromite-0.0.2-r2648:  * The ebuild environment file is located at '/build/guado_moblab/tmp/portage/chromeos-base/chromite-0.0.2-r2648/temp/environment'.
chromite-0.0.2-r2648:  * Working directory: '/mnt/host/source/chromite/cbuildbot'
chromite-0.0.2-r2648:  * S: '/mnt/host/source/src/platform/../../chromite'
 
Hmm. Maybe 30s is too little time on highly loaded builders. I'll up it to 90.
Project Member

Comment 3 by bugdroid1@chromium.org, Nov 1 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/5427be46ab10b7e14aaecc06a48c2844c412280a

commit 5427be46ab10b7e14aaecc06a48c2844c412280a
Author: Aviv Keshet <akeshet@chromium.org>
Date: Mon Oct 31 19:27:59 2016

venv: increase lock timeout to 90s

BUG= chromium:660894 
TEST=None

Change-Id: I78171b48bcad5c6834cc49acdb53ad5576f5859d
Reviewed-on: https://chromium-review.googlesource.com/405608
Commit-Ready: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Ningning Xia <nxia@chromium.org>

[modify] https://crrev.com/5427be46ab10b7e14aaecc06a48c2844c412280a/venv/create_env.sh

Comment 4 by nxia@chromium.org, Nov 11 2016

Status: Fixed (was: Untriaged)

Comment 5 by nxia@chromium.org, Nov 11 2016

Cc: -akes...@chromium.org
Owner: akes...@chromium.org

Sign in to add a comment