New issue
Advanced search Search tips

Issue 678677 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Feb 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Chromite unittest error at "Failed to acquire lock on virtualenv"

Project Member Reported by nxia@chromium.org, Jan 5 2017

Issue description

Saw this error several places in paladin, should extend the waiting time or increase the retry times?


https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fauron-paladin%2F9177%2F%2B%2Frecipes%2Fsteps%2FUnitTest%2F0%2Fstdout


https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fauron-paladin%2F9174%2F%2B%2Frecipes%2Fsteps%2FUnitTest%2F0%2Fstdout



chromite-0.0.2-r2735: 06:20:25: ERROR: The following 2 tests failed:
chromite-0.0.2-r2735:   scripts/export_to_gcloud_unittest
chromite-0.0.2-r2735:   scripts/virtualenv_wrapper_unittest
chromite-0.0.2-r2735:  * ERROR: chromeos-base/chromite-0.0.2-r2735::chromiumos failed (test phase):
chromite-0.0.2-r2735:  *   (no error message)
chromite-0.0.2-r2735:  * 
chromite-0.0.2-r2735:  * Call stack:
chromite-0.0.2-r2735:  *     ebuild.sh, line  93:  Called src_test
chromite-0.0.2-r2735:  *   environment, line 5880:  Called die
chromite-0.0.2-r2735:  * The specific snippet of code:
chromite-0.0.2-r2735:  *       cd "${CHROMITE_DIR}/cbuildbot" && sudo -u "${PORTAGE_USERNAME}" PATH="${CROS_WORKON_SRCROOT}/../depot_tools:${PATH}" ./run_tests || die
chromite-0.0.2-r2735:  * 
chromite-0.0.2-r2735:  * If you need support, post the output of `emerge --info '=chromeos-base/chromite-0.0.2-r2735::chromiumos'`,
chromite-0.0.2-r2735:  * the complete build log and the output of `emerge -pqv '=chromeos-base/chromite-0.0.2-r2735::chromiumos'`.
chromite-0.0.2-r2735:  * The complete build log is located at '/build/auron/tmp/portage/logs/chromeos-base:chromite-0.0.2-r2735:20170105-141709.log'.
chromite-0.0.2-r2735:  * For convenience, a symlink to the build log is located at '/build/auron/tmp/portage/chromeos-base/chromite-0.0.2-r2735/temp/build.log'.
chromite-0.0.2-r2735:  * The ebuild environment file is located at '/build/auron/tmp/portage/chromeos-base/chromite-0.0.2-r2735/temp/environment'.
chromite-0.0.2-r2735:  * Working directory: '/mnt/host/source/chromite/cbuildbot'
chromite-0.0.2-r2735:  * S: '/mnt/host/source/src/platform/../../chromite'
chromite-0.0.2-r2735: >>> Failed to emerge chromeos-base/chromite-0.0.2-r2735 for /build/auron/, Log file:
chromite-0.0.2-r2735: >>>  '/build/auron/tmp/portage/logs/chromeos-base:chromite-0.0.2-r2735:20170105-141709.log'
chromite-0.0.2-r2735: 
chromite-0.0.2-r2735:  * Messages for package chromeos-base/chromite-0.0.2-r2735 merged to /build/auron/:
chromite-0.0.2-r2735: 
chromite-0.0.2-r2735:  * ERROR: chromeos-base/chromite-0.0.2-r2735::chromiumos failed (test phase):
chromite-0.0.2-r2735:  *   (no error message)
chromite-0.0.2-r2735:  * 
chromite-0.0.2-r2735:  * Call stack:
chromite-0.0.2-r2735:  *     ebuild.sh, line  93:  Called src_test
chromite-0.0.2-r2735:  *   environment, line 5880:  Called die
chromite-0.0.2-r2735:  * The specific snippet of code:
chromite-0.0.2-r2735:  *       cd "${CHROMITE_DIR}/cbuildbot" && sudo -u "${PORTAGE_USERNAME}" PATH="${CROS_WORKON_SRCROOT}/../depot_tools:${PATH}" ./run_tests || die
chromite-0.0.2-r2735:  * 
chromite-0.0.2-r2735:  * If you need support, post the output of `emerge --info '=chromeos-base/chromite-0.0.2-r2735::chromiumos'`,
chromite-0.0.2-r2735:  * the complete build log and the output of `emerge -pqv '=chromeos-base/chromite-0.0.2-r2735::chromiumos'`.
chromite-0.0.2-r2735:  * The complete build log is located at '/build/auron/tmp/portage/logs/chromeos-base:chromite-0.0.2-r2735:20170105-141709.log'.
chromite-0.0.2-r2735:  * For convenience, a symlink to the build log is located at '/build/auron/tmp/portage/chromeos-base/chromite-0.0.2-r2735/temp/build.log'.
chromite-0.0.2-r2735:  * The ebuild environment file is located at '/build/auron/tmp/portage/chromeos-base/chromite-0.0.2-r2735/temp/environment'.
chromite-0.0.2-r2735:  * Working directory: '/mnt/host/source/chromite/cbuildbot'
chromite-0.0.2-r2735:  * S: '/mnt/host/source/src/platform/../../chromite'
=== Complete: job chromite-0.0.2-r2735 (3m30.0s) ===
Failed chromeos-base/chromite-0.0.2-r2735 (in 3m30.0s), retrying later.
 
Bumping timeout: https://chromium-review.googlesource.com/#/c/425197/

create_venv should only be grabbing the lock for ~7 seconds, but it seems like it can increase by a lot when under load.  There doesn't seem to be anything else wrong with the lock file.  I'm bumping the timeout up and seeing if that fixes it.

Comment 3 Deleted

Comment 4 by nxia@chromium.org, Jan 6 2017

oh, sorry, the last one wasn't caused by the lock. let me delete it. 
Project Member

Comment 5 by bugdroid1@chromium.org, Jan 6 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/infra_virtualenv/+/79a4e3aa370a94d65a6115581a4d1c42b4c28374

commit 79a4e3aa370a94d65a6115581a4d1c42b4c28374
Author: Allen Li <ayatane@google.com>
Date: Thu Jan 05 22:41:09 2017

Increase timeout for grabbing lock

Allow more leeway if machine is under heavy load

BUG= chromium:678677 
TEST=None

Change-Id: Id2bd44bea5e3e103e4fbfebd107d0b439330fbdc

[modify] https://crrev.com/79a4e3aa370a94d65a6115581a4d1c42b4c28374/create_venv

It looks like pulling chromite into the full virtualenv takes a lot of time.  I'll investigate, but that shouldn't be causing this bug? (the only thing using the full virtualenv is sysmon.  Since the builders are not managed by us or our Puppet manifests, sysmon shouldn't be enabled.  Rather, the Chrome team's sysmon should be in place).
Status: Fixed (was: Untriaged)
I think this is fixed now.

Comment 8 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 9 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61

Comment 11 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)

Sign in to add a comment