Chromite unittest error at "Failed to acquire lock on virtualenv" |
||||||
Issue descriptionSaw this error several places in paladin, should extend the waiting time or increase the retry times? https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fauron-paladin%2F9177%2F%2B%2Frecipes%2Fsteps%2FUnitTest%2F0%2Fstdout https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fauron-paladin%2F9174%2F%2B%2Frecipes%2Fsteps%2FUnitTest%2F0%2Fstdout chromite-0.0.2-r2735: [1;31m06:20:25: ERROR: The following 2 tests failed: chromite-0.0.2-r2735: scripts/export_to_gcloud_unittest chromite-0.0.2-r2735: scripts/virtualenv_wrapper_unittest[0m chromite-0.0.2-r2735: * ERROR: chromeos-base/chromite-0.0.2-r2735::chromiumos failed (test phase): chromite-0.0.2-r2735: * (no error message) chromite-0.0.2-r2735: * chromite-0.0.2-r2735: * Call stack: chromite-0.0.2-r2735: * ebuild.sh, line 93: Called src_test chromite-0.0.2-r2735: * environment, line 5880: Called die chromite-0.0.2-r2735: * The specific snippet of code: chromite-0.0.2-r2735: * cd "${CHROMITE_DIR}/cbuildbot" && sudo -u "${PORTAGE_USERNAME}" PATH="${CROS_WORKON_SRCROOT}/../depot_tools:${PATH}" ./run_tests || die chromite-0.0.2-r2735: * chromite-0.0.2-r2735: * If you need support, post the output of `emerge --info '=chromeos-base/chromite-0.0.2-r2735::chromiumos'`, chromite-0.0.2-r2735: * the complete build log and the output of `emerge -pqv '=chromeos-base/chromite-0.0.2-r2735::chromiumos'`. chromite-0.0.2-r2735: * The complete build log is located at '/build/auron/tmp/portage/logs/chromeos-base:chromite-0.0.2-r2735:20170105-141709.log'. chromite-0.0.2-r2735: * For convenience, a symlink to the build log is located at '/build/auron/tmp/portage/chromeos-base/chromite-0.0.2-r2735/temp/build.log'. chromite-0.0.2-r2735: * The ebuild environment file is located at '/build/auron/tmp/portage/chromeos-base/chromite-0.0.2-r2735/temp/environment'. chromite-0.0.2-r2735: * Working directory: '/mnt/host/source/chromite/cbuildbot' chromite-0.0.2-r2735: * S: '/mnt/host/source/src/platform/../../chromite' chromite-0.0.2-r2735: >>> Failed to emerge chromeos-base/chromite-0.0.2-r2735 for /build/auron/, Log file: chromite-0.0.2-r2735: >>> '/build/auron/tmp/portage/logs/chromeos-base:chromite-0.0.2-r2735:20170105-141709.log' chromite-0.0.2-r2735: chromite-0.0.2-r2735: * Messages for package chromeos-base/chromite-0.0.2-r2735 merged to /build/auron/: chromite-0.0.2-r2735: chromite-0.0.2-r2735: * ERROR: chromeos-base/chromite-0.0.2-r2735::chromiumos failed (test phase): chromite-0.0.2-r2735: * (no error message) chromite-0.0.2-r2735: * chromite-0.0.2-r2735: * Call stack: chromite-0.0.2-r2735: * ebuild.sh, line 93: Called src_test chromite-0.0.2-r2735: * environment, line 5880: Called die chromite-0.0.2-r2735: * The specific snippet of code: chromite-0.0.2-r2735: * cd "${CHROMITE_DIR}/cbuildbot" && sudo -u "${PORTAGE_USERNAME}" PATH="${CROS_WORKON_SRCROOT}/../depot_tools:${PATH}" ./run_tests || die chromite-0.0.2-r2735: * chromite-0.0.2-r2735: * If you need support, post the output of `emerge --info '=chromeos-base/chromite-0.0.2-r2735::chromiumos'`, chromite-0.0.2-r2735: * the complete build log and the output of `emerge -pqv '=chromeos-base/chromite-0.0.2-r2735::chromiumos'`. chromite-0.0.2-r2735: * The complete build log is located at '/build/auron/tmp/portage/logs/chromeos-base:chromite-0.0.2-r2735:20170105-141709.log'. chromite-0.0.2-r2735: * For convenience, a symlink to the build log is located at '/build/auron/tmp/portage/chromeos-base/chromite-0.0.2-r2735/temp/build.log'. chromite-0.0.2-r2735: * The ebuild environment file is located at '/build/auron/tmp/portage/chromeos-base/chromite-0.0.2-r2735/temp/environment'. chromite-0.0.2-r2735: * Working directory: '/mnt/host/source/chromite/cbuildbot' chromite-0.0.2-r2735: * S: '/mnt/host/source/src/platform/../../chromite' === Complete: job chromite-0.0.2-r2735 (3m30.0s) === Failed chromeos-base/chromite-0.0.2-r2735 (in 3m30.0s), retrying later.
,
Jan 5 2017
Bumping timeout: https://chromium-review.googlesource.com/#/c/425197/ create_venv should only be grabbing the lock for ~7 seconds, but it seems like it can increase by a lot when under load. There doesn't seem to be anything else wrong with the lock file. I'm bumping the timeout up and seeing if that fixes it.
,
Jan 6 2017
oh, sorry, the last one wasn't caused by the lock. let me delete it.
,
Jan 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra_virtualenv/+/79a4e3aa370a94d65a6115581a4d1c42b4c28374 commit 79a4e3aa370a94d65a6115581a4d1c42b4c28374 Author: Allen Li <ayatane@google.com> Date: Thu Jan 05 22:41:09 2017 Increase timeout for grabbing lock Allow more leeway if machine is under heavy load BUG= chromium:678677 TEST=None Change-Id: Id2bd44bea5e3e103e4fbfebd107d0b439330fbdc [modify] https://crrev.com/79a4e3aa370a94d65a6115581a4d1c42b4c28374/create_venv
,
Jan 6 2017
It looks like pulling chromite into the full virtualenv takes a lot of time. I'll investigate, but that shouldn't be causing this bug? (the only thing using the full virtualenv is sysmon. Since the builders are not managed by us or our Puppet manifests, sysmon shouldn't be enabled. Rather, the Chrome team's sysmon should be in place).
,
Feb 13 2017
I think this is fixed now.
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Oct 14 2017
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by nxia@chromium.org
, Jan 5 2017