New issue
Advanced search Search tips

Issue 756001 link

Starred by 0 users

Issue metadata

Status: Fixed
Owner:
Closed: Sep 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

chromite: venv usage w/unittest runner is flaky

Project Member Reported by vapier@chromium.org, Aug 16 2017

Issue description

i've seen on clean installs (no venv yet) that cbuildbot_run flakes heavily until venv has been fully initialized

the bots are showing this too sometimes:
https://uberchromegw.corp.google.com/i/chromeos/builders/poppy-release/builds/657/steps/UnitTest/logs/stdio

chromite-0.0.2-r3272: 20:23:57: INFO: Finished [198/198] lib/workqueue/tasks_unittest (0:01:19.262856)
chromite-0.0.2-r3272: 
chromite-0.0.2-r3272: 20:23:57: ERROR: ### LOG: scripts/extract_gerrit_history_unittest
chromite-0.0.2-r3272: Traceback (most recent call last):
chromite-0.0.2-r3272:   File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
chromite-0.0.2-r3272:     "__main__", fname, loader, pkg_name)
chromite-0.0.2-r3272:   File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
chromite-0.0.2-r3272:     exec code in run_globals
chromite-0.0.2-r3272:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/scripts/create_venv.py", line 46, in <module>
chromite-0.0.2-r3272:     main()
chromite-0.0.2-r3272:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/scripts/create_venv.py", line 35, in main
chromite-0.0.2-r3272:     print(venv.ensure())
chromite-0.0.2-r3272:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/venvlib.py", line 173, in ensure
chromite-0.0.2-r3272:     with flock.FileLock(self._paths.lockfile):
chromite-0.0.2-r3272:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/flock.py", line 69, in __enter__
chromite-0.0.2-r3272:     self._acquire_lock()
chromite-0.0.2-r3272:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/flock.py", line 78, in _acquire_lock
chromite-0.0.2-r3272:     self._clean_stale_lock()
chromite-0.0.2-r3272:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/flock.py", line 107, in _clean_stale_lock
chromite-0.0.2-r3272:     self._clean_orphaned_lock()
chromite-0.0.2-r3272:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/flock.py", line 117, in _clean_orphaned_lock
chromite-0.0.2-r3272:     if self._is_lock_orphaned():
chromite-0.0.2-r3272:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/flock.py", line 129, in _is_lock_orphaned
chromite-0.0.2-r3272:     and not _is_pid_running(self._get_lock_pid()))
chromite-0.0.2-r3272:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/flock.py", line 133, in _get_lock_pid
chromite-0.0.2-r3272:     return _procdir.getpid(os.readlink(self._lockfile))
chromite-0.0.2-r3272: OSError: [Errno 2] No such file or directory: '/home/chrome-bot/.cache/cros_venv/venv-2.7.10-e5e7c29bac8f6c49d86dffd36c05d6fa/change.lock'
chromite-0.0.2-r3272: Traceback (most recent call last):
chromite-0.0.2-r3272:   File "scripts/extract_gerrit_history_unittest", line 93, in <module>
chromite-0.0.2-r3272:     main()
chromite-0.0.2-r3272:   File "scripts/extract_gerrit_history_unittest", line 31, in main
chromite-0.0.2-r3272:     venvdir = _CreateVenv()
chromite-0.0.2-r3272:   File "scripts/extract_gerrit_history_unittest", line 39, in _CreateVenv
chromite-0.0.2-r3272:     _REQUIREMENTS,
chromite-0.0.2-r3272:   File "/usr/lib64/python2.7/subprocess.py", line 573, in check_output
chromite-0.0.2-r3272:     raise CalledProcessError(retcode, cmd, output=output)
chromite-0.0.2-r3272: subprocess.CalledProcessError: Command '['/mnt/host/source/chromite/../infra_virtualenv/bin/create_venv', '/mnt/host/source/chromite/venv/requirements.txt']' returned non-zero exit status 1

 
Owner: pprabhu@chromium.org
Status: Started (was: Available)
I _think_ I know what's going on here.
Owner: ayatane@chromium.org
Stealing
Project Member

Comment 5 by bugdroid1@chromium.org, Sep 12 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/infra_virtualenv/+/24c6ba5768fda87cd690c16885f252431570b247

commit 24c6ba5768fda87cd690c16885f252431570b247
Author: Allen Li <ayatane@google.com>
Date: Tue Sep 12 08:12:04 2017

Fix locking

Remove race conditions that cause referenced bug.

The test script does not reproduce said bug, but it provides some
psychological comfort at least that the locking code works no worse
than previously.

See the docstring in the flock.py for more rationale on design
choices.

BUG= chromium:756001 
TEST=test script
TEST=unit tests
TEST=Run turtle script

Change-Id: I74070ca9f4fa8118233a8ebd91880521cb47d2a8
Reviewed-on: https://chromium-review.googlesource.com/622246
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>

[modify] https://crrev.com/24c6ba5768fda87cd690c16885f252431570b247/venv/requirements.txt
[add] https://crrev.com/24c6ba5768fda87cd690c16885f252431570b247/bin/test_locking
[add] https://crrev.com/24c6ba5768fda87cd690c16885f252431570b247/venv/.coveragerc
[modify] https://crrev.com/24c6ba5768fda87cd690c16885f252431570b247/venv/cros_venv/test_flock.py
[modify] https://crrev.com/24c6ba5768fda87cd690c16885f252431570b247/bin/run_tests
[add] https://crrev.com/24c6ba5768fda87cd690c16885f252431570b247/venv/pytest.ini
[modify] https://crrev.com/24c6ba5768fda87cd690c16885f252431570b247/venv/cros_venv/flock.py
[modify] https://crrev.com/24c6ba5768fda87cd690c16885f252431570b247/venv/cros_venv/test_venvlib.py
[modify] https://crrev.com/24c6ba5768fda87cd690c16885f252431570b247/venv/.gitignore

Status: Fixed (was: Started)

Comment 7 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Comment 8 by dchan@chromium.org, Jan 23 2018

Status: Fixed (was: Archived)

Sign in to add a comment