Issue metadata
Sign in to add a comment
|
python presubmit hook fails outside chroot: could not load venv/lucifer/conftest.py |
||||||||||||||||||||||||
Issue description
Outside cros_sdk, the error is "failed to setup a virtualenv".
Inside, it's (I think) "no module called chromite"
(What is autotest_monkeypatcher?)
OUTSIDE CHROOT:
Errors:
* Hook script "./bin/test_lucifer" failed with code 1:
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/google3/home/semenzato/chromiumos/infra_virtualenv/venv/cros_venv/scripts/create_venv.py", line 46, in <module>
main()
File "/usr/local/google3/home/semenzato/chromiumos/infra_virtualenv/venv/cros_venv/scripts/create_venv.py", line 35, in main
print(venv.ensure())
File "/usr/local/google3/home/semenzato/chromiumos/infra_virtualenv/venv/cros_venv/venvlib.py", line 100, in ensure
self._check_or_create()
File "/usr/local/google3/home/semenzato/chromiumos/infra_virtualenv/venv/cros_venv/venvlib.py", line 108, in _check_or_create
self._create()
File "/usr/local/google3/home/semenzato/chromiumos/infra_virtualenv/venv/cros_venv/venvlib.py", line 117, in _create
logfile=logfile)
File "/usr/local/google3/home/semenzato/chromiumos/infra_virtualenv/venv/cros_venv/venvlib.py", line 209, in _create_venv
_log_check_call(command, logfile=logfile)
File "/usr/local/google3/home/semenzato/chromiumos/infra_virtualenv/venv/cros_venv/venvlib.py", line 254, in wrapped_command
call_func(args, stdout=logfile, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
cros_venv.venvlib.VirtualenvMissingError: virtualenv is not installed (caused by [Errno 2] No such file or directory)
ERROR: Failed to set up a virtualenv.
INSIDE CHROOT:
Errors:
* Hook script "./bin/test_lucifer" failed with code 1:
============================= test session starts ==============================
platform linux2 -- Python 2.7.10, pytest-3.1.3, py-1.4.34, pluggy-0.4.0
rootdir: /mnt/host/source/src/third_party/autotest/files/venv, inifile: pytest.ini
plugins: cov-2.5.1, catchlog-1.2.2
collected 26 items
lucifer/autotest_unittest.py F........
lucifer/eventlib_unittest.py .....
lucifer/leasing_unittest.py ..........
lucifer/loglib_unittest.py ..
---------- coverage: platform linux2, python 2.7.10-final-0 ----------
Name Stmts Miss Branch BrPart Cover Missing
--------------------------------------------------------------------------
lucifer/autotest.py 61 12 4 0 82% 63, 79-95, 124-129
lucifer/autotest_unittest.py 32 1 0 0 97% 26
lucifer/cmd/job_aborter.py 77 51 26 0 25% 53-64, 68-76, 91-98, 106-108, 116-118, 125, 134-136, 151-156, 165-166, 179-180, 197-200, 208-219
lucifer/cmd/job_reporter.py 63 39 2 0 37% 48-64, 72-82, 87-92, 101-103, 119-123, 128-131, 139-140, 144, 148
lucifer/handlers.py 148 118 44 0 16% 40-44, 47-54, 58, 62, 65-76, 79-82, 85-95, 98-107, 110-116, 124-131, 140-143, 149-159, 163-176, 180, 187-190, 195-207, 212-217, 226-230, 241-245, 256-258
--------------------------------------------------------------------------
TOTAL 721 221 94 0 64%
13 files skipped due to complete coverage.
=========================== slowest 5 test durations ===========================
0.16s call lucifer/leasing_unittest.py::test_obtain_lease_with_error_removes_files
0.15s call lucifer/leasing_unittest.py::test_Lease__expired
0.15s call lucifer/leasing_unittest.py::test_Job_abort
0.14s call lucifer/leasing_unittest.py::test_obtain_lease_succesfully_removes_file
0.14s call lucifer/leasing_unittest.py::test_Job_abort_with_closed_socket
=================================== FAILURES ===================================
_______________________________ test_monkeypatch _______________________________
@pytest.mark.slow
def test_monkeypatch():
"""Test monkeypatch()."""
common_file = subprocess32.check_output(
[sys.executable, '-m',
> 'lucifer.cmd.test.autotest_monkeypatcher'])
lucifer/autotest_unittest.py:25:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
popenargs = (['/home/semenzato/.cache/cros_venv/venv-2.7.10-5e245727ed23dd962b0bb6d1c7a40f7d/bin/python', '-m', 'lucifer.cmd.test.autotest_monkeypatcher'],)
kwargs = {}, timeout = None
process = <subprocess32.Popen object at 0x7fbaf62db0d0>, output = ''
unused_err = None, retcode = 1
def check_output(*popenargs, **kwargs):
r"""Run command with arguments and return its output as a byte string.
If the exit code was non-zero it raises a CalledProcessError. The
CalledProcessError object will have the return code in the returncode
attribute and output in the output attribute.
The arguments are the same as for the Popen constructor. Example:
>>> check_output(["ls", "-l", "/dev/null"])
'crw-rw-rw- 1 root root 1, 3 Oct 18 2007 /dev/null\n'
The stdout argument is not allowed as it is used internally.
To capture standard error in the result, use stderr=STDOUT.
>>> check_output(["/bin/sh", "-c",
... "ls -l non_existent_file ; exit 0"],
... stderr=STDOUT)
'ls: non_existent_file: No such file or directory\n'
"""
timeout = kwargs.pop('timeout', None)
if 'stdout' in kwargs:
raise ValueError('stdout argument not allowed, it will be overridden.')
process = Popen(stdout=PIPE, *popenargs, **kwargs)
try:
output, unused_err = process.communicate(timeout=timeout)
except TimeoutExpired:
process.kill()
output, unused_err = process.communicate()
raise TimeoutExpired(process.args, timeout, output=output)
retcode = process.poll()
if retcode:
> raise CalledProcessError(retcode, process.args, output=output)
E CalledProcessError: Command '['/home/semenzato/.cache/cros_venv/venv-2.7.10-5e245727ed23dd962b0bb6d1c7a40f7d/bin/python', '-m', 'lucifer.cmd.test.autotest_monkeypatcher']' returned non-zero exit status 1
/home/semenzato/.cache/cros_venv/venv-2.7.10-5e245727ed23dd962b0bb6d1c7a40f7d/lib/python2.7/site-packages/subprocess32.py:638: CalledProcessError
----------------------------- Captured stderr call -----------------------------
autotest_monkeypatcher: 2018-01-24 17:03:10,872:ERROR:autotest:_global_setup:60:Uncaught exception escaped Autotest setup
Traceback (most recent call last):
File "lucifer/autotest.py", line 56, in _global_setup
yield
File "lucifer/autotest.py", line 47, in monkeypatch
_monkeypatch_body()
File "lucifer/autotest.py", line 76, in _monkeypatch_body
importlib.import_module('chromite')
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named chromite
===================== 1 failed, 25 passed in 1.41 seconds ======================
Preupload failed due to errors in project(s). HINTS:
- To disable some source style checks, and for other hints, see <checkout_dir>/src/repohooks/README
- To upload only current project, run 'repo upload .'
,
Jan 25 2018
Thank you Mike. The advice to run third_party/autotest/files/utils/build_externals.py worked around the immediate problem, and now I can run the presubmit scripts, but only in the chroot. Outside the chroot I was still getting the initial error:
cros_venv.venvlib.VirtualenvMissingError: virtualenv is not installed (caused by [Errno 2] No such file or directory)
ERROR: Failed to set up a virtualenv.
So I ran
sudo apt-get install python-virtualenv
and now I get this error, which seems different.
Errors in PROJECT *chromiumos/third_party/autotest*!
COMMIT 9251f4b2:
Description:
<....>
>Change-Id: I96bf15faa5fde2981b5e29f0129e554e0b66b90b
>
>
Errors:
* Hook script "./bin/test_lucifer" failed with code 4:
Traceback (most recent call last):
File "/usr/local/google/home/semenzato/.cache/cros_venv/venv-2.7.6-5e245727ed23dd962b0bb6d1c7a40f7d/local/lib/python2.7/site-packages/_pytest/config.py", line 373, in _importconftest
mod = conftestpath.pyimport()
File "/usr/local/google/home/semenzato/.cache/cros_venv/venv-2.7.6-5e245727ed23dd962b0bb6d1c7a40f7d/local/lib/python2.7/site-packages/py/_path/local.py", line 680, in pyimport
raise self.ImportMismatchError(modname, modfile, self)
ImportMismatchError: ('lucifer.conftest', '/mnt/host/source/src/third_party/autotest/files/venv/lucifer/conftest.py', local('/usr/local/google3/home/semenzato/chromiumos/src/third_party/autotest/files/venv/lucifer/conftest.py'))
ERROR: could not load /usr/local/google3/home/semenzato/chromiumos/src/third_party/autotest/files/venv/lucifer/conftest.py
so I am reopening.
,
Jan 25 2018
By the way, if it's too hard to run the presubmit hook outside the chroot, a reasonable "fix" would be to check for it early on and recommend a course of action.
,
Jan 25 2018
,
Jan 25 2018
re #2 you have stale Python bytecode files, run find . -name "*pyc" -delete or similar. This happens if you switch between running inside and outside the chroot since the path of the file changes (read the error message). I'll add a cleanup step to the test script.
,
Jan 25 2018
Thanks! Run it where? chromite, autotest?
,
Jan 25 2018
Those didn't help. Do you mean /usr/lib/python2.7?
,
Jan 25 2018
no, never muck with paths in /usr (inside or outside of the chroot) he most likely meant autotest/, but clearing chromite/ and autotest/ wouldn't hurt since pyc would get regenerated on demand
,
Jan 25 2018
Sorry, I meant autotest. The version of chromite that autotest uses is in site-packages which is built by build_externals.py, so cleaning up pyc files recursively under autotest should be enough. Are you sure it doesn't work? Can you confirm that the pyc file corresponding to the error message gets removed, and that you get the same error message when rerunning the test and generating fresh pyc files?
,
Jan 25 2018
OK---I removed the pyc files in ~/.cache, then I removed those in autotest again, and now it works. Feel free to close if this is just the way it is, or after you add the cleanup step mentioned in #5. Thanks!
,
Jan 26 2018
The first error outside the chroot is just a matter of cleaning up pyc (CL pending) and installing virtualenv (the error message should be clear, I think). The error inside the chroot is actually also why lucifer wasn't working on moblab, have a CL for that too. Bug is https://bugs.chromium.org/p/chromium/issues/detail?id=800492 I don't think there is remaining unique work for this bug. |
|||||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||||
Comment 1 by vapier@chromium.org
, Jan 25 2018Status: Duplicate (was: Untriaged)