New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 716241 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
OOO until 2019-01-24
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocked on:
issue 776804



Sign in to add a comment

Local installations of some Python packages break typ and Telemetry

Project Member Reported by kbr@chromium.org, Apr 27 2017

Issue description

In https://github.com/catapult-project/catapult/issues/3525 we found that having this package installed:

python-egenix-mxdatetime

causes:

./content/test/gpu/run_unittests.py

to core dump in Python code while discovering the unit test list.

ynovikov@ found that having the package:

python-logilab-common

installed also breaks:

./content/test/gpu/run_unittests.py lint_unittest

with:

[1/1] gpu_tests.lint_unittest.LintTest.testPassingPylintCheckForGpuTestsDir failed unexpectedly 0.0003s:
  Traceback (most recent call last):
    File "/usr/local/google/work/clank/src/content/test/gpu/gpu_tests/lint_unittest.py", line 42, in testPassingPylintCheckForGpuTestsDir
      self.assertTrue(LintCheckPassed(os.path.abspath(os.path.dirname(__file__))))
    File "/usr/local/google/work/clank/src/content/test/gpu/gpu_tests/lint_unittest.py", line 30, in LintCheckPassed
      assert lint, 'pylint module cannot be found'
  AssertionError: pylint module cannot be found

It looks like when running these tests via typ, the local installations in /usr/lib/python2.7/dist-packages/ are taking precedence over the copies in src/third_party/ -- in particular, src/third_party/logilab/ . This is a significant problem.

Can typ be made to be more "hermetic"?

 
For the "python-logilab-common" case, I think we can fix it by updating the code to add path to dir in https://cs.chromium.org/chromium/src/content/test/gpu/gpu_tests/path_util.py?rcl=8ba4e2e2f7cab6455607be4902f72379c9e49060&l=18 to insert to the 2nd spot in sys.path (just like https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/__init__.py?rcl=67894d267da3cd26af31413ec51764f5baabee26&l=23)

ynovikov@, can you try that and see if that fixes it for you?

You mean I should change "sys.path.append(path)" to "sys.path.insert(1, path)"?
That didn't help.

Comment 3 by kbr@chromium.org, Apr 28 2017

Note that I wasn't able to reproduce Yuly's problem by installing the python-logilab-common package. There must be an interaction with some other package.

Sorry, I was wrong in #2, this change did help.
The cause of my mistake was python-logilab-common installing python-egenix-mxdatetime as a dependency.

Comment 5 by zmo@chromium.org, May 23 2017

Cc: zmo@chromium.org xlai@chromium.org
I also encountered this problem (mxdatetime), and it's likely xlai also ran into this when trying to run roll_webgl_conformance.py and it never finished because of presubmit check failure.

Comment 6 by zmo@chromium.org, May 23 2017

I changed "sys.path.append(path)" to "sys.path.insert(1, path)" and it didn't help my case

Comment 7 by kbr@chromium.org, Jan 16 2018

Cc: kbr@chromium.org
 Issue 802403  has been merged into this issue.

Comment 8 by kbr@chromium.org, Jan 16 2018

Owner: kbr@chromium.org
Status: Assigned (was: Available)
This showed up again for enne in  Issue 802403 . Thanks to the Infra folks' deployment of vpython, I can confirm that switching to using that interpreter categorically fixes this crash!

Comment 9 by kbr@chromium.org, Jan 17 2018

Blockedon: 776804
Project Member

Comment 10 by bugdroid1@chromium.org, Jan 17 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dd4683fdaecfebad642eaf095a962fd304e66e5e

commit dd4683fdaecfebad642eaf095a962fd304e66e5e
Author: Kenneth Russell <kbr@chromium.org>
Date: Wed Jan 17 05:43:15 2018

Always run GPU unit tests with vpython.

The vpython tool that's now in depot_tools (and which should be
on all Chromium developers' and bots' paths) is sufficiently hermetic
to guard against breakage caused by Python related packages being
installed on the system.

More widespread usage of vpython in the GPU tests will follow.

Bug:  716241 
Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: Id01974c5e1f1d0f76a8ab2e57638de52e233e9f9
Reviewed-on: https://chromium-review.googlesource.com/869114
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Cr-Commit-Position: refs/heads/master@{#529634}
[modify] https://crrev.com/dd4683fdaecfebad642eaf095a962fd304e66e5e/content/test/gpu/run_unittests.py

Project Member

Comment 11 by bugdroid1@chromium.org, Mar 9 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8b319633ee79471b5cc0d5dab9a36a537c8ba274

commit 8b319633ee79471b5cc0d5dab9a36a537c8ba274
Author: Kenneth Russell <kbr@chromium.org>
Date: Fri Mar 09 00:03:53 2018

Use vpython for GPU integration tests.

Most developers use this harness when running the tests locally, and
they should all have depot_tools' vpython on their PATH. This will
help ensure that they get the hermetic Python that the bots use.

This won't have any effect on the bots since they invoke vpython
earlier, and run more wrapper scripts before getting to this one.

TBR=iannucci@chromium.org, ynovikov@chromium.org

Bug:  716241 
Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I820ae66b0232387328c75f937c241050f8776c59
Reviewed-on: https://chromium-review.googlesource.com/956553
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#541946}
[modify] https://crrev.com/8b319633ee79471b5cc0d5dab9a36a537c8ba274/content/test/gpu/run_gpu_integration_test.py

Comment 12 by kbr@chromium.org, Mar 9 2018

Status: Fixed (was: Assigned)
This has now been fixed for the common cases of developers running the GPU tests locally.

Sign in to add a comment