flaky python time.strptime on various Android bots |
||
Issue description
Saw this when sheriffing today.
Parts of error message:
File "/b/s/w/ir/build/android/tombstones.py", line 45, in _ListTombstones
entries = device.StatDirectory('/data/tombstones', as_root=True)
File "/b/s/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1807, in StatDirectory
time.strptime(d['st_mtime'], _LS_DATE_FORMAT))
AttributeError: _strptime_time
Examples:
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.android%2FLollipop_Phone_Tester%2F12059%2F%2B%2Frecipes%2Fsteps%2Fsandbox_linux_unittests_on_Android%2F0%2Fstdout
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.android%2FLollipop_Phone_Tester%2F12065%2F%2B%2Frecipes%2Fsteps%2Fsandbox_linux_unittests_on_Android%2F0%2Fstdout
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.android%2FLollipop_Phone_Tester%2F12078%2F%2B%2Frecipes%2Fsteps%2Fsandbox_linux_unittests_on_Android%2F0%2Fstdout
Cause is well known - race condition in time.strptime: http://bugs.python.org/issue7980
Grepping the codebase(s) it appears that we have seen it already multiple times. For example, in Issue 584730 .
,
May 22 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/bfb763ee9499a5f88993910c2401ac5550e9ac38 commit bfb763ee9499a5f88993910c2401ac5550e9ac38 Author: pasko <pasko@chromium.org> Date: Mon May 22 10:45:57 2017 Workaround python time.strptime flake in Android tests In general time.strptime is not threadsafe in python: [1]. This causes flaky errors on various bots that test Android because catapult [2] talks to devices from different threads (for multiple reasons). One observation in the python bug ([1]) says that only importing parts of strptime is racy, so importing it early enough before any threading is started would remove this kind of flake. This workaround is used in other parts of chromium and catapult itself, so we can probably be more preventive by doing it in a central place. For now let's just reduce the flake. [1] time.strptime not thread safe https://bugs.python.org/issue7980 [2] Catapult is the home for several performance tools that span from gathering, displaying and analyzing performance data. https://github.com/catapult-project/catapult BUG= 724524 Review-Url: https://codereview.chromium.org/2891403002 Cr-Commit-Position: refs/heads/master@{#473545} [modify] https://crrev.com/bfb763ee9499a5f88993910c2401ac5550e9ac38/build/android/test_runner.py
,
May 24 2017
|
||
►
Sign in to add a comment |
||
Comment 1 by pasko@chromium.org
, May 19 2017