gclient runhooks flakily fails with "unknown encoding: string-escape" |
||||
Issue descriptionHappened at https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/ios-device/107771 and https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/ios-uirefresh-simulator/10287. Both succeeded on the next run. Error stack is: ________ running 'vpython src/tools/perf/fetch_benchmark_deps.py -f' in '/b/s/w/ir/cache/builder' ________ running '/b/s/w/ir/cache/vpython/e55465/bin/python src/third_party/depot_tools/download_from_google_storage.py --no_resume --no_auth --num_threads=4 --bucket chromium-binary-patching/zucchini_testdata --recursive -d src/components/zucchini' in '/b/s/w/ir/cache/builder' Traceback (most recent call last): File "/b/s/w/ir/kitchen-checkout/depot_tools/metrics.py", line 262, in print_notice_and_exit yield File "/b/s/w/ir/kitchen-checkout/depot_tools/gclient.py", line 3107, in <module> sys.exit(main(sys.argv[1:])) File "/b/s/w/ir/kitchen-checkout/depot_tools/gclient.py", line 3093, in main return dispatcher.execute(OptionParser(), argv) File "/b/s/w/ir/kitchen-checkout/depot_tools/subcommand.py", line 252, in execute return command(parser, args[1:]) File "/b/s/w/ir/kitchen-checkout/depot_tools/gclient.py", line 2771, in CMDrunhooks return client.RunOnDeps('runhooks', args) File "/b/s/w/ir/kitchen-checkout/depot_tools/gclient.py", line 1739, in RunOnDeps self.RunHooksRecursively(self._options, pm) File "/b/s/w/ir/kitchen-checkout/depot_tools/gclient.py", line 1047, in RunHooksRecursively hook.run() File "/b/s/w/ir/kitchen-checkout/depot_tools/gclient.py", line 240, in run cmd, cwd=self.effective_cwd, always=self._verbose) File "/b/s/w/ir/kitchen-checkout/depot_tools/gclient_utils.py", line 314, in CheckCallAndFilterAndHeader return CheckCallAndFilter(args, **kwargs) File "/b/s/w/ir/kitchen-checkout/depot_tools/gclient_utils.py", line 525, in CheckCallAndFilter **kwargs) File "/b/s/w/ir/kitchen-checkout/depot_tools/subprocess2.py", line 246, in __init__ super(Popen, self).__init__(args, **kwargs) File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/subprocess.py", line 390, in __init__ errread, errwrite) File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/subprocess.py", line 1024, in _execute_child child_exception = pickle.loads(data) File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/pickle.py", line 1388, in loads return Unpickler(file).load() File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/pickle.py", line 864, in load dispatch[key](self) File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/pickle.py", line 977, in load_string self.append(rep.decode("string-escape")) LookupError: unknown encoding: string-escape step returned non-zero exit code: 1
,
Dec 5
Alright, for posterity: subprocess uses pickle to load exception information; pickle at some level uses string-escape. In some scenarios (which we haven't characterized), pickle (and thus subprocess) can fail to find string-escape via its normal scanning logic (https://github.com/python/cpython/blob/2.7/Python/codecs.c#L125). This seems to be because something is interfering with said logic. In other scenarios, this has apparently been due to calling os.chroot (e.g. https://mail.python.org/pipermail/python-list/2011-September/611815.html); it's not clear if chroot is responsible here or if it's something else manifesting similarly. Regardless, I'm assuming that this interference is the issue rather than the alternatives: that the string-escape codec actually flakily disappears in some scenarios (which would seem to be a whole separate can of worms) or that the codec saved in the cache is being reset after initial load (as I see no path by which that could happen in codecs.c) codecs.lookup caches what it finds and uses that prior to scanning anything (https://github.com/python/cpython/blob/2.7/Python/codecs.c#L117). I'm theorizing that, if we look up the string-escape codec sufficiently early, we can force it to be cached and thus avoid this error (and possibly reveal another one).
,
Dec 5
,
Dec 5
Nothing secret here.
,
Dec 5
,
Dec 5
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools/+/9875e180e5fd1b4096ca48f9386680f0bda6f709 commit 9875e180e5fd1b4096ca48f9386680f0bda6f709 Author: John Budorick <jbudorick@chromium.org> Date: Wed Dec 05 22:57:31 2018 subprocess2: cache the string-escape codec at load time. This is a speculative fix for subprocess being unable to find string-escape later in execution; the theory being that something (unclear what) is interfering w/ codecs' lookup logic and that calling lookup earlier would allow string-escape to be cached before any such interference. See crbug.com/912292#c2 for more. Bug: 912292 Change-Id: I0abdd394253c9a4984db7c6c81d044087edc68fd Reviewed-on: https://chromium-review.googlesource.com/c/1363753 Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Commit-Queue: John Budorick <jbudorick@chromium.org> [modify] https://crrev.com/9875e180e5fd1b4096ca48f9386680f0bda6f709/subprocess2.py
,
Dec 6
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e26bca2a0e136f7b93527bf1cd751d16c54b4af4 commit e26bca2a0e136f7b93527bf1cd751d16c54b4af4 Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Date: Thu Dec 06 01:19:58 2018 Roll src/third_party/depot_tools 19238fc343b2..9875e180e5fd (1 commits) https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/19238fc343b2..9875e180e5fd git log 19238fc343b2..9875e180e5fd --date=short --no-merges --format='%ad %ae %s' 2018-12-05 jbudorick@chromium.org subprocess2: cache the string-escape codec at load time. Created with: gclient setdep -r src/third_party/depot_tools@9875e180e5fd The AutoRoll server is located here: https://autoroll.skia.org/r/depot-tools-chromium-autoroll Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. BUG= chromium:912292 TBR=agable@chromium.org Change-Id: I8b39f0ac7afb696804b7296443230825fb5b4568 Reviewed-on: https://chromium-review.googlesource.com/c/1364012 Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#614218} [modify] https://crrev.com/e26bca2a0e136f7b93527bf1cd751d16c54b4af4/DEPS
,
Today
(10 hours ago)
We haven't seen reports of this in more than a month. Tentatively marking as fixed. If this error recurs, please feel free to reopen this issue (or issue 912165 , if it occurs in tests). |
||||
►
Sign in to add a comment |
||||
Comment 1 by jbudorick@chromium.org
, Dec 5Status: Assigned (was: Untriaged)