presubmit_support.py traceback on Ctrl-C way too long |
|||||||
Issue description
Hitting Ctrl-C during presubmit_support.py (when run manually or via git cl upload) results in a traceback with *thousands* of lines referencing modules in /usr/lib/python2.7/multiprocessing/. This is really annoying as it clears my scrollback and is generally unpleasant.
presubmit_support.py itself wraps main() in a try/except KeyboardInterrupt, but that doesn't work for worker processes from the Pool.
I tried fixing this myself with:
CTRL_C = signal.CTRL_C_EVENT if sys.platform == 'win32' else signal.SIGINT
signal.signal(CTRL_C, lambda x, y: RaiseKeyboardInterrupt)
in a function passed as the 2nd argument to multiprocessing.Pool, but tests/presubmit_unittest.py tests fail because they expect an exact set of members and imports to be present in presubmit_support.py.
Could someone look into this and if this is the right way to fix it, explain where to put the new function and import so the presubmit_unittest.py tests don't freak out?
,
Sep 1 2016
,
Sep 4 2017
This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue. Sorry for the inconvenience if the bug really should have been left as Available. If you change it back, also remove the "Hotlist-Recharge-Cold" label. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Sep 5 2017
That's actually a pretty good idea. I shamelessly stole your code and uploaded a change here: https://chromium-review.googlesource.com/651764 PTAL?
,
Sep 5 2017
Awesome, your CL eliminates most or all of the backtraces depending on when the Ctrl-C's happen.
,
Sep 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools/+/873c28d17541fd547e2c90de83083a28ea65a8b9 commit 873c28d17541fd547e2c90de83083a28ea65a8b9 Author: Aaron Gable <agable@chromium.org> Date: Wed Sep 06 00:15:55 2017 Capture ctrl-c in presubmit multiprocessing pool Presubmit spins up lots of multiprocessing processes to run each individual test. If you cancel your presubmit run with <ctrl>+c, that signal gets passed through to each of those, which then raises its own KeyboardInterrupt, and prints its own stacktrace. This change has each member of the multiprocessing pool instead exit gracefully (albeit with an error code) so that only the parent process prints its stacktrace. R=michaelpg@chromium.org Bug: 635196 Change-Id: If9081910a359889a43bc1b72c91a859ebe37a1d6 Reviewed-on: https://chromium-review.googlesource.com/651764 Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Commit-Queue: Aaron Gable <agable@chromium.org> [modify] https://crrev.com/873c28d17541fd547e2c90de83083a28ea65a8b9/presubmit_support.py [modify] https://crrev.com/873c28d17541fd547e2c90de83083a28ea65a8b9/tests/presubmit_unittest.py
,
Sep 6 2017
,
Sep 6 2017
I got following error when executing "git cl upload":
Could you tell me workaround of this?
Running presubmit upload checks ...
Traceback (most recent call last):
File "c:\src\depot_tools\\git_cl.py", line 6112, in <module>
sys.exit(main(sys.argv[1:]))
File "c:\src\depot_tools\\git_cl.py", line 6094, in main
return dispatcher.execute(OptionParser(), argv)
File "c:\src\depot_tools\subcommand.py", line 252, in execute
return command(parser, args[1:])
File "c:\src\depot_tools\\git_cl.py", line 4943, in CMDupload
return cl.CMDUpload(options, args, orig_args)
File "c:\src\depot_tools\\git_cl.py", line 1641, in CMDUpload
change=change)
File "c:\src\depot_tools\\git_cl.py", line 1573, in RunHook
gerrit_obj=self._codereview_impl.GetGerritObjForPresubmit())
File "c:\src\depot_tools\presubmit_support.py", line 1368, in DoPresubmitChecks
results += executer.ExecPresubmitScript(presubmit_script, filename)
File "c:\src\depot_tools\presubmit_support.py", line 1262, in ExecPresubmitScript
gerrit_obj=self.gerrit, dry_run=self.dry_run)
File "c:\src\depot_tools\presubmit_support.py", line 451, in __init__
self.cpu_count, _capture_interrupt)
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\multiprocessing\__init__.py", line 232, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild)
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\multiprocessing\pool.py", line 159, in __init__
self._repopulate_pool()
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\multiprocessing\pool.py", line 222, in _repopulate_pool
w.start()
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\multiprocessing\process.py", line 130, in start
self._popen = Popen(self)
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\multiprocessing\forking.py", line 277, in __init__
dump(process_obj, to_child, HIGHEST_PROTOCOL)
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\multiprocessing\forking.py", line 199, in dump
ForkingPickler(file, protocol).dump(obj)
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\pickle.py", line 224, in dump
self.save(obj)
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\pickle.py", line 419, in save_reduce
save(state)
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\pickle.py", line 681, in _batch_setitems
save(v)
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\pickle.py", line 562, in save_tuple
save(element)
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\pickle.py", line 748, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle <function _capture_interrupt at 0x0312DA70>: it's not found as presubmit_support._capture_interrupt
C:\src\w\cr2\src>Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\multiprocessing\forking.py", line 381, in main
self = load(from_parent)
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\pickle.py", line 1378, in load
return Unpickler(file).load()
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\pickle.py", line 858, in load
dispatch[key](self)
File "c:\src\depot_tools\win_tools-2_7_6_bin\python\bin\lib\pickle.py", line 880, in load_eof
raise EOFError
EOFError
,
Sep 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools/+/5254da18432cfefbcfff46fa7b473f8ccfcc0543 commit 5254da18432cfefbcfff46fa7b473f8ccfcc0543 Author: Aaron Gable <agable@chromium.org> Date: Wed Sep 06 21:06:00 2017 Revert "Capture ctrl-c in presubmit multiprocessing pool" This reverts commit 873c28d17541fd547e2c90de83083a28ea65a8b9. Reason for revert: Broken on Windows ( crbug.com/762389 ) Original change's description: > Capture ctrl-c in presubmit multiprocessing pool > > Presubmit spins up lots of multiprocessing processes to run > each individual test. If you cancel your presubmit run with > <ctrl>+c, that signal gets passed through to each of those, > which then raises its own KeyboardInterrupt, and prints its > own stacktrace. > > This change has each member of the multiprocessing pool instead > exit gracefully (albeit with an error code) so that only the > parent process prints its stacktrace. > > R=michaelpg@chromium.org > > Bug: 635196 > Change-Id: If9081910a359889a43bc1b72c91a859ebe37a1d6 > Reviewed-on: https://chromium-review.googlesource.com/651764 > Reviewed-by: Robbie Iannucci <iannucci@chromium.org> > Commit-Queue: Aaron Gable <agable@chromium.org> TBR=iannucci@chromium.org,agable@chromium.org,michaelpg@chromium.org Change-Id: Ib8e5b2f59b0060dfbfbeba348e211db292318b3b No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 635196 Reviewed-on: https://chromium-review.googlesource.com/653434 Reviewed-by: Aaron Gable <agable@chromium.org> Commit-Queue: Aaron Gable <agable@chromium.org> [modify] https://crrev.com/5254da18432cfefbcfff46fa7b473f8ccfcc0543/presubmit_support.py [modify] https://crrev.com/5254da18432cfefbcfff46fa7b473f8ccfcc0543/tests/presubmit_unittest.py
,
Sep 6 2017
Had to revert. Discussion in issue 762389 . Will reland with fixes when I get a chance.
,
Aug 10
Unassigning from myself because I'm on sabbatical. |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by michae...@chromium.org
, Aug 6 2016