New issue
Advanced search Search tips

Issue 699173 link

Starred by 1 user
Status: Fixed
Owner:
aga...@chromium.org
Closed: Mar 2017
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

"git cl upload" hanging

Project Member Reported by lgrey@chromium.org, Mar 7 2017 
This is happening both with an update to an existing CL, and a brand new test CL.

Some (not all) of the time instead of hanging it crashes with this backtrace:

Traceback (most recent call last):
  File "/Users/lgrey/src/depot_tools/git_cl.py", line 5333, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/Users/lgrey/src/depot_tools/git_cl.py", line 5315, in main
    return dispatcher.execute(OptionParser(), argv)
  File "/Users/lgrey/src/depot_tools/subcommand.py", line 252, in execute
    return command(parser, args[1:])
  File "/Users/lgrey/src/depot_tools/git_cl.py", line 4301, in CMDupload
    return cl.CMDUpload(options, args, orig_args)
  File "/Users/lgrey/src/depot_tools/git_cl.py", line 1552, in CMDUpload
    ret = self.CMDUploadChange(options, git_diff_args, change)
  File "/Users/lgrey/src/depot_tools/git_cl.py", line 2196, in CMDUploadChange
    issue, patchset = upload.RealMain(upload_args)
  File "/Users/lgrey/src/depot_tools/third_party/upload.py", line 2390, in RealMain
    vcs = GuessVCS(options)
  File "/Users/lgrey/src/depot_tools/third_party/upload.py", line 2211, in GuessVCS
    (vcs, extra_output) = GuessVCSName(options)
  File "/Users/lgrey/src/depot_tools/third_party/upload.py", line 2164, in GuessVCSName
    res = RunDetectCommand(VCS_MERCURIAL, ["hg", "root"])
  File "/Users/lgrey/src/depot_tools/third_party/upload.py", line 2154, in RunDetectCommand
    out, returncode = RunShellWithReturnCode(command)
  File "/Users/lgrey/src/depot_tools/third_party/upload.py", line 880, in RunShellWithReturnCode
    universal_newlines, env)
  File "/Users/lgrey/src/depot_tools/third_party/upload.py", line 871, in RunShellWithReturnCodeAndStderr
    env=env)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1334, in _execute_child
    child_exception = pickle.loads(data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
    return Unpickler(file).load()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 966, in load_string
    raise ValueError, "insecure string pickle"
ValueError: insecure string pickle

Comment 1 by tandrii@chromium.org, Mar 13 2017

Components: Infra>SDK
Status: Unconfirmed (was: Untriaged)
Wow, that's really bizarre. Can you re-try with "git cl upload -v -v -v" and also tell us which directory and which repo you are operating in.

I suspect it fails/hangs when trying to run hg or svn. Can you run

$ hg root
$ svn info

and do they terminate quickly? Maybe they produce gigabytes of output?

Also, what's the version of your python distribution 2.7.___?


FTR, it seems that there is an error launching subprocess, and an unusual one s.t., it creates too much data (see subprocess.py), which is why pickle fails.

Comment 2 by tandrii@chromium.org, Mar 13 2017

Status: Untriaged (was: Unconfirmed)

Comment 3 by lgrey@chromium.org, Mar 13 2017 

I think you're right; I was able to work around this with CODEREVIEW_VCS=git
-v -v -v was no help at the time

Mercurial isn't installed on this machine. svn info returns very quickly with 
`svn: E155007: '/Users/lgrey/src/chromium/src' is not a working copy`

The repo is the main browser repo for lack of a better description (I work on Mac browser if that helps). Python is Apple's Python 2.7.10 unless depot tools interposes its own

Comment 4 by aga...@chromium.org, Mar 15 2017

Owner: aga...@chromium.org
Status: Started (was: Untriaged)
This is https://bugs.python.org/issue26083, which was fixed in 2.7.12:

- Issue #26083: Workaround a subprocess bug that raises an incorrect
  "ValueError: insecure string pickle" exception instead of the actual
  exception on some platforms such as Mac OS X when an exception raised
  in the forked child process prior to the exec() was large enough that
  it overflowed the internal errpipe_read pipe buffer.

Two possible workarounds from our end:
* use subprocess32
* catch the exception and treat it as "command not found"

I'm doing the latter here: https://chromium-review.googlesource.com/455537

Can you patch it into your local depot_tools and confirm that it resolves the issue for you?

Comment 5 by lgrey@chromium.org, Mar 15 2017 

Yup, everything works fine with that patched in!
Project Member

Comment 6 by bugdroid1@chromium.org, Mar 15 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/edfd7cd8dadab055dc1c3585bdaa97752c450d05

commit edfd7cd8dadab055dc1c3585bdaa97752c450d05
Author: Aaron Gable <agable@chromium.org>
Date: Wed Mar 15 17:26:59 2017

Workaround Python Issue 26083 in upload.py

This issue is fixed in Python 2.7.12, but we don't guarantee
that we use that version everywhere. The root issue is that,
on certain platforms (e.g. Mac), the exception indicating that
the underlying command (e.g. 'hg') was not found is too large,
and loading the pickle representing the exception fails.

This CL catches this specific exception and treats it identically
to the OS properly raising a 'command not found' code.

BUG= 699173 

Change-Id: I0483ab0d1d0b6a3dfcfd26f2d9644817b5b5284f
Reviewed-on: https://chromium-review.googlesource.com/455537
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>

[modify] https://crrev.com/edfd7cd8dadab055dc1c3585bdaa97752c450d05/third_party/upload.py

Comment 7 by aga...@chromium.org, Mar 15 2017

Status: Fixed (was: Started)
►

Sign in to add a comment