Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Starred by 1 user
Status: WontFix
Owner: ----
Closed: Jan 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment
git clone: [Errno 2] No such file or directory: ...
Project Member Reported by semenzato@chromium.org, Dec 28 2016 Back to list
Happened on canary master.

https://uberchromegw.corp.google.com/i/chromeos/builders/master-release/builds/2720/steps/BranchUtilTest/logs/stdio

(snippet below)

Note that git claims to be retrying---so maybe it failed twice?  Can we tell it to try three times?

Searching crbug.com for the error message ("a TLS packet with unexpected length was received") shows that this happens with a low frequency (see for instance  issue 298189 ).  Maybe too low to worry about?

Also, the cascading error ("No such file or directory") probably should not happen.


-----------------

BUILD_STEP: ManifestVersionedSync
************************************************************
** Start Stage ManifestVersionedSync - Tue, 27 Dec 2016 18:16:02 -0800 (PST)
** 
** Stage that generates a unique manifest file, and sync's to it.
************************************************************
Preconditions for the stage successfully met. Beginning to execute stage...

STEP_TEXT: 9128.0.0
18:16:02: INFO: No manifest-versions checkout exists at /tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal
18:16:02: INFO: Cloning fresh manifest-versions checkout.
18:16:02: INFO: RunCommand: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal in /tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal
18:16:35: WARNING: git reported transient error (cmd=clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal); retrying
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 122, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/cros_build_lib.py", line 625, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 128; command: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal
Cloning into '/tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal'...
error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

cmd=['git', 'clone', 'https://chrome-internal.googlesource.com/chromeos/manifest-versions', '/tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal'], cwd=/tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal
18:16:38: INFO: RunCommand: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal in /tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal

STEP_FAILURE
18:16:38: ERROR: <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: None; command: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal
[Errno 2] No such file or directory: '/tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal'
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/failures_lib.py", line 172, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 757, in PerformStage
    next_manifest = self.ForceVersion(version)
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 567, in ForceVersion
    return self.manifest_manager.BootstrapFromVersion(version)
  File "/b/cbuild/internal_master/chromite/cbuildbot/manifest_version.py", line 696, in BootstrapFromVersion
    self.RefreshManifestCheckout()
  File "/b/cbuild/internal_master/chromite/cbuildbot/manifest_version.py", line 437, in RefreshManifestCheckout
    RefreshManifestCheckout(self.manifest_dir, self.manifest_repo)
  File "/b/cbuild/internal_master/chromite/cbuildbot/manifest_version.py", line 98, in RefreshManifestCheckout
    repository.CloneGitRepo(manifest_dir, manifest_repo)
  File "/b/cbuild/internal_master/chromite/cbuildbot/repository.py", line 107, in CloneGitRepo
    git.RunGit(working_dir, cmd, print_cmd=True)
  File "/b/cbuild/internal_master/chromite/lib/git.py", line 808, in RunGit
    ['git'] + cmd, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 122, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/cros_build_lib.py", line 630, in RunCommand
    raise RunCommandError(estr, CommandResult(cmd=cmd), exception=e)
RunCommandError: return code: None; command: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal
[Errno 2] No such file or directory: '/tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal'

************************************************************
** Finished Stage ManifestVersionedSync - Tue, 27 Dec 2016 18:16:38 -0800 (PST)
 
Actually, I think the real error is: 

[Errno 2] No such file or directory: '/tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal'

Git commands are actually made to retry (up to 10 times) on the TLS error: https://cs.corp.google.com/chromeos_internal/chromite/lib/git.py?rcl=79317bbdc821170341e5f9ee8c5b290e2242be14&l=56


So the question now is... did '/tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/' go away? 

Summary: git clone: [Errno 2] No such file or directory: ... (was: git clone: a TLS packet with unexpected length was received)
So it's just a coincidence that we hit the TLS packet error just before trying to save the file?

What about these error messages?

error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received.
fatal: The remote end hung up unexpectedly

"fatal" doesn't sound like a recoverable error.

Oh, but possibly git uses curl, and curl is sending that message... aw.

It will be good when we're past the middle ages of software engineering.

have we seem more examples of this? If not, sounds like possible flake. Sounds like we already have the correct re-tries in place. 
Comment 5 by autumn@chromium.org, Jan 11 2017
Status: WontFix
Cc: grundler@chromium.org hashimoto@chromium.org aaboagye@chromium.org
This seemed to happen again on a recent tricky-paladin run: http://shortn/_YlXL6NsIYC.

I'm not sure why git isn't able to determine the current working directory after the curl error. We then retry, but are met with the ENOENT error. The directory should exist, especially when we're providing an absolute path.

------------------8<---------------------
06:59:06: INFO: No manifest-versions checkout exists at /b/cbuild/manifest-versions-internal
06:59:06: INFO: Cloning fresh manifest-versions checkout.
06:59:06: INFO: RunCommand: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /b/cbuild/manifest-versions-internal in /b/cbuild/manifest-versions-internal
06:59:16: WARNING: git reported transient error (cmd=clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /b/cbuild/manifest-versions-internal); retrying
Traceback (most recent call last):
  File "/b/cbuild/chromite/lib/retry_util.py", line 122, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/b/cbuild/chromite/lib/cros_build_lib.py", line 625, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 128; command: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /b/cbuild/manifest-versions-internal
Cloning into '/b/cbuild/manifest-versions-internal'...
error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
2017/05/23 06:59:13 WARNING: Transient error string identified in STDERR: "error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received.\n"
2017/05/23 06:59:13 WARNING: Retrying after 3s (rc=128): transient error string encountered
fatal: Could not get current working directory: No such file or directory
2017/05/23 06:59:16 WARNING: Command completed with rc 128 after 1 transient failure(s).
cmd=['git', 'clone', u'https://chrome-internal.googlesource.com/chromeos/manifest-versions', '/b/cbuild/manifest-versions-internal'], cwd=/b/cbuild/manifest-versions-internal
06:59:19: INFO: RunCommand: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /b/cbuild/manifest-versions-internal in /b/cbuild/manifest-versions-internal
06:59:20: ERROR: <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: None; command: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /b/cbuild/manifest-versions-internal
[Errno 2] No such file or directory: '/b/cbuild/manifest-versions-internal'
Traceback (most recent call last):
  File "/b/cbuild/chromite/lib/failures_lib.py", line 190, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/cbuild/chromite/cbuildbot/stages/sync_stages.py", line 741, in PerformStage
    next_manifest = self.ForceVersion(version)
  File "/b/cbuild/chromite/cbuildbot/stages/sync_stages.py", line 834, in ForceVersion
    manifest = super(MasterSlaveLKGMSyncStage, self).ForceVersion(version)
  File "/b/cbuild/chromite/cbuildbot/stages/sync_stages.py", line 549, in ForceVersion
    return self.manifest_manager.BootstrapFromVersion(version)
  File "/b/cbuild/chromite/cbuildbot/manifest_version.py", line 676, in BootstrapFromVersion
    self.RefreshManifestCheckout()
  File "/b/cbuild/chromite/cbuildbot/manifest_version.py", line 436, in RefreshManifestCheckout
    RefreshManifestCheckout(self.manifest_dir, self.manifest_repo)
  File "/b/cbuild/chromite/cbuildbot/manifest_version.py", line 95, in RefreshManifestCheckout
    repository.CloneGitRepo(manifest_dir, manifest_repo)
  File "/b/cbuild/chromite/cbuildbot/repository.py", line 115, in CloneGitRepo
    git.RunGit(working_dir, cmd, print_cmd=True)
  File "/b/cbuild/chromite/lib/git.py", line 817, in RunGit
    ['git'] + cmd, **kwargs)
  File "/b/cbuild/chromite/lib/retry_util.py", line 122, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/b/cbuild/chromite/lib/cros_build_lib.py", line 630, in RunCommand
    raise RunCommandError(estr, CommandResult(cmd=cmd), exception=e)
RunCommandError: return code: None; command: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /b/cbuild/manifest-versions-internal
[Errno 2] No such file or directory: '/b/cbuild/manifest-versions-internal'
------------------8<---------------------
My guess is the failure path is cleaning up the directory that is supposed to contain the git repo. I don't know where the code lives that runs the "git clone" operation - this would be something easy to check.
The code lives here: https://chromium.googlesource.com/chromiumos/chromite/+/b818156c0bd4701dd8887ea596c18fa9ace0a4d7/cbuildbot/repository.py#80 Which also calls into RunGit() which has a wrapper for retries, but I didn't see any cleaning up going on.

Is it git itself that attempts to clobber the dir?
git will not delete the current work directory - it will only create a new dir for the "clone".

I suppose if the script that is running creates the "git clone" destination, cd's into that dir, then invokes "git clone" from the original location, this could happen. Maybe. But it seems pretty unlikely.
It looks like:
06:59:06: INFO: Cloning fresh manifest-versions checkout.

is coming from RefreshManifestCheckout():
https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/manifest_version.py?q=CloneGitRepo&dr=C&l=95

92  if reinitialize:
93    logging.info('Cloning fresh manifest-versions checkout.')
94    osutils.RmDir(manifest_dir, ignore_missing=True)
95    repository.CloneGitRepo(manifest_dir, manifest_repo)

I'm having trouble finding the definition of osutils.RmDir (using cs.corp.google.com). I just wanted to verify this is a synchronous call that guarantees the RmDir completes before returning to this caller.
aaboagye found the definition and it _looks_ syncronous. So likely something else was running (timeouts?) that deleted the working directory.
Sign in to add a comment