Actually, I think the real error is: 

[Errno 2] No such file or directory: '/tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/manifest-versions-internal'

Git commands are actually made to retry (up to 10 times) on the TLS error: https://cs.corp.google.com/chromeos_internal/chromite/lib/git.py?rcl=79317bbdc821170341e5f9ee8c5b290e2242be14&l=56


So the question now is... did '/tmp/cbuildbot-tmphy8wAJ/tmptXMeZI/' go away? 

So it's just a coincidence that we hit the TLS packet error just before trying to save the file?

What about these error messages?

error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received.
fatal: The remote end hung up unexpectedly

"fatal" doesn't sound like a recoverable error.

Oh, but possibly git uses curl, and curl is sending that message... aw.

It will be good when we're past the middle ages of software engineering.

have we seem more examples of this? If not, sounds like possible flake. Sounds like we already have the correct re-tries in place. 

This seemed to happen again on a recent tricky-paladin run: http://shortn/_YlXL6NsIYC.

I'm not sure why git isn't able to determine the current working directory after the curl error. We then retry, but are met with the ENOENT error. The directory should exist, especially when we're providing an absolute path.

------------------8<---------------------
06:59:06: INFO: No manifest-versions checkout exists at /b/cbuild/manifest-versions-internal
06:59:06: INFO: Cloning fresh manifest-versions checkout.
06:59:06: INFO: RunCommand: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /b/cbuild/manifest-versions-internal in /b/cbuild/manifest-versions-internal
06:59:16: WARNING: git reported transient error (cmd=clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /b/cbuild/manifest-versions-internal); retrying
Traceback (most recent call last):
  File "/b/cbuild/chromite/lib/retry_util.py", line 122, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/b/cbuild/chromite/lib/cros_build_lib.py", line 625, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 128; command: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /b/cbuild/manifest-versions-internal
Cloning into '/b/cbuild/manifest-versions-internal'...
error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
2017/05/23 06:59:13 WARNING: Transient error string identified in STDERR: "error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received.\n"
2017/05/23 06:59:13 WARNING: Retrying after 3s (rc=128): transient error string encountered
fatal: Could not get current working directory: No such file or directory
2017/05/23 06:59:16 WARNING: Command completed with rc 128 after 1 transient failure(s).
cmd=['git', 'clone', u'https://chrome-internal.googlesource.com/chromeos/manifest-versions', '/b/cbuild/manifest-versions-internal'], cwd=/b/cbuild/manifest-versions-internal
06:59:19: INFO: RunCommand: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /b/cbuild/manifest-versions-internal in /b/cbuild/manifest-versions-internal
06:59:20: ERROR: <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: None; command: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /b/cbuild/manifest-versions-internal
[Errno 2] No such file or directory: '/b/cbuild/manifest-versions-internal'
Traceback (most recent call last):
  File "/b/cbuild/chromite/lib/failures_lib.py", line 190, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/cbuild/chromite/cbuildbot/stages/sync_stages.py", line 741, in PerformStage
    next_manifest = self.ForceVersion(version)
  File "/b/cbuild/chromite/cbuildbot/stages/sync_stages.py", line 834, in ForceVersion
    manifest = super(MasterSlaveLKGMSyncStage, self).ForceVersion(version)
  File "/b/cbuild/chromite/cbuildbot/stages/sync_stages.py", line 549, in ForceVersion
    return self.manifest_manager.BootstrapFromVersion(version)
  File "/b/cbuild/chromite/cbuildbot/manifest_version.py", line 676, in BootstrapFromVersion
    self.RefreshManifestCheckout()
  File "/b/cbuild/chromite/cbuildbot/manifest_version.py", line 436, in RefreshManifestCheckout
    RefreshManifestCheckout(self.manifest_dir, self.manifest_repo)
  File "/b/cbuild/chromite/cbuildbot/manifest_version.py", line 95, in RefreshManifestCheckout
    repository.CloneGitRepo(manifest_dir, manifest_repo)
  File "/b/cbuild/chromite/cbuildbot/repository.py", line 115, in CloneGitRepo
    git.RunGit(working_dir, cmd, print_cmd=True)
  File "/b/cbuild/chromite/lib/git.py", line 817, in RunGit
    ['git'] + cmd, **kwargs)
  File "/b/cbuild/chromite/lib/retry_util.py", line 122, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/b/cbuild/chromite/lib/cros_build_lib.py", line 630, in RunCommand
    raise RunCommandError(estr, CommandResult(cmd=cmd), exception=e)
RunCommandError: return code: None; command: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /b/cbuild/manifest-versions-internal
[Errno 2] No such file or directory: '/b/cbuild/manifest-versions-internal'
------------------8<---------------------

My guess is the failure path is cleaning up the directory that is supposed to contain the git repo. I don't know where the code lives that runs the "git clone" operation - this would be something easy to check.

The code lives here: https://chromium.googlesource.com/chromiumos/chromite/+/b818156c0bd4701dd8887ea596c18fa9ace0a4d7/cbuildbot/repository.py#80 Which also calls into RunGit() which has a wrapper for retries, but I didn't see any cleaning up going on.

Is it git itself that attempts to clobber the dir?

git will not delete the current work directory - it will only create a new dir for the "clone".

I suppose if the script that is running creates the "git clone" destination, cd's into that dir, then invokes "git clone" from the original location, this could happen. Maybe. But it seems pretty unlikely.

It looks like:
06:59:06: INFO: Cloning fresh manifest-versions checkout.

is coming from RefreshManifestCheckout():
https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/manifest_version.py?q=CloneGitRepo&dr=C&l=95

92  if reinitialize:
93    logging.info('Cloning fresh manifest-versions checkout.')
94    osutils.RmDir(manifest_dir, ignore_missing=True)
95    repository.CloneGitRepo(manifest_dir, manifest_repo)

I'm having trouble finding the definition of osutils.RmDir (using cs.corp.google.com). I just wanted to verify this is a synchronous call that guarantees the RmDir completes before returning to this caller.

aaboagye found the definition and it _looks_ syncronous. So likely something else was running (timeouts?) that deleted the working directory.