I introduced this last week with CL:503483.

is this the real problem? previously the cleanup stage also run cleanup on the build_root which might not be a valid repo checkout.

It is, I can reproduce locally, and verify that the fix works.

I think the old cbuildbot code checked more before calling cleanup.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/77315c6962d75dc46034b9eb26e5be8f6699e560

commit 77315c6962d75dc46034b9eb26e5be8f6699e560
Author: Don Garrett <dgarrett@google.com>
Date: Mon May 15 23:55:49 2017

repository: BuildRootGitCleanup exits quietly if not a repo.

We often create a repository object for directories that are not (yet)
repo root directories. This cleanup function currently crashes if that
directory isn't a repo checkout, so harden it to just quietly exit if
there is nothing to cleanup.

BUG= chromium:722619 
TEST=Unittests + local build with empty buildroot.

Change-Id: I28abb214c10668dd9b5e880e6f5b15f4e2987393
Reviewed-on: https://chromium-review.googlesource.com/506593
Tested-by: Don Garrett <dgarrett@chromium.org>
Trybot-Ready: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/77315c6962d75dc46034b9eb26e5be8f6699e560/cbuildbot/repository.py

Found the reason:
http://shortn/_7wFVtxsqtE

It only triggers clean up when manifest is not None

manifest = git.ManifestCheckout.Cached(self._build_root, search=False)

following comment #5:

I think the CL:506593 is not the right fix.
you need check the manifest first, if manifest is not None, then you call _BuildRootGitCleanup.

master-paladin is experiencing the no manifest error:

https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/14635/steps/InitialCheckout/logs/stdio


cbuildbot_launch: Unhandled exception:
Traceback (most recent call last):
  File "/b/build/slave/master-paladin-master/build/chromite/scripts/cbuildbot_launch", line 168, in <module>
    DoMain()
  File "/b/build/slave/master-paladin-master/build/chromite/scripts/cbuildbot_launch", line 164, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/build/slave/master-paladin-master/build/chromite/lib/commandline.py", line 837, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/build/slave/master-paladin-master/build/chromite/scripts/cbuildbot_launch.py", line 213, in main
    InitialCheckout(branchname, buildroot, git_cache_dir)
  File "/b/build/slave/master-paladin-master/build/chromite/scripts/cbuildbot_launch.py", line 44, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/build/slave/master-paladin-master/build/chromite/scripts/cbuildbot_launch.py", line 160, in InitialCheckout
    repo.BuildRootGitCleanup(prune_all=True)
  File "/b/build/slave/master-paladin-master/build/chromite/cbuildbot/repository.py", line 416, in BuildRootGitCleanup
    git.ManifestCheckout.Cached(self.directory).ListCheckouts()]
  File "/b/build/slave/master-paladin-master/build/chromite/lib/git.py", line 766, in Cached
    md5 = cls._GetManifestHash(manifest_path)
  File "/b/build/slave/master-paladin-master/build/chromite/lib/git.py", line 516, in _GetManifestHash
    with open(source, 'rb') as f:
IOError: [Errno 2] No such file or directory: '/b/cbuild/repository/.repo/manifest.xml'

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/8d3147938a8ff289c0c4aeab767bd14e39e1b854

commit 8d3147938a8ff289c0c4aeab767bd14e39e1b854
Author: Don Garrett <dgarrett@google.com>
Date: Fri May 19 17:05:13 2017

cbuildbot_launch: If repo cleanup fails, wipe buildroot.

The is the same logic used in the Sync stages, except for being a bit
more straight forward, and triggering on any exception. This does NOT
preserve cbuildbot_logs, trybot markers or other files that aren't
relevent on the builders.

BUG= chromium:722619 
TEST=Unittests + local builds.

Change-Id: I0cdee84aac3317c4faa9996b2f3c594983189587
Reviewed-on: https://chromium-review.googlesource.com/508853
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Ningning Xia <nxia@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/8d3147938a8ff289c0c4aeab767bd14e39e1b854/cbuildbot/repository.py
[modify] https://crrev.com/8d3147938a8ff289c0c4aeab767bd14e39e1b854/scripts/cbuildbot_launch.py
[modify] https://crrev.com/8d3147938a8ff289c0c4aeab767bd14e39e1b854/scripts/cbuildbot_launch_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/8d3147938a8ff289c0c4aeab767bd14e39e1b854

commit 8d3147938a8ff289c0c4aeab767bd14e39e1b854
Author: Don Garrett <dgarrett@google.com>
Date: Fri May 19 17:05:13 2017

cbuildbot_launch: If repo cleanup fails, wipe buildroot.

The is the same logic used in the Sync stages, except for being a bit
more straight forward, and triggering on any exception. This does NOT
preserve cbuildbot_logs, trybot markers or other files that aren't
relevent on the builders.

BUG= chromium:722619 
TEST=Unittests + local builds.

Change-Id: I0cdee84aac3317c4faa9996b2f3c594983189587
Reviewed-on: https://chromium-review.googlesource.com/508853
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Ningning Xia <nxia@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/8d3147938a8ff289c0c4aeab767bd14e39e1b854/cbuildbot/repository.py
[modify] https://crrev.com/8d3147938a8ff289c0c4aeab767bd14e39e1b854/scripts/cbuildbot_launch.py
[modify] https://crrev.com/8d3147938a8ff289c0c4aeab767bd14e39e1b854/scripts/cbuildbot_launch_unittest.py

Fixed with a much more general fix.