I am not able to repro the bug locally 

This seems to fail during the "gclient sync" phase. It's a git error when running 'git rev-list -n 1 HEAD' in one of the repo 'chrome_root/src/chromeos/assistant/libassistant/src/third_party/boringssl/src'

I can successfully run this git command in this repo manually.

Since this is a git error and happens on this particular bot, I suspect it's a corrupted git repo.  Is it possible to ssh to the bot and manually remove this repo?  gclient sync should be able to recreate during next sync.

The same error is now observed on caroline chrome pfq as well, see https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8926695166732722992

chromeos-chrome-73.0.3645.0_rc-r1: [0:03:15] _____ Conflicting directory found in /home/chrome-bot/chrome_root/src/chromeos/assistant/libassistant/src/third_party/android_tools. Moving to /home/chrome-bot/chrome_root/src/chromeos/assistant/libassistant/src/_bad_scm/third_party/android_toolsl8NCIY.
chromeos-chrome-73.0.3645.0_rc-r1: [0:03:15] 
chromeos-chrome-73.0.3645.0_rc-r1: 
chromeos-chrome-73.0.3645.0_rc-r1: ________ running 'git -c core.deltaBaseCacheLimit=2g clone --no-checkout --progress https://chromium.googlesource.com/android_tools.git /home/chrome-bot/chrome_root/src/chromeos/assistant/libassistant/src/third_party/_gclient_android_tools_xyXkOJ' in '/home/chrome-bot/chrome_root/src/chromeos/assistant/libassistant/src'
chromeos-chrome-73.0.3645.0_rc-r1: [0:03:15] Cloning into '/home/chrome-bot/chrome_root/src/chromeos/assistant/libassistant/src/third_party/_gclient_android_tools_xyXkOJ'...
chromeos-chrome-73.0.3645.0_rc-r1: [0:03:45] warning: You appear to have cloned an empty repository.
chromeos-chrome-73.0.3645.0_rc-r1: ----------------------------------------
chromeos-chrome-73.0.3645.0_rc-r1: Error: Command 'git rev-parse --verify FETCH_HEAD' returned non-zero exit status 128 in /home/chrome-bot/chrome_root/src/chromeos/assistant/libassistant/src/third_party/android_tools
chromeos-chrome-73.0.3645.0_rc-r1: fatal: Needed a single revision
chromeos-chrome-73.0.3645.0_rc-r1: 
chromeos-chrome-73.0.3645.0_rc-r1: 
chromeos-chrome-73.0.3645.0_rc-r1: 
chromeos-chrome-73.0.3645.0_rc-r1: Warnings:
chromeos-chrome-73.0.3645.0_rc-r1: Conflicting directory /home/chrome-bot/chrome_root/src/chromeos/assistant/libassistant/src/third_party/android_tools moved to /home/chrome-bot/chrome_root/src/chromeos/assistant/libassistant/src/_bad_scm/third_party/android_toolsl8NCIY.
chromeos-chrome-73.0.3645.0_rc-r1: Traceback (most recent call last):
chromeos-chrome-73.0.3645.0_rc-r1:   File "src/chromeos/assistant/internal/build_libassistant.py", line 172, in <module>
chromeos-chrome-73.0.3645.0_rc-r1:     sys.exit(main(sys.argv[1:]))
chromeos-chrome-73.0.3645.0_rc-r1:   File "src/chromeos/assistant/internal/build_libassistant.py", line 99, in main
chromeos-chrome-73.0.3645.0_rc-r1:     cwd=os.path.join(libassistant_standalone_path, 'src'))
chromeos-chrome-73.0.3645.0_rc-r1:   File "/usr/lib64/python2.7/subprocess.py", line 540, in check_call
chromeos-chrome-73.0.3645.0_rc-r1:     raise CalledProcessError(retcode, cmd)
chromeos-chrome-73.0.3645.0_rc-r1: subprocess.CalledProcessError: Command '['/mnt/host/depot_tools/gclient', 'sync', '-j32']' returned non-zero exit status 1
chromeos-chrome-73.0.3645.0_rc-r1: Error: Command '/usr/bin/python2.7 src/chromeos/assistant/internal/build_libassistant.py --sync_only' returned non-zero exit status 1 in /home/chrome-bot/chrome_root/.
chromeos-chrome-73.0.3645.0_rc-r1: Hook '/usr/bin/python2.7 src/chromeos/assistant/internal/build_libassistant.py --sync_only' took 301.43 secs
chromeos-chrome-73.0.3645.0_rc-r1:  * ERROR: chromeos-base/chromeos-chrome-73.0.3645.0_rc-r1::chromiumos failed (configure phase):
chromeos-chrome-73.0.3645.0_rc-r1:  *   (no error message)

Good news is the new run of cyan chrome pfq has passed BuildPackages step. But since this similar error just showed up on another board (caroline), I'll keep monitoring these builds to see if it's a flaky issue.

#2 is a different git error. there is a warning of empty repo
chromeos-chrome-73.0.3645.0_rc-r1: [0:03:15] Cloning into '/home/chrome-bot/chrome_root/src/chromeos/assistant/libassistant/src/third_party/_gclient_android_tools_xyXkOJ'...
chromeos-chrome-73.0.3645.0_rc-r1: [0:03:45] warning: You appear to have cloned an empty repository.

This repo isn't empty. This seems a sign of GOB problem. Maybe loop in some ChromeInfra or Chrome git admin to see if they have some clue.

Similar failure is also observed on kevin-arcnext-chrome-pfq: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8926695142565101600, not in BuildPackages step though, but is in TestSimpleChromeWorkflow step, see https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8926695142565101600/+/steps/TestSimpleChromeWorkflow/0/stdout

Selected error messages:

________ running '/b/swarming/w/ir/kitchen-workdir/python_bin/python src/chromeos/assistant/internal/build_libassistant.py --sync_only' in '/b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/.'

third_party/grpc/grpc (ERROR)
----------------------------------------
[0:00:00] Started.
----------------------------------------
Error: Command 'git rev-parse --verify FETCH_HEAD' returned non-zero exit status 128 in /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/chromeos/assistant/libassistant/src/third_party/grpc/grpc
fatal: Needed a single revision

Traceback (most recent call last):
  File "src/chromeos/assistant/internal/build_libassistant.py", line 172, in <module>
    sys.exit(main(sys.argv[1:]))
  File "src/chromeos/assistant/internal/build_libassistant.py", line 99, in main
    cwd=os.path.join(libassistant_standalone_path, 'src'))
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['gclient', 'sync', '-j32']' returned non-zero exit status 1
Error: Command '/b/swarming/w/ir/kitchen-workdir/python_bin/python src/chromeos/assistant/internal/build_libassistant.py --sync_only' returned non-zero exit status 1 in /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/.
Hook '/b/swarming/w/ir/kitchen-workdir/python_bin/python src/chromeos/assistant/internal/build_libassistant.py --sync_only' took 32.39 secs

Since all these failure are related to libassistant, it seem like some config change in libassistant that caused this. 

In all these failed builds, chrome version are the same: 73.0.3645.0. Though the latest cyan_chrome_pfq with the same chrome version has just cycled green.

Because of xiaohuic@'s suspicion in #2 and #4, loop in Chrome sheriff and Chrome OS deputy to see if they have any clue.

Now it started show up on caroline-chrome-pfq https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8926586505273140320. 

Note it's not the same as the build in comment#2, the failed build in comment#2 is caroline-arcnext-chrome-pfq.

According to previous experience on other boards, the error should go away automatically on the second run. Keep monitoring.

I am not sure I am the right person to look into this. We will need someone with repo/git expertise from infra to see why the repos are acting up.

I have no idea what would cause that HEAD error, but if I'm looking at the goldeneye info correctly (and I really have little idea what I'm looking at there), these builders are all green again, so maybe it's fixed?

BTW, if you do think there is a git problem, you should remove yourself from Owner and put this bug in the Infra>Git>Admin queue.

Hm, Infra>Git>Admins aren't here for debugging general git issues. As a git admin, I can help grant lacking read permissions, but I need specific account used to authenticate to Git servers && list of repos. Thus, removing from Infra>Git>Admin bugs queue.

As a bystander, this looks to me that like repo tool isn't able to correctly cleanup when some combination of git repos is moved around in the checkout tree. However, I'm not at all familiar with inner working of repo tool. +vapier@

this is `gclient` running, not `repo`.  we don't use `repo` to sync any Chrome browser code.  i don't know who watches over that tool and its process.