When bootstrap.py tries to move from a newer to an older branch, it can fail with sync errors:

When running this test:

BOOTSTRAP="${SANDBOX_DIR}/chromite/scripts/bootstrap --debug --buildbot --notests --buildroot ~/sand/buildroot"

${BOOTSTRAP} mario-incremental &&
  ${BOOTSTRAP} -b release-R57-9202.B mario-incremental &&
  ${BOOTSTRAP} -b release-R35-5712.B mario-incremental


It fails on the R35 checkout with:

error: Cannot fetch chromeos/toolchain-utils (GitError: --force-sync not enabled; cannot overwrite a local work tree. If you're comfortable with the possibility of losing the work tree's git metadata, use `repo sync --force-sync src/third_party/toolchain-utils` to proceed.)
Exception in thread Thread-100:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/google/home/dgarrett/sand/buildroot/.repo/repo/subcmds/sync.py", line 276, in _FetchProjectList
    success = self._FetchHelper(opt, project, *args, **kwargs)
  File "/usr/local/google/home/dgarrett/sand/buildroot/.repo/repo/subcmds/sync.py", line 321, in _FetchHelper
    cache_dir=opt.cache_dir)
  File "/usr/local/google/home/dgarrett/sand/buildroot/.repo/repo/project.py", line 1293, in Sync_NetworkHalf
    self._InitGitDir(force_sync=force_sync)
  File "/usr/local/google/home/dgarrett/sand/buildroot/.repo/repo/project.py", line 2346, in _InitGitDir
    raise e
GitError: --force-sync not enabled; cannot overwrite a local work tree. If you're comfortable with the possibility of losing the work tree's git metadata, use `repo sync --force-sync src/third_party/toolchain-utils` to proceed.

This is not unexpected, and it caused by mixing newer versions of repo with older versions of cbuildbot.

The solution may be to wipe the checkout before building any more older than the current stable branch. Since all such builds are rare (and usually single boards one-offs for firmware or factory), this is a reasonable solution.

After we move the chromeos checkout into bootstrap.py, the wipe would no longer be required.

 Issue 685766  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/86881cb2ed2e1e9d89930e511efe6fc7faccdf48

commit 86881cb2ed2e1e9d89930e511efe6fc7faccdf48
Author: Don Garrett <dgarrett@google.com>
Date: Sat Feb 25 12:19:51 2017

bootstrap: Rework bootstrap to do full ChromeOS checkouts.

In order to safely move between new and old branches, the bootstrap
script needs to do checkouts on the branch before invoking the
branched version of cbuildbot.

This will have the advantage of allowing old branches to use
git-cache, after bootstrap supports git-cache.

This is a nearly full rewrite of the bootstrap script.

BUG= chromium:684907 
TEST=Unittests.

Change-Id: I887116ef8b921c19c5f83d728d8d9fc96aa3d4ec
Reviewed-on: https://chromium-review.googlesource.com/443138
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/86881cb2ed2e1e9d89930e511efe6fc7faccdf48/scripts/cbuildbot.py
[modify] https://crrev.com/86881cb2ed2e1e9d89930e511efe6fc7faccdf48/scripts/bootstrap.py
[modify] https://crrev.com/86881cb2ed2e1e9d89930e511efe6fc7faccdf48/scripts/bootstrap_unittest.py

Over time, I've revised my test script to this:

While I'm not able to build it successfully (with or without bootstrap), I can also transition to/from an R35 build and get past sync stages, and everything else until it hits compile errors that I think are just based on branch age.


#!/bin/bash                                                                                                                   

export TMP="/usr/local/google/home/dgarrett/tmp"

BOOTSTRAP="${SANDBOX_DIR}/chromite/scripts/bootstrap                                                                          
             --debug --remote-trybot --notests                                                                                
             --buildroot /usr/local/google/home/dgarrett/sand/buildroot                                                       
             --git-cache-dir /usr/local/google/home/dgarrett/sand/git-cache"

function build {
  echo "&&&&& Starting $@"

  ${BOOTSTRAP} "$@"
  result=$?

  echo "&&&&& Finished $@"

  return $result
}

(echo &&
  build -b firmware-oak-8438.B oak-firmware &&
  build lumpy-paladin &&
  build -b release-R55-8872.B lumpy-paladin &&
  build -b release-R57-9202.B lumpy-paladin &&
  build lumpy-paladin) |& tee ~/tmp/test.log

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/7ade05adf87ed4faa01cfad70e8074796950372d

commit 7ade05adf87ed4faa01cfad70e8074796950372d
Author: Don Garrett <dgarrett@google.com>
Date: Thu Mar 02 02:42:13 2017

bootstrap: Wipe selected buildroot contents as needed.

When we reuse a buildroot between branches, we can't always update the
chroot correctly, so wipe it if necessary. Note that this failure can
happen when transitioning from new to old as well as old to new.

BUG= chromium:684907 
TEST=Unittests and manual testing.

Change-Id: Icc0af6352a323041c06e95cdc23a8bd16455d91c
Reviewed-on: https://chromium-review.googlesource.com/444082
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/7ade05adf87ed4faa01cfad70e8074796950372d/scripts/bootstrap.py
[modify] https://crrev.com/7ade05adf87ed4faa01cfad70e8074796950372d/scripts/bootstrap_unittest.py
[modify] https://crrev.com/7ade05adf87ed4faa01cfad70e8074796950372d/lib/osutils_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/399636001f2f52a725230b48f8d6ae4eab99a012

commit 399636001f2f52a725230b48f8d6ae4eab99a012
Author: Don Garrett <dgarrett@google.com>
Date: Sat Mar 04 01:11:34 2017

bootstrap: Clean Chrome checkout on branch checkouts.

gclient can't handle moving Chrome checkouts between branches
reliably, so wipe the Chrome checkout on changes. This is what some
versions of cbuildbot already do, but not older branches.

Chrome is checked out into a subdirectory of .cache, but we don't want
to wipe all of .cache if we can help it, so just wipe
.cache/distfiles.

BUG= chromium:684907 
TEST=Same as the rest of the CL stack.

Change-Id: Idf30214e1da28ec8658d0c5784d495c56b34da83
Reviewed-on: https://chromium-review.googlesource.com/447219
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Ningning Xia <nxia@chromium.org>

[modify] https://crrev.com/399636001f2f52a725230b48f8d6ae4eab99a012/scripts/bootstrap.py
[modify] https://crrev.com/399636001f2f52a725230b48f8d6ae4eab99a012/scripts/bootstrap_unittest.py