New issue
Advanced search Search tips

Issue 877220 link

Starred by 4 users

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

cros-workon: divergent codepaths leads to bot failures not seen locally

Project Member Reported by vapier@chromium.org, Aug 23

Issue description

when bots run through the git clone codepath, the logic is different than when using `cros_workon start` locally with CROS_WORKON_OUTOFTREE_BUILD=1.

when cloning git repos, the code looks like:
cros-workon_src_unpack() {
  ...
    if [[ "${fetch_method}" == "git" ]] ; then
      ...
        if all_local; then
            for (( i = 0; i < project_count; ++i )); do
              ...do the clones...
            done
            if [[ ${fetched} -eq ${project_count} ]]; then
                ...
                cros-workon_enforce_subtrees
                return
            else
  ...

this leads to an interesting edge case: if one repo has a subdir that clashes with a later repo, this code path will fail.  for example, libbrillo-9999.ebuild has:
CROS_WORKON_LOCALNAME=("platform2" "aosp/external/libbrillo")
CROS_WORKON_PROJECT=("chromiumos/platform2" "aosp/platform/external/libbrillo")
CROS_WORKON_DESTDIR=("${S}/platform2" "${S}/platform2/libbrillo")
CROS_WORKON_SUBTREE=("common-mk" "")

this works because platform2 doesn't have a libbrillo/ subdir.  but when we merged it back into platform2 in  issue 830940 , bots started to fail:
  >>> Unpacking source...
   * CROS_WORKON_SUBTREE: common-mk
   * CROS_WORKON_REPO: https://chromium.googlesource.com https://chromium.googlesource.com
   * CROS_WORKON_PROJECT: chromiumos/platform2 aosp/platform/external/libbrillo
   * CROS_WORKON_LOCALNAME: platform2 aosp/external/libbrillo
   * CROS_WORKON_DESTDIR: /var/tmp/portage/chromeos-base/libbrillo-0.0.1-r1358/work/libbrillo-0.0.1/platform2 /var/tmp/portage/chromeos-base/libbrillo-0.0.1-r1358/work/libbrillo-0.0.1/platform2/libbrillo
   * CROS_WORKON_COMMIT: 98eb44e578e3a267aa6fdc5bb006459d7869eb2a 84a40fb6c08e7b502e37576325bb19cd8f8af230
   * CROS_WORKON_SRCPATH:
  Cloning into '/var/tmp/portage/chromeos-base/libbrillo-0.0.1-r1358/work/libbrillo-0.0.1/platform2'...
  done.
  fatal: destination path '/var/tmp/portage/chromeos-base/libbrillo-0.0.1-r1358/work/libbrillo-0.0.1/platform2/libbrillo' already exists and is not an empty directory.

the annoying thing is that, if you're using cros-workon locally to test 9999 ebuild changes, you won't see this failure because we don't run through the git clone codepath cited above.

cros-workon_enforce_subtrees could have saved us except for two issues:
(1) it runs after all the git repos are cloned, not on a per-repo basis as we're cloning
(2) it doesn't delete the untracked paths, it uses sandbox to add deny paths (for better debugging)

i'm not sure trying to make cros-workon_enforce_subtrees smarter by itself is sufficient.  we still have divergent code paths between when we run against source directly out of the manifest checkout and when we do the git clone.

one idea to rectify these things would be to have the git clone phase, instead of cloning directly into CROS_WORKON_DESTDIR paths, sets up a dedicated source tree akin to what the manifest has.  then we converge by bringing all the repos back together before building.
 
Cc: bleung@google.com akhouderchah@chromium.org
 Issue 877213  has been merged into this issue.
This is also breaking builds on setup_board - any short term workaround?
Sorry used the wrong account in #2
you're probably describing  issue 877213 .  sync up in a bit and it'll be resolved already.
Problem got resolved, thanks @vapier!

Sign in to add a comment