I've restarted the builders with clobber.  If the problem is merely
some source out-of-sync-ness, that should help.

I'm not sure whether that's a sure thing, but we'll find out soon
enough.  :-(

The builders are all into BuildPackages, which should mean they're
not experiencing this bug any more.

OK.  Now the canaries have this disease.

Clobber worked before, so I'll be looking into doing that again,
but I'd kind of like a root cause, too.

Clobber appears to be a temporary "fix". I think something is broken in one of the scripts?

Has this problem re-surfaced on the PFQ?

I was assuming that this problem was caused by a change that
causes a one-time glitch that can be fixed by clobber.  If the
problem is able to recur, that's a slightly different problem...

Also, "clobber" doesn't work for the canaries; it causes bad
events.  See  bug 624460 .

Yes, it is affecting the PFQ now.

My theory is:

* Somehow we are triggering something (gclient I think) that is generating this line of output in the sdio (from the first 'steps' stage):


________ deleting 'src' in '/b/cbuild/internal_master/.cache/distfiles/target/chrome-src-internal'


That seems bad. 'src' includes everything, it doesn't seem like we should be deleting it.

Then, later when we run this:


cros_sdk '--chrome_root=/b/cbuild/internal_master/.cache/distfiles/target/chrome-src-internal' ...


We get:


ERROR   cros_sdk:enter_chroot: Not mounting chrome source: could not find CHROME_ROOT/src dir.


Which is consistent with what I see locally if I pass a path that does not contain a valid 'src' directory to cros_sdk.

OK.  My leading theory is that this problem originates with a
change in the Chrome sources.  Do we have a notion of how to go
after that theory?

I don't see how the chrome source could be causing this, especially if we are seeing this on the canaries (since we haven't upreved chrome in a while), but we can see if this reproduces on a try job, I will start one now.

Also, FWIW:
* We are only seeing this on the internal (USE internal) builders
* we are *not* seeing this symptom on the informational chrome PFQ builders, e.g. https://uberchromegw.corp.google.com/i/chromeos.chrome/waterfall

My reason for suspecting Chrome sources:
 1) The problem showed up first on the PFQ.
 2) The problem showed up on the canary sometime after
    this PFQ uprev:
        https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/3066
 3) The CQ hasn't been affected (the CQ doesn't build chrome).
 4) The symptoms mentioned are showing up in the Chrome build logs.

Actually we had a green Chrome uprev here:

https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/3066

From the emails, looks like maybe the ARC++ gardener clobbered the chrome pfq to get it to pass. Unfortunately this made it so the problem hit master +cnwan

Comparing the output of the SynChrome stage from successful builds vs unsuccessful the difference in the failing sync starts with:

CHROME_DEPS (Elapsed: 0:00:00)
----------------------------------------
[0:00:00] Started.
_____ CHROME_DEPS at refs/remotes/origin/master
[0:00:00] ________ unmanaged solution; skipping CHROME_DEPS
[0:00:00] Finished.

Followed by a bunch of "________ deleting 'src/...' " output, none of which occurs in any successful build I have looked at.

Yeah I saw that too. Maybe somehow we invalidated our gclient files?

Does this look suspicious? https://codereview.chromium.org/2099153003
  "Switch gclient config to default to unmanaged mode."

+adding hinoka/agable

Looks like all our chrome gclient files are being treated as unmanaged on our builders and our builders are deleting their own chrome source checkouts.

Sending over to agable:

Here's more info maybe you can tell us what's wrong:

This is what is failing:

Loaded .gclient config in /b/cbuild/internal_master/.cache/distfiles/target/chrome-src-internal:
solutions = [{'custom_deps': {},
  'custom_vars': {},
  'deps_file': 'releases/53.0.2782.4/DEPS',
  'name': 'CHROME_DEPS',
  'url': 'https://chrome-internal.googlesource.com/chrome/tools/buildspec.git'}]
cache_dir = '/b/git-cache'


CHROME_DEPS (Elapsed: 0:00:00)
----------------------------------------
[0:00:00] Started.
_____ CHROME_DEPS at refs/remotes/origin/master
[0:00:00] ________ unmanaged solution; skipping CHROME_DEPS
[0:00:00] Finished.

Code that generates the gclient file is gclient.py here: https://chromium.googlesource.com/chromiumos/chromite/+/master/lib/

Yes, that appears to be the case. You'll need to make chromite put "managed: True" in the spec when it runs "gclient config" for the chrome buildspecs. I don't have an easy way to modify chromite, so assigning back to sosa@.

In particular, here's what chromite is doing now:

@@@BUILD_STEP@SyncChrome@@@
************************************************************
** Start Stage SyncChrome - Tue, 28 Jun 2016 14:43:11 -0700 (PDT)
** 
** Stage that syncs Chrome sources if needed.
************************************************************
<snip>
@@@STEP_TEXT@tag 53.0.2782.4@@@
14:43:44: INFO: RunCommand: /b/cbuild/internal_master/chromite/bin/sync_chrome --reset --internal --tag 53.0.2782.4 /b/cbuild/internal_master/.cache/distfiles/target/chrome-src-internal in /b/cbuild/internal_master
14:43:45: INFO: RunCommand: /b/depot_tools/gclient revert --nohooks in /b/cbuild/internal_master/.cache/distfiles/target/chrome-src-internal
14:44:54: INFO: RunCommand: /b/depot_tools/gclient config --spec "solutions = [{'custom_deps': {},
  'custom_vars': {},
  'deps_file': 'releases/53.0.2782.4/DEPS',
  'name': 'CHROME_DEPS',
  'url': 'https://chrome-internal.googlesource.com/chrome/tools/buildspec.git'}]
cache_dir = '/b/git-cache'
" in /b/cbuild/internal_master/.cache/distfiles/target/chrome-src-internal
14:44:55: INFO: RunCommand: /b/depot_tools/gclient sync --verbose --nohooks --transitive --with_branch_heads --with_tags --reset --force --delete_unversioned_trees in /b/cbuild/internal_master/.cache/distfiles/target/chrome-src-internal



The spec which is passed to "--spec" in "gclient config" should gain another entry for "'managed': True,". Or, the "gclient sync" should gain "--head" as an additional argument.

Steven has been driving this. Sending it his way

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/71162245a070e3d1d51ecaa29f2d59e452350827

commit 71162245a070e3d1d51ecaa29f2d59e452350827
Author: Steven Bennetts <stevenjb@chromium.org>
Date: Wed Jun 29 18:37:38 2016

gclient wrapper: Add --managed flag

BUG= chromium:624177 
TEST=PFQ works again

Change-Id: I686228ca2868e04be1e0dcc4af8bf7b728d96637
Reviewed-on: https://chromium-review.googlesource.com/357140
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Steven Bennetts <stevenjb@chromium.org>

[modify] https://crrev.com/71162245a070e3d1d51ecaa29f2d59e452350827/lib/gclient.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/09d9c25ddfd2fd511278a1e3b4750b5246c0d71e

commit 09d9c25ddfd2fd511278a1e3b4750b5246c0d71e
Author: Steven Bennetts <stevenjb@chromium.org>
Date: Wed Jun 29 22:04:49 2016

Fix gclient test expectations for Manged: True change

BUG= chromium:624177 
TEST=./lib/gclient_unittest

Change-Id: I7dfd3b5b08b818117225ffb82cc1d22d85bf1cfe
Reviewed-on: https://chromium-review.googlesource.com/357361
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Steven Bennetts <stevenjb@chromium.org>

[modify] https://crrev.com/09d9c25ddfd2fd511278a1e3b4750b5246c0d71e/lib/gclient_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/c1636094f934868ae39c86ef547d0dc2f060c804

commit c1636094f934868ae39c86ef547d0dc2f060c804
Author: Steven Bennetts <stevenjb@chromium.org>
Date: Wed Jun 29 18:37:38 2016

gclient wrapper: Add --managed flag

BUG= chromium:624177 
TEST=PFQ works again

Change-Id: I6807c0025fde8d803a8a067a75225cf88d7ec1ef
Reviewed-on: https://chromium-review.googlesource.com/357570
Reviewed-by: Steven Bennetts <stevenjb@chromium.org>
Tested-by: Richard Barnette <jrbarnette@chromium.org>

[modify] https://crrev.com/c1636094f934868ae39c86ef547d0dc2f060c804/lib/gclient.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/731f8178a4a53e086524a058cadd1e102706b8d0

commit 731f8178a4a53e086524a058cadd1e102706b8d0
Author: Steven Bennetts <stevenjb@chromium.org>
Date: Wed Jun 29 22:04:49 2016

Fix gclient test expectations for Manged: True change

BUG= chromium:624177 
TEST=./lib/gclient_unittest

Change-Id: I82b327f9ea7cbc279939d2de5e5abb44b0908fcb
Reviewed-on: https://chromium-review.googlesource.com/357571
Reviewed-by: Steven Bennetts <stevenjb@chromium.org>
Tested-by: Richard Barnette <jrbarnette@chromium.org>

[modify] https://crrev.com/731f8178a4a53e086524a058cadd1e102706b8d0/lib/gclient_unittest.py

Bulk verified

Issue 624328 has been merged into this issue.