I tried to kick off a build on rambi factory branch below, and exiting R40-6420.169.0 was built successfully (I'm confused with this version which already showed up on GoldenEye) 
https://uberchromegw.corp.google.com/i/chromeos.branch/builders/rambi%20factory%20factory-rambi-6420.B/builds/138

What's purpose of preflight builder on factory branch? Will it block the branch builder due to corresponding failed preflight builder?

Noted YH tried to kick off a pre-flight build on rambi factory branch below, but it failed at "buildpackages" and "masterslavesynccompletion".

https://uberchromegw.corp.google.com/i/chromeos.branch/builders/rambi%20pre-flight%20factory-rambi-6420.B/builds/222

(view as text)
@@@BUILD_STEP BuildPackages@@@
************************************************************
@@@STEP_LINK@stdout-->stdio@https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos.branch%2Frambi_pre-flight_factory-rambi-6420.B%2F222%2F%2B%2Frecipes%2Fsteps%2FBuildPackages%2F0%2Fstdout@@@
** Start Stage BuildPackages - Mon, 13 Mar 2017 10:36:23 -0700 (PDT)
** 
** Build Chromium OS packages.
************************************************************
10:36:23: INFO: RunCommand: cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/home/chrome-bot/trunk/chroot/tmp/tmpOUi33P' -- /home/chrome-bot/trunk/chromite/bin/cros_extract_deps '--format=deps' virtual/target-sdk chromeos-base/chromite in /b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B
10:36:29: INFO: RunCommand: cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/home/chrome-bot/trunk/chroot/tmp/tmpLNBZNE' -- /home/chrome-bot/trunk/chromite/bin/cros_extract_deps --board rambi '--format=deps' virtual/target-os virtual/target-os-dev virtual/target-os-test chromeos-base/chromeos-installshim chromeos-base/chromeos-factory chromeos-base/chromeos-hwid chromeos-base/autotest-factory-install chromeos-base/autotest-all in /b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B
10:36:37: INFO: Recording packages under test
10:36:37: INFO: RunCommand: cros_sdk '--chrome_root=/b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B/.cache/distfiles/target/chrome-src-internal' 'PARALLEL_EMERGE_STATUS_FILE=/home/chrome-bot/trunk/chroot/tmp/tmpibsVbl' 'USE=chrome_internal' 'IGNORE_PREFLIGHT_BINHOST=1' 'CHROME_ORIGIN=LOCAL_SOURCE' -- ./build_packages '--board=rambi' '--accept_licenses=@CHROMEOS' --skip_chroot_upgrade --noworkon virtual/target-os virtual/target-os-dev virtual/target-os-test chromeos-base/chromeos-installshim chromeos-base/chromeos-factory chromeos-base/chromeos-hwid chromeos-base/autotest-factory-install chromeos-base/autotest-all in /b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B
ERROR   cros_sdk:enter_chroot: Not mounting chrome source: could not find CHROME_ROOT/src dir.
ERROR   cros_sdk:enter_chroot: Full path we tried: /b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B/.cache/distfiles/target/chrome-src-internal/src
ERROR   cros_sdk:enter_chroot: script called:  '--chroot' '/b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B/chroot' '--cache_dir' '/b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B/.cache' '--chrome_root' '/b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B/.cache/distfiles/target/chrome-src-internal' '--' 'PARALLEL_EMERGE_STATUS_FILE=/home/chrome-bot/trunk/chroot/tmp/tmpibsVbl' 'USE=chrome_internal' 'IGNORE_PREFLIGHT_BINHOST=1' 'CHROME_ORIGIN=LOCAL_SOURCE' './build_packages' '--board=rambi' '--accept_licenses=@CHROMEOS' '--skip_chroot_upgrade' '--noworkon' 'virtual/target-os' 'virtual/target-os-dev' 'virtual/target-os-test' 'chromeos-base/chromeos-installshim' 'chromeos-base/chromeos-factory' 'chromeos-base/chromeos-hwid' 'chromeos-base/autotest-factory-install' 'chromeos-base/autotest-all'
ERROR   cros_sdk:enter_chroot: Backtrace:  (most recent call is last)
ERROR   cros_sdk:enter_chroot:   file enter_chroot.sh, line 495, called: setup_env 
ERROR   cros_sdk:enter_chroot:   file enter_chroot.sh, line 492, called: die 'setup_env failed'
ERROR   cros_sdk:enter_chroot: 
ERROR   cros_sdk:enter_chroot: Error was:
ERROR   cros_sdk:enter_chroot:   setup_env failed
Running ['/b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B/src/scripts/sdk_lib/enter_chroot.sh', '--chroot', '/b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B/chroot', '--cache_dir', '/b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B/.cache', '--chrome_root', '/b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B/.cache/distfiles/target/chrome-src-internal', '--', 'PARALLEL_EMERGE_STATUS_FILE=/home/chrome-bot/trunk/chroot/tmp/tmpibsVbl', 'USE=chrome_internal', 'IGNORE_PREFLIGHT_BINHOST=1', 'CHROME_ORIGIN=LOCAL_SOURCE', './build_packages', '--board=rambi', '--accept_licenses=@CHROMEOS', '--skip_chroot_upgrade', '--noworkon', 'virtual/target-os', 'virtual/target-os-dev', 'virtual/target-os-test', 'chromeos-base/chromeos-installshim', 'chromeos-base/chromeos-factory', 'chromeos-base/chromeos-hwid', 'chromeos-base/autotest-factory-install', 'chromeos-base/autotest-all'] failed with exit code 1
10:36:37: ERROR: 
return code: 1
Failed command "cros_sdk '--chrome_root=/b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B/.cache/distfiles/target/chrome-src-internal' 'PARALLEL_EMERGE_STATUS_FILE=/home/chrome-bot/trunk/chroot/tmp/tmpibsVbl' 'USE=chrome_internal' 'IGNORE_PREFLIGHT_BINHOST=1' 'CHROME_ORIGIN=LOCAL_SOURCE' -- ./build_packages '--board=rambi' '--accept_licenses=@CHROMEOS' --skip_chroot_upgrade --noworkon virtual/target-os virtual/target-os-dev virtual/target-os-test chromeos-base/chromeos-installshim chromeos-base/chromeos-factory chromeos-base/chromeos-hwid chromeos-base/autotest-factory-install chromeos-base/autotest-all", cwd=/b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B, extra env={'PARALLEL_EMERGE_STATUS_FILE': '/home/chrome-bot/trunk/chroot/tmp/tmpibsVbl', 'USE': 'chrome_internal', 'IGNORE_PREFLIGHT_BINHOST': '1', 'CHROME_ORIGIN': 'LOCAL_SOURCE'}

@@@STEP_FAILURE@@@
10:36:37: ERROR: ./build_packages failed (code=1)
************************************************************
** Finished Stage BuildPackages - Mon, 13 Mar 2017 10:36:37 -0700 (PDT)
************************************************************



--------------------------------------------------------------------------------
started: Mon Mar 13 10:36:27 2017
ended: Mon Mar 13 10:36:42 2017
duration: 15 secs
status: FAILURE
status reason: 

Investigating.

hinkoka@ Is this related to your recipe changes?

If so, I probably have a fix for it, but I want to confirm.

Recipe change?

Didn't you land a Chrome Infra recipe change related to the goma command line arguments for cbuildbot? Is the time frame right for explaining this?

Oh this is a month ago.  Even so I don't quite recall doing anything with Goma, much less how this relates to it.

There had been a lot of kitchen related changes that impacted how goma finds tmpdirs lately, maybe thats related?

This builder doesn't use goma, or even build Chrome. In addition, it's building a branch that is several years old. But it's dying when it tries to mount Chrome source code into it's chroot.

I'm not sure why it's trying to do that, or why it would have worked before. I was thinking if that your changes modified the cbuildbot command line options, you might have tickled this behavior.

Here is another builder failing with the same error:

https://uberchromegw.corp.google.com/i/chromeos.branch/builders/rikku%20factory%20factory-auron-6772.B

I just realized, this builder last passed on March 3rd of 2016.

I believe the failure is caused by one of these CLs, which were submitted in May of 2016:

https://chromium-review.googlesource.com/344810
https://chromium-review.googlesource.com/344763
https://chromium-review.googlesource.com/344773
https://chromium-review.googlesource.com/344830
https://chromium-review.googlesource.com/344851

Thanks dgarrett@.

Those five merged CLs were submitted by ttsang@ who left Google.

YH, do you have bandwidth to take a look if this regression comes from the list metnioned at c#13? Thanks

ChromeOS Infra P1 Bugscrub.

P1 Bugs in this component should be important enough to get weekly status updates.

Is this already fixed?  -> Fixed
Is this no longer relevant? -> Archived or WontFix
Is this not a P1, based on go/chromeos-infra-bug-slo rubric? -> lower priority.
Is this a Feature Request rather than a bug? Type -> Feature
Is this missing important information or scope needed to decide how to proceed? -> Ask question on bug, possibly reassign.
Does this bug have the wrong owner? -> reassign.

Bugs that remain in this state next week will be downgraded to P2.

ChromeOS Infra P1 Bugscrub.

Issue untouched in a week after previous message. Downgrading to P2.

Error remains: https://uberchromegw.corp.google.com/i/chromeos.branch/builders/rambi%20pre-flight%20factory-rambi-6420.B/builds/235

Note it only affect pre-flight, not factory builder itself.

https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos.branch%2Fbuddy_factory_factory-auron-6772.B%2F61%2F%2B%2Frecipes%2Fsteps%2FBuildPackages%2F0%2Fstdout

auron-6772 factory branch has this exact issue so may I know the there is any existed solution? Thanks.

Tries to enable sync_chrome=True and see the result in tryjob - https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941110390774757616

> 08:21:26: INFO: RunCommand: /b/cbuild/shared_internal/chromite/bin/cros_sdk '--chrome_root=/b/cbuild/shared_internal/.cache/distfiles/target-factory-auron-6772.B/chrome-src-internal'

Does any one know where this catch came from? It seems the failed build_package was caused by can't find that source.

Hi akeshet,

> 23:32:01: INFO: RunCommand: cros_sdk '--chrome_root=/b/cbuild/rambi-pre-flight-branch-factory-rambi-6420.B/.cache/distfiles/target/chrome-src-internal' ...
> ERROR   cros_sdk:enter_chroot: Not mounting chrome source: could not find CHROME_ROOT/src dir.

I would like like to do some investigation but not be familiar with cbuiltbot so may I know who could be the consultant here for the question like #23? Thanks.

#25: dgarrett@ (CCd) may know the most about cbuildbot. I will also try to answer if I have a clue.

Build config changes may/may not take effect in tryjobs. That error looks like sync_chrome wasn't around in time to actually sync_chrome, but was around in time to expect it to have run when creating the chroot (and thus to mount the source into the chroot).

re#27,

Not only rambi-pre-flight-branch but also all boards in factory-auron-6772.B hit this same issue so it is not related to tryjobs.

cc Nick and Vapier for breakage of factory branch related to infra or cbuildbot problems.

how much do we actually care about rambi factory ?  this has been broken for over a year ...

This bug got revived because partners recently needed RMA shim updates and complained to us they couldn't get them from cpfe. Bumping priority.

This is a P2 per our bug SLO's: https://docs.google.com/document/d/1ZAd3EZ3hCx9rWbXh4rcIRSF4ON-CXJ2T9revQGsxMXo/edit. 

 Issue 864873  has been merged into this issue.

pretty sure this has been broken since this CL landed which disabled sync_chrome:
  https://chromium-review.googlesource.com/344810

but i guess Trevor is no longer at Google

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/ff3f4001cb3e77585a54f34eeb16227f2e877be6

commit ff3f4001cb3e77585a54f34eeb16227f2e877be6
Author: Mike Frysinger <vapier@chromium.org>
Date: Wed Jul 18 20:32:28 2018

cbuildbot_config: enable sync chrome for factory

We need the Chrome source on factory branches to build Chrome.
This was disabled in CL:344810, but we haven't passed since.

BUG=chromium:700813
TEST=None

Change-Id: Ifa618f723b5049b57fec0e1c97a9a2a380b4b629
Reviewed-on: https://chromium-review.googlesource.com/1142352
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/ff3f4001cb3e77585a54f34eeb16227f2e877be6/cbuildbot/config_dump.json
[modify] https://crrev.com/ff3f4001cb3e77585a54f34eeb16227f2e877be6/cbuildbot/cbuildbot_config.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/11f67f2697250e3ec5b9672e0e7f3e90f3258097

commit 11f67f2697250e3ec5b9672e0e7f3e90f3258097
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat Jul 21 08:08:02 2018

cbuildbot: re-enable the SyncChrome stage too

Follow up to previous commit.

BUG=chromium:700813
TEST=None

Change-Id: Ifb17f444c7b8410016a447c60eb9a84d8e7c295a
Reviewed-on: https://chromium-review.googlesource.com/1146048
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/11f67f2697250e3ec5b9672e0e7f3e90f3258097/scripts/cbuildbot.py

those CLs unbreak Trevor's CLs so Chrome is syncing again.  not entirely sure what he as going for and he's not around to ask.

however, now we find out that this ancient version of Chrome is broken.  its DEPS file has:
  'src/third_party/leveldatabase/src':
    (Var("git.chromium.org")) + '/external/leveldb.git@3f77584eb3f9754bbb7079070873ece3f30a1e6b',

the clone dies with:
Error: Command git checkout --quiet 3f77584eb3f9754bbb7079070873ece3f30a1e6b returned non-zero exit status 128 in /b/c/cbuild/repository/.cache/distfiles/target/chrome-src-internal/src/third_party/leveldatabase/src
fatal: reference is not a tree: 3f77584eb3f9754bbb7079070873ece3f30a1e6b

and indeed, that git ref no longer exists on the server:
  https://chromium.googlesource.com/external/leveldb/+/3f77584eb3f9754bbb7079070873ece3f30a1e6b

that shouldn't have happened.  this was noticed a while back:
  https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/usLyeZynBYQ

Google says AOSP actually has a copy of that ref, so we should be able to recover it:
  https://android.googlesource.com/platform/external/chromium_org/third_party/leveldatabase/src/+/3f77584eb3f9754bbb7079070873ece3f30a1e6b

i'll file another bug on the topic for the Chromium team.