betty-chrome-pfq, caroline-chrome-pfq,chell-chrome-pfq, coral-chrome-pfq, cyan-chrome-pfq, daisy_skate-chrome-pfq, peach_pit-chrome-pfq, peppy-chrome-pfq, reef-chrome-pfq, tricky-chrome-pfq, etc. 

All the bots in #1 are failing to build for the reason issue.

This is currently causing multiple PFQ failures. Who can we assign this to?

gardener should drive this. we can grant access to builders if needed. ccing bmgordon as i wonder if it's a chroot issue.

I'm trying to figure out what might have caused this by looking at the recent changes under src/scripts and chromite/ to see if anything is relevant. 

+achuith and +dgarret: Do you have an idea where else I should look?

Looking at the root-chrome-pfq, it's been failing since the 20th, with two exceptions.

Are all of those failures the same? And why did those two exceptions pass?

I don't see any relevant CLs in the blame lists.

Was Goma updated at a relevant point in time? Have Chrome Makefiles (or whatever) been updated at relevant points?

I have no answers, only questions.

The FYI builders are fine, just the PFQ builders are failing.

This is the first failing PFQ build on betty:
https://uberchromegw.corp.google.com/i/chromeos/builders/betty-chrome-pfq/builds/1357

This CL is the only one that looks suspicious:
https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/969903

Mike - can we trying reverting your CL?

It also looks like build_packages succeeded for 3 builds late Mar 22 before starting to fail again.

That CL isn't related.

Looking at the build: https://uberchromegw.corp.google.com/i/chromeos/builders/peach_pit-chrome-pfq/builds/5231

The two commands run in the "SyncChrome" step are:

- /b/c/cbuild/repository/chromite/bin/sync_chrome --reset --ignore_locks --internal --tag 67.0.3380.0 /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal in /b/c/cbuild/repository

-  /b/c/cbuild/repository/chromium/tools/depot_tools/gclient revert --nohooks in /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal

The command run in "InitSDK" to enter chroot is:

- /b/c/cbuild/repository/chromite/bin/cros_sdk --buildbot-log-version --create --replace '--chrome_root=/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal' in /b/c/cbuild/repository
23:51:23: NOTICE: Deleting chroot.


However, the command and error we get at the BuildPackages step are:

00:05:00: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros_sdk --goma_dir /b/c/goma_client --goma_client_json /creds/service_accounts/service-account-goma-client.json '--chrome_root=/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal' 'GOMA_SERVICE_ACCOUNT_JSON_FILE=/creds/service_accounts/service-account-goma-client.json' 'USE=chrome_internal' 'USE_GOMA=true' 'FEATURES=separatedebug' 'CHROME_ORIGIN=LOCAL_SOURCE' 'GOMA_BACKEND_SOFT_STICKINESS=false' 'GOMA_CACHE_DIR=/home/chrome-bot/goma/goma_cache/BuildPackages' 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpuxuvJ3' 'GOMA_MAX_COMPILER_DISABLED_TASKS=30' 'GOMA_DIR=/home/chrome-bot/goma' 'GLOG_log_dir=/tmp/goma_tmp_dir.X3hCbO/log_dir' 'GOMA_TMP_DIR=/tmp/goma_tmp_dir.X3hCbO' 'GOMA_DEPS_CACHE_FILE=goma.deps' -- ./build_packages '--board=peach_pit' '--accept_licenses=@CHROMEOS' --withdebugsymbols --skip_chroot_upgrade --run_goma --withevents '--eventfile=/mnt/host/source/buildbot_archive/peach_pit-chrome-pfq/R67-10518.0.0-rc3/build-events.json' virtual/target-os virtual/target-os-dev virtual/target-os-test virtual/target-os-factory virtual/target-os-factory-shim chromeos-base/autotest-all in /b/c/cbuild/repository


ERROR   cros_sdk:enter_chroot: Not mounting chrome source: could not find CHROME_ROOT/src dir.

Note that the chrome_root argument to all the above commands match: "/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal"

Adding some folks from the goma team if they can see something relevant.

Shinya, will you take a look?

Hmm, not sure why goma is related... anyway I'll take a look from goma side.

The error occurred in cros_sdk:enter_chroot, and it's saying CHROME_ROOT/src dir is not found. That process is before goma is used, so IMO goma is not related (though goma can be related with some super exceptional reason, but possibility seems low)

Marking as P0 since PFQ is broken since last week

The PFQ builds are no longer suffering from this issue; they no longer fail at the BuildPackages step. The current failures are due to other reasons that I will track in other bugs.

Marking this as WontFix

Oh.. sorry to point the finger at Goma.

I'm really confused about why you would get an error entering the chroot after InitSdk finished.