New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 814455 link

Starred by 1 user
Status: Fixed
Owner:
█ nxia@chromium.org
Last visit > 30 days ago
Closed: Feb 2018
Cc:
dgarr...@chromium.org
vapier@chromium.org
█ rongchang@chromium.org
█ waihong@chromium.org
achuith@chromium.org
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

*-chrome-pfq builders are failing to start buildpackages

Project Member Reported by norvez@chromium.org, Feb 21 2018 

For example https://luci-milo.appspot.com/buildbot/chromeos/chell-chrome-pfq/1992 but it looks like all the -chrome-pfq builders have the same error. The -chromium-pfq builders are passing.

It looks like the problem is here:
"
Not mounting chrome source: could not find CHROME_ROOT/src dir.
"


"
11:12:57: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros_sdk --goma_dir /b/c/goma_client --goma_client_json /creds/service_accounts/service-account-goma-client.json '--chrome_root=/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal' 'GOMA_SERVICE_ACCOUNT_JSON_FILE=/creds/service_accounts/service-account-goma-client.json' 'USE=-debug_fission -thinlto -transparent_hugepage chrome_internal' 'USE_GOMA=true' 'FEATURES=separatedebug' 'CHROME_ORIGIN=LOCAL_SOURCE' 'GOMA_BACKEND_SOFT_STICKINESS=false' 'GOMA_CACHE_DIR=/home/chrome-bot/goma/goma_cache/BuildPackages' 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmp4nA7mH' 'GOMA_MAX_COMPILER_DISABLED_TASKS=30' 'GOMA_DIR=/home/chrome-bot/goma' 'GLOG_log_dir=/tmp/goma_tmp_dir.JOqe0Z/log_dir' 'GOMA_TMP_DIR=/tmp/goma_tmp_dir.JOqe0Z' 'GOMA_DEPS_CACHE_FILE=goma.deps' -- ./build_packages '--board=chell' '--accept_licenses=@CHROMEOS' --withdebugsymbols --skip_chroot_upgrade --run_goma --withevents '--eventfile=/mnt/host/source/buildbot_archive/chell-chrome-pfq/R66-10425.0.0-rc1/build-events.json' virtual/target-os virtual/target-os-dev virtual/target-os-test virtual/target-os-factory virtual/target-os-factory-shim chromeos-base/autotest-all in /b/c/cbuild/repository
ERROR   cros_sdk:enter_chroot: Not mounting chrome source: could not find CHROME_ROOT/src dir.
ERROR   cros_sdk:enter_chroot: Full path we tried: /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src
ERROR   cros_sdk:enter_chroot: Wed Feb 21 11:12:59 PST 2018
ERROR   cros_sdk:enter_chroot:  PGID  PPID   PID     ELAPSED     TIME %CPU COMMAND
ERROR   cros_sdk:enter_chroot: Arguments of 10:  '--chroot' '/b/c/cbuild/repository/chroot' '--cache_dir' '/b/c/cbuild/repository/.cache' '--chrome_root' '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal' '--goma_dir' '/b/c/goma_client' '--goma_client_json' '/creds/service_accounts/service-account-goma-client.json' '--' 'GOMA_SERVICE_ACCOUNT_JSON_FILE=/creds/service_accounts/service-account-goma-client.json' 'USE=-debug_fission -thinlto -transparent_hugepage chrome_internal' 'USE_GOMA=true' 'FEATURES=separatedebug' 'CHROME_ORIGIN=LOCAL_SOURCE' 'GOMA_BACKEND_SOFT_STICKINESS=false' 'GOMA_CACHE_DIR=/home/chrome-bot/goma/goma_cache/BuildPackages' 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmp4nA7mH' 'GOMA_MAX_COMPILER_DISABLED_TASKS=30' 'GOMA_DIR=/home/chrome-bot/goma' 'GLOG_log_dir=/tmp/goma_tmp_dir.JOqe0Z/log_dir' 'GOMA_TMP_DIR=/tmp/goma_tmp_dir.JOqe0Z' 'GOMA_DEPS_CACHE_FILE=goma.deps' '--' './build_packages' '--board=chell' '--accept_licenses=@CHROMEOS' '--withdebugsymbols' '--skip_chroot_upgrade' '--run_goma' '--withevents' '--eventfile=/mnt/host/source/buildbot_archive/chell-chrome-pfq/R66-10425.0.0-rc1/build-events.json' 'virtual/target-os' 'virtual/target-os-dev' 'virtual/target-os-test' 'virtual/target-os-factory' 'virtual/target-os-factory-shim' 'chromeos-base/autotest-all'
ERROR   cros_sdk:enter_chroot: Backtrace:  (most recent call is last)
ERROR   cros_sdk:enter_chroot:  enter_chroot.sh:577:main(), called: setup_env  
ERROR   cros_sdk:enter_chroot:  enter_chroot.sh:574:setup_env(), called: die 'setup_env failed' 
ERROR   cros_sdk:enter_chroot: 
ERROR   cros_sdk:enter_chroot: Error was:
ERROR   cros_sdk:enter_chroot:   setup_env failed
11:13:00: ERROR: 
"

Assigning to the gardener.

Comment 1 by achuith@chromium.org, Feb 21 2018

Cc: vapier@chromium.org
Hmm, this is almost certainly a cros CL, let me look.

Comment 2 by achuith@chromium.org, Feb 21 2018

Status: Fixed (was: Assigned)
Looks like this was a transient error. The next PFQ run succeeded:
https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/5840

The chrome informational bots also didn't see this error. I'll mark this fixed for now, and we can re-open if we see it again.

Comment 3 by vapier@chromium.org, Feb 21 2018

Cc: nxia@chromium.org
do we have to worry about locks here ?  we just landed issue 813284 to disable them when syncing the source.

looking at the SyncChrome stage, there seems to be a lot of repo deletions going on in there.  is that normal ?
[0:00:02] Finished.
----------------------------------------
________ deleting 'src/third_party/libevdev/src' in '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal'
________ deleting 'src/third_party/visualmetrics/src' in '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal'
________ deleting 'build/scripts/private/data/reliability' in '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal'
________ deleting 'src/third_party/flac' in '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal'
...lots more...

Comment 4 by dgarr...@chromium.org, Feb 21 2018 

In theory, git-cache is now safe without locks. No Chrome build recipes are using them anymore.

That doesn't mean that there aren't problems we see that they don't.

Comment 5 by achuith@chromium.org, Feb 21 2018 

Right, I saw this CL and it seems like a likely culprit:
https://chromium-review.googlesource.com/c/chromiumos/chromite/+/927687

Comment 6 by achuith@chromium.org, Feb 21 2018

Cc: -nxia@chromium.org dgarr...@chromium.org achuith@chromium.org
Owner: nxia@chromium.org
►

Sign in to add a comment