many Chrome PFQ builders fail with "could not find CHROME_ROOT/src" |
|||||||
Issue descriptionE.g. https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-chrome-pfq/builds/148 10:30:10: INFO: RunCommand: /b/cbuild/internal_master/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpRY7xDk' -- /mnt/host/source/chromite/bin/cros_extract_deps '--format=deps' virtual/target-sdk chromeos-base/chromite --output-path /tmp/tmp2Jzfwl in /b/cbuild/internal_master 10:30:28: INFO: RunCommand: /b/cbuild/internal_master/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpxUJ9tO' -- /mnt/host/source/chromite/bin/cros_extract_deps --board veyron_minnie '--format=deps' virtual/target-os virtual/target-os-dev virtual/target-os-test virtual/target-os-factory virtual/target-os-factory-shim chromeos-base/autotest-all --output-path /tmp/tmph5hKDZ in /b/cbuild/internal_master 10:32:36: INFO: Recording packages under test 10:32:36: INFO: RunCommand: /b/cbuild/internal_master/chromite/bin/cros_sdk '--chrome_root=/b/cbuild/internal_master/.cache/distfiles/target/chrome-src-internal' 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmppWoZ5N' 'USE=chrome_internal' 'FEATURES=separatedebug' 'CHROME_ORIGIN=LOCAL_SOURCE' -- ./build_packages '--board=veyron_minnie' '--accept_licenses=@CHROMEOS' --withdebugsymbols --skip_chroot_upgrade --noworkon virtual/target-os virtual/target-os-dev virtual/target-os-test virtual/target-os-factory virtual/target-os-factory-shim chromeos-base/autotest-all in /b/cbuild/internal_master ERROR cros_sdk:enter_chroot: Not mounting chrome source: could not find CHROME_ROOT/src dir. ERROR cros_sdk:enter_chroot: Full path we tried: /b/cbuild/internal_master/.cache/distfiles/target/chrome-src-internal/src ERROR cros_sdk:enter_chroot: Thu Sep 8 10:32:39 PDT 2016 ERROR cros_sdk:enter_chroot: PGID PPID PID ELAPSED TIME %CPU COMMAND ERROR cros_sdk:enter_chroot: Arguments of 10: '--chroot' '/b/cbuild/internal_master/chroot' '--cache_dir' '/b/cbuild/internal_master/.cache' '--chrome_root' '/b/cbuild/internal_master/.cache/distfiles/target/chrome-src-internal' '--' 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmppWoZ5N' 'USE=chrome_internal' 'FEATURES=separatedebug' 'CHROME_ORIGIN=LOCAL_SOURCE' '--' './build_packages' '--board=veyron_minnie' '--accept_licenses=@CHROMEOS' '--withdebugsymbols' '--skip_chroot_upgrade' '--noworkon' 'virtual/target-os' 'virtual/target-os-dev' 'virtual/target-os-test' 'virtual/target-os-factory' 'virtual/target-os-factory-shim' 'chromeos-base/autotest-all' ERROR cros_sdk:enter_chroot: Backtrace: (most recent call is last) ERROR cros_sdk:enter_chroot: enter_chroot.sh:551:main(), called: setup_env ERROR cros_sdk:enter_chroot: enter_chroot.sh:548:setup_env(), called: die 'setup_env failed' ERROR cros_sdk:enter_chroot: ERROR cros_sdk:enter_chroot: Error was: ERROR cros_sdk:enter_chroot: setup_env failed 10:32:40: ERROR: return code: 1; command: /b/cbuild/internal_master/chromite/bin/cros_sdk '--chrome_root=/b/cbuild/internal_master/.cache/distfiles/target/chrome-src-internal' 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmppWoZ5N' 'USE=chrome_internal' 'FEATURES=separatedebug' 'CHROME_ORIGIN=LOCAL_SOURCE' -- ./build_packages '--board=veyron_minnie' '--accept_licenses=@CHROMEOS' --withdebugsymbols --skip_chroot_upgrade --noworkon virtual/target-os virtual/target-os-dev virtual/target-os-test virtual/target-os-factory virtual/target-os-factory-shim chromeos-base/autotest-all cwd=/b/cbuild/internal_master, extra env={'PARALLEL_EMERGE_STATUS_FILE': '/tmp/tmppWoZ5N', 'USE': 'chrome_internal', 'FEATURES': 'separatedebug', 'CHROME_ORIGIN': 'LOCAL_SOURCE'}
,
Sep 8 2016
Note that in the SyncChrome step for that build, gclient appears to delete the Chromium src directory. https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-chrome-pfq/builds/148/steps/SyncChrome/logs/stdio Nothing jumped out at me in a quick search for "gclient" in the Chrome CL's that were picked up, but I certainly didn't do an exhaustive search. https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-chrome-pfq/builds/148/steps/SyncChrome/logs/stdio
,
Sep 8 2016
Oops. Corrected link to the Chrome CL's https://chromium.googlesource.com/chromium/src/+log/55.0.2852.0..55.0.2854.0?pretty=fuller&n=10000
,
Sep 8 2016
Yup, the last delete line in SyncChrome looks bad: ________ deleting 'src' in '/b/cbuild/internal_master/.cache/distfiles/target/chrome-src-internal' This should affect chromium, so adding the current chromium sheriff in case he has any insight.
,
Sep 8 2016
The files are being deleted because of the --delete_unversioned_trees option passed to gclient sync on the builders. I tried the same command line locally, but wasn't able to repro the problem. So I looked at the depot_tools/gclient.py source. The message is printed when gclient sync detects files not under version control. There are no recent changes there, but there may be in the scm Python module it uses.
,
Sep 8 2016
The continuous builders are ok, and this is the kind of non-platform specific behavior that should affect all chrome builds I think. It's possible we picked a bad CL that has since been reverted.
,
Sep 8 2016
This is happening again: https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-chrome-pfq/builds/149/steps/BuildPackages/logs/stdio Thus 'reverted bad CL' is unlikely. I am also trying to reproduce this locally...
,
Sep 8 2016
I'll take a look as well. Yup, the two builds are using different chrome tags, so reverted bad CL is unlikely
,
Sep 8 2016
I see that bad run has gclient(706) ParseDepsFile:use_relative_paths enabled. gclient(712) ParseDepsFile:Updating deps by prepending CHROME_DEPS. but, good run didn't do that. This was changed in https://codereview.chromium.org/2324513003 @agable, do you think this relative-path change can confuse "--delete_unversioned_trees" scenario?
,
Sep 8 2016
So, https://codereview.chromium.org/2324513003 indeed looks like a culprit for me. IIUC, after this change 'src' is checked out to: /b/cbuild/internal_master/.cache/distfiles/target/chrome-src-internal/CHROME_DEPS/src (there was no CHROME_DEPS before) Again, IIUC, this is the reason why "/b/cbuild/internal_master/.cache/distfiles/target/chrome-src-internal/src" is deleted as unversioned. But cros_sdk is not aware about CHROME_DEPS and fails. @agable, can you confirm this analysis is correct?
,
Sep 8 2016
,
Sep 8 2016
,
Sep 8 2016
Aaron - wdyt about reverting this CL now while working on a fix? Many builders on the chromeos PFQ are failing due to this.
,
Sep 8 2016
It is not the fault of that CL. It is the fault of the buildspec that is being checked out. The last working build checked out 2583, which does not set use_relative_paths=True: https://chrome-internal.googlesource.com/chrome/tools/buildspec.git/+/master/releases/55.0.2853.0/DEPS The first failing build checked out 2584, which *does* set use_relative_paths=True: https://chrome-internal.googlesource.com/chrome/tools/buildspec.git/+/master/releases/55.0.2854.0/DEPS I don't know why that buildspec sets use_relative_paths=True (it obviously shouldn't, as the chromium DEPS themselves don't set that value), but reverting the gclient CL won't fix a re-run of that build.
,
Sep 8 2016
mmoss says that this is because one of the many DEPS files which gets combined into the buildspec sets use_relative_paths, and that a solution is being worked on. Assigning to mmoss@.
,
Sep 8 2016
Everyone else: Please follow https://bugs.chromium.org/p/chromium/issues/detail?id=645176 for further updates, or dupe this bug into that one if you like.
,
Sep 8 2016
Looks like 55.0.2854.0 and 55.0.2854.1 are the only buildspecs that set use_relative_paths=True, everything else before and after don't set this.
,
Sep 8 2016
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by levarum@chromium.org
, Sep 8 2016