HostIsCIBuilder breaks non-builder GCE machines. |
||||||||||||||
Issue description
I created a new GCE instance to test something, and am hitting a weird gclient problem.
I do a ChromiumOS checkout.
Run:
src/scripts/build_packages --board lumpy --nousepkg
This sets up the chroot, and builds everything normally, except for Chrome, which gets this error:
chromeos-chrome-55.0.2852.0_rc-r1: >>> Emerging (1 of 1) chromeos-base/chromeos-chrome-55.0.2852.0_rc-r1::chromiumos for /build/lumpy/
chromeos-chrome-55.0.2852.0_rc-r1: * Running stacked hooks for pre_pkg_setup
chromeos-chrome-55.0.2852.0_rc-r1: * sysroot_build_bin_dir ...
chromeos-chrome-55.0.2852.0_rc-r1: [ ok ]
chromeos-chrome-55.0.2852.0_rc-r1: * Running stacked hooks for pre_src_unpack
chromeos-chrome-55.0.2852.0_rc-r1: * python_multilib_setup ...
chromeos-chrome-55.0.2852.0_rc-r1: [ ok ]
chromeos-chrome-55.0.2852.0_rc-r1: >>> Unpacking source...
chromeos-chrome-55.0.2852.0_rc-r1: * CHROME_ORIGIN VALUE is SERVER_SOURCE
chromeos-chrome-55.0.2852.0_rc-r1: * Using CHROME_VERSION = 55.0.2852.0
chromeos-chrome-55.0.2852.0_rc-r1: * /mnt/host/source/chromite/bin/sync_chrome --tag=55.0.2852.0 --reset --gclient=/home/dgarrett/depot_tools/gclient /var/cache/c
hromeos-cache/distfiles/target/chrome-src
chromeos-chrome-55.0.2852.0_rc-r1: 21:25:20: INFO: RunCommand: /home/dgarrett/depot_tools/gclient revert --nohooks in /var/cache/chromeos-cache/distfiles/target/ch
rome-src
chromeos-chrome-55.0.2852.0_rc-r1: * ACCESS DENIED: mkdir: /b
chromeos-chrome-55.0.2852.0_rc-r1:
chromeos-chrome-55.0.2852.0_rc-r1:
chromeos-chrome-55.0.2852.0_rc-r1: src (ERROR)
chromeos-chrome-55.0.2852.0_rc-r1: ----------------------------------------
chromeos-chrome-55.0.2852.0_rc-r1: [0:00:00] Started.
chromeos-chrome-55.0.2852.0_rc-r1: [0:00:00] _____ src is missing, synching instead
chromeos-chrome-55.0.2852.0_rc-r1: ----------------------------------------
chromeos-chrome-55.0.2852.0_rc-r1: Traceback (most recent call last):
chromeos-chrome-55.0.2852.0_rc-r1: File "/home/dgarrett/depot_tools/gclient.py", line 2359, in <module>
chromeos-chrome-55.0.2852.0_rc-r1: sys.exit(main(sys.argv[1:]))
chromeos-chrome-55.0.2852.0_rc-r1: File "/home/dgarrett/depot_tools/gclient.py", line 2345, in main
chromeos-chrome-55.0.2852.0_rc-r1: return dispatcher.execute(OptionParser(), argv)
chromeos-chrome-55.0.2852.0_rc-r1: File "/mnt/host/depot_tools/subcommand.py", line 252, in execute
chromeos-chrome-55.0.2852.0_rc-r1: return command(parser, args[1:])
chromeos-chrome-55.0.2852.0_rc-r1: File "/home/dgarrett/depot_tools/gclient.py", line 2160, in CMDrevert
chromeos-chrome-55.0.2852.0_rc-r1: return client.RunOnDeps('revert', args)
chromeos-chrome-55.0.2852.0_rc-r1: File "/home/dgarrett/depot_tools/gclient.py", line 1550, in RunOnDeps
chromeos-chrome-55.0.2852.0_rc-r1: work_queue.flush(revision_overrides, command, args, options=self._options)
chromeos-chrome-55.0.2852.0_rc-r1: File "/mnt/host/depot_tools/gclient_utils.py", line 1035, in run
chromeos-chrome-55.0.2852.0_rc-r1: self.item.run(*self.args, **self.kwargs)
chromeos-chrome-55.0.2852.0_rc-r1: File "/home/dgarrett/depot_tools/gclient.py", line 850, in run
chromeos-chrome-55.0.2852.0_rc-r1: file_list)
chromeos-chrome-55.0.2852.0_rc-r1: File "/mnt/host/depot_tools/gclient_scm.py", line 166, in RunCommand
chromeos-chrome-55.0.2852.0_rc-r1: return getattr(self, command)(options, args, file_list)
chromeos-chrome-55.0.2852.0_rc-r1: File "/mnt/host/depot_tools/gclient_scm.py", line 744, in revert
chromeos-chrome-55.0.2852.0_rc-r1: return self.update(options, [], file_list)
chromeos-chrome-55.0.2852.0_rc-r1: File "/mnt/host/depot_tools/gclient_scm.py", line 442, in update
chromeos-chrome-55.0.2852.0_rc-r1: self._UpdateMirror(mirror, options)
chromeos-chrome-55.0.2852.0_rc-r1: File "/mnt/host/depot_tools/gclient_scm.py", line 903, in _UpdateMirror
chromeos-chrome-55.0.2852.0_rc-r1: lock_timeout=getattr(options, 'lock_timeout', 0))
chromeos-chrome-55.0.2852.0_rc-r1: File "/mnt/host/depot_tools/git_cache.py", line 422, in populate
chromeos-chrome-55.0.2852.0_rc-r1: gclient_utils.safe_makedirs(self.GetCachePath())
chromeos-chrome-55.0.2852.0_rc-r1: File "/mnt/host/depot_tools/gclient_utils.py", line 253, in safe_makedirs
chromeos-chrome-55.0.2852.0_rc-r1: os.makedirs(tree)
chromeos-chrome-55.0.2852.0_rc-r1: File "/usr/lib64/python2.7/os.py", line 150, in makedirs
chromeos-chrome-55.0.2852.0_rc-r1: makedirs(head, mode)
chromeos-chrome-55.0.2852.0_rc-r1: File "/usr/lib64/python2.7/os.py", line 157, in makedirs
chromeos-chrome-55.0.2852.0_rc-r1: mkdir(name, mode)
chromeos-chrome-55.0.2852.0_rc-r1: OSError: [Errno 13] Permission denied: '/b'
chromeos-chrome-55.0.2852.0_rc-r1: 21:25:20: WARNING: Chrome checkout appears corrupt. Clobbering.
,
Sep 8 2016
Hum... since this image was created from a stock Ubuntu image not the ChromeOS builder image I didn't think it would run Puppet, but it looks like it is somehow.
,
Sep 8 2016
Note that sync_chrome (and so gclient) is being invoked from the ebuild, and so from inside the chroot. I'm not sure why it's trying to reference /b directly, but that seems wrong.
,
Sep 8 2016
Is there anything in gclient that is aware it's on a builder and changes it's behavior? If so, how does it figure that out?
,
Sep 9 2016
No, gclient doesn't change any behavior based on host. It seems most likely that the .gclient file contains a "cache_dir = /b/some/path" directive, so it is trying to ensure that path exists.
,
Sep 9 2016
Ah, that makes sense! One of things I found was that puppet had updated this machine when I wasn't expecting it too, so that setting is present.
,
Sep 9 2016
Ah... not puppet, but a bit of cbuildbot code that is looking at the domain name. HostIsCIBuilder looks at the domain name, and that controls the .gclient file we write out. https://cs.corp.google.com/chromeos_public/chromite/lib/cros_build_lib.py?rcl=b70a67edd69a04ff7cb7e13c65f73ef39ee4cf59&l=769 https://cs.corp.google.com/chromeos_public/chromite/lib/gclient.py?rcl=25352ba86a350a1fc58bfc3f6a9857eca5d5e45b&l=188
,
Sep 9 2016
Do we know why it got access denied? The fix I made in HostIsCIBuilder was to let it include GCE builders. this problem should have been there since the cache was introduced, at least in the physical machines. Figuring out the access denied problem should help to resolve this issue? The HostIsCIBuilder is a hack solution though (as what it's commented there).
,
Sep 10 2016
When you emerge Chrome... 1) This is done INSIDE the chroot. 2) It first looks for a binary prebuilt, if there isn't one, it does a build. 3) When it builds, it looks for source. If it doesn't have source, it fetches it. On the builders, we always fetch the Chrome source in the SyncChrome stage, which is run OUTSIDE the chroot. /b/git-cache is a valid directory to use at that time. When the ebuild looks for source, it finds it, and so never tries to fetch it. So.... the problems I see.... 1) We never test fetching source via the ebuild. Most ebuilds work, but Chrome has a super weird ebuild. 2) We use domain names to see if we are on builder. That broke when I used a GCE instance that was NOT a build machine to do a build because the domain was wrong. 3) We try to use /b/git-cache inside the chroot, where it will probably never work.
,
Sep 10 2016
So.... I see two fixes, and it would be nice to have both. 1) Stop using domain names to see if we are a builder. 2) _GetGclientSpec should also check to see if the build is inside a chroot, or whoever sets use_cache should use False if invoked from the ebuild. Whatever is simpler.
,
Sep 13 2016
,
Sep 22 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/6077dbb941eae71358a066d791bb5fc10b637ac4 commit 6077dbb941eae71358a066d791bb5fc10b637ac4 Author: Ningning Xia <nxia@chromium.org> Date: Sat Sep 10 01:09:44 2016 Change cache-dir for gclient inside chroot. Enable cache-dir as '/tmp/b/git-cache' for gclient inside chroot. BUG= chromium:645239 TEST=run build_packages Change-Id: Id1a4f1f40d8de5d45b14782a6259142ae33e3e73 Reviewed-on: https://chromium-review.googlesource.com/384315 Commit-Ready: Ningning Xia <nxia@chromium.org> Tested-by: Ningning Xia <nxia@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/6077dbb941eae71358a066d791bb5fc10b637ac4/lib/gclient_unittest.py [modify] https://crrev.com/6077dbb941eae71358a066d791bb5fc10b637ac4/lib/gclient.py
,
Sep 22 2016
,
Oct 7 2016
,
Oct 10 2016
,
Nov 19 2016
,
Jan 21 2017
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Oct 14 2017
|
||||||||||||||
►
Sign in to add a comment |
||||||||||||||
Comment 1 by dgarr...@chromium.org
, Sep 8 2016