New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 716120 link

Starred by 5 users

Issue metadata

Status: Verified
Owner:
Closed: Aug 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

autotest-deps-clouddpctest and autotest-deps-playmusic fail to emerge on ToT: ACCESS DENIED on symlinks to SYSROOT

Project Member Reported by semenzato@chromium.org, Apr 27 2017

Issue description



DEBUG:root:Running 'tar -cf /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp -C /build/caroline/usr/local/build/autotest/client/deps/CloudDPCTest-apks --use-compress-prog=pbzip2  .'
 * ACCESS DENIED:  creat:        /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp
ERROR:root:[stderr] tar (child): /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp: Cannot open: Permission denied
ERROR:root:[stderr] tar (child): Error is not recoverable: exiting now
ERROR:root:[stderr] tar: /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp: Cannot write: Broken pipe
ERROR:root:[stderr] tar: Child returned status 2
ERROR:root:[stderr] tar: Error is not recoverable: exiting now
 * ACCESS DENIED:  unlink:       /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp
Traceback (most recent call last):
  File "/build/caroline/usr/local/build/autotest/utils/packager.py", line 374, in <module>
    main()
  File "/build/caroline/usr/local/build/autotest/utils/packager.py", line 356, in main
    action=cur_action, dest_dir=options.output_dir)
  File "/build/caroline/usr/local/build/autotest/utils/packager.py", line 148, in process_packages
    build_dir, exclude_string)
  File "/build/caroline/usr/local/build/autotest/client/common_lib/packages.py", line 836, in tar_package
    os.unlink(temp_path)
OSError: [Errno 13] Permission denied: '/build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp'
>>> Source compiled.
 * --------------------------- ACCESS VIOLATION SUMMARY ---------------------------
 * LOG FILE: "/var/log/sandbox/sandbox-19930.log"
 * 
VERSION 1.0
FORMAT: F - Function called
FORMAT: S - Access Status
FORMAT: P - Path as passed to function
FORMAT: A - Absolute Path (not canonical)
FORMAT: R - Canonical Path
FORMAT: C - Command Line

F: creat
S: deny
P: /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp
A: /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp
R: /build/caroline/usr/local/build/autotest/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp
C: tar -cf /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp -C /build/caroline/usr/local/build/autotest/client/deps/CloudDPCTest-apks --use-compress-prog=pbzip2 . 

F: unlink
S: deny
P: /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp
A: /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp
R: /build/caroline/usr/local/build/autotest/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp
C: /usr/bin/python2.7 -B /build/caroline/usr/local/build/autotest/utils/packager.py -r /build/caroline/usr/local/build/autotest/packages  --dep=CloudDPCTest-apks  -a tar_only -o /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work 
 * --------------------------------------------------------------------------------

>>> Failed to emerge chromeos-base/autotest-deps-clouddpctest-0.3 for /build/caroline/, Log file:

 
Somewhat unrelated, but I saw this on the firewalld package after building with cros_workon_make and then building with build_packages. I had to use --scrub in cros_workon_make before build_packages worked again.

Comment 2 by vapier@chromium.org, Apr 27 2017

Cc: vapier@chromium.org
Components: OS>Packages
Summary: autotest-deps-clouddpctest and autotest-deps-playmusic fail to emerge on ToT: ACCESS DENIED on symlinks to SYSROOT (was: autotest-deps-clouddpctest and autotest-deps-playmusic fail to emerge on ToT)
looks like $WORKDIR has some symlinks to the SYSROOT, and it's attempting to write to those, and sandbox is (correctly) denying them

notice how the R path says:
/build/caroline/usr/local/build/...

the autotest code shouldn't be doing this.
#1 Thanks.  I tried removing /build/caroline/tmp/portage/chromeos-base but that didn't help.

I tried using FEATURES=noclean, but the "work" directory is almost empty:

(cr) ((fdbefe2...)) semenzato@luigi ~/trunk/src/scripts $ ls -la /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/
total 1888
drwxr-xr-x 2 semenzato portage    4096 Apr 27 14:01 .
drwxrwxr-x 7 semenzato portage    4096 Apr 27 14:01 ..
-rw-r--r-- 1 semenzato portage 1913930 Jan  6 13:22 arc_test_release_signed.apk
-rwxr-x--- 1 semenzato portage     584 Apr 27 14:01 CloudDPCTest-apks.py
-rw-r--r-- 1 semenzato portage       4 Apr 27 14:01 .version
(cr) ((fdbefe2...)) semenzato@luigi ~/trunk/src/scripts $ 

so I don't know how to explain this:

DEBUG:root:Running 'tar -cf /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp -C /build/caroline/usr/local/build/autotest/client/deps/CloudDPCTest-apks --use-compress-prog=pbzip2  .'
 * ACCESS DENIED:  creat:        /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp
ERROR:root:[stderr] tar (child): /build/caroline/tmp/portage/chromeos-base/autotest-deps-clouddpctest-0.3/work/autotest-work/client/deps/CloudDPCTest-apks/dep-CloudDPCTest-apks.tar.bz2.tmp: Cannot open: Permission denied

#2 Thanks.  I found the overlay that specifies a dependency on that ebuild (in project-cheets-private) but it's not been touched since last August.
#2 Oh I see:

	# Unlike all other autotest-dep projects which are under autotest repo, this
	# ebuild needs to manully create autotest package-like environment during
	# emerge stage. In addition, all autotest package requires basic fake test
	# during compile stage, so we also need to create ${PACKAGE}.py to workdir
	# and create autotest_workdir manually.
	cp "${FILESDIR}/setup.py" "${WORKDIR}/${PACKAGE}.py" || die
	if [[ ! -e "${AUTOTEST_WORKDIR}/client/deps/${PACKAGE}" ]]; then
		mkdir -p "${AUTOTEST_WORKDIR}/client/deps"
		ln -s "${WORKDIR}" "${AUTOTEST_WORKDIR}/client/deps/${PACKAGE}" || die
	fi
	autotest_src_compile
	if [[ -e "${ROOT}/${AUTOTEST_BASE}/client/deps/${PACKAGE}/.version" ]]; then
		cp "${ROOT}/${AUTOTEST_BASE}/client/deps/${PACKAGE}/.version" "${WORKDIR}/" || die
	fi
	# Clean up autotest workdir which we don't need in final package.
	rm -rf "${AUTOTEST_WORKDIR}"

Is there any way of relaxing this (new) restriction so I can build?

I tried building without the --withautotests flag to build_packages, but that didn't work, the build still tried to build those packages.

So I edited autotest-tests-cheets-9999.ebuild and removed those dependencies, but I don't understand why autotest-tests-cheets was being built if I didn't use --withautotests.

It's still building, I will report soon.

As I had hoped, the workaround of removing dependencies for those packages works fine.  I am leaving Pri-1 because I think this may break other people's builds as well.

Cc: sonnyrao@chromium.org
OK so I guess this was from stale state in my build?  Feel free to close.

Comment 10 by ecgh@chromium.org, Jun 29 2017

I just saw this same error from build_packages. I'm confused as to whether I did something wrong to cause it, or if there is something that needs to be fixed somewhere. I did "sudo rm -rf /build/pyro" to get build_packages to succeed again.
I'm still seeing this on every other kevin build, and the only "weird" thing I've done is using cros_workon_make. If we're not supposed to use cros_workon_make, we should tell people.
Owner: cychiang@chromium.org
Status: Assigned (was: Untriaged)
Saw this today, and I can't figure out how to get this to build properly, other than to remove these packages from my builds.

The autotest eclass is very strange; it symlinks a bunch of .../usr/local directories into our work tree for us, and then we unpack junk to it. I'm not sure which part is more wrong.
Cc: -chadversary@chromium.org
Oh, this issue didn't go away, right?
 

when building the package for bob. I still built the package failed on the below.


Packages failed:
	chromeos-base/autotest-deps-clouddpctest-0.3
	chromeos-base/autotest-deps-novalegacy-1.1.5
	chromeos-base/autotest-deps-cryptomigration-0.0.1
	chromeos-base/autotest-deps-notification-0.0.5-r1
	chromeos-base/autotest-deps-cameraorientation-0.2
...
The CL
https://chrome-internal-review.googlesource.com/c/chromeos/overlays/project-cheets-private/+/281598
refactored most of the autotest-deps-sth ebuilds.

autotest-deps-clouddpctest and autotest-deps-playmusic should follow it.

Ref: The doc for cheets autotest: https://docs.google.com/document/d/1JI_sh1RiRKL8tpPXnndfaDRySbeTBwC7qaGM25d5is0/edit#

 
I've hit this at least a couple of times a week for a while. Jimmy, are you the right owner for this?
Cc: cywang@chromium.org
Hi jorgelo@, do you mean you saw the failures of autotest-deps-clouddpctest and autotest-deps-playmusic ?
I can fix the ebuild of these two packages so they work like other autotest-deps-sth packages.

But if you are seeing failures of other packages like those in #14, maybe the problem is in autotet-external-deps eclass.

https://cs.corp.google.com/chromeos_public/src/third_party/chromiumos-overlay/eclass/autotest-external-dep.eclass?q=autotest-external-dep+package:%5Echromeos_public$&dr=C&l=4 
Yep, I saw failures with those two specific packages.
Got it. Thanks for the info and I am sorry for the trouble. I will fix it next week.
Status: Started (was: Assigned)
Project Member

Comment 22 by bugdroid1@chromium.org, Aug 29 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/project-cheets-private/+/0a05e27b98d0ff1e791ac1fe8b4f97383e518fbd

commit 0a05e27b98d0ff1e791ac1fe8b4f97383e518fbd
Author: Cheng-Yi Chiang <cychiang@google.com>
Date: Tue Aug 29 11:45:51 2017

Thanks Jimmy!
Did it work after the https://chrome-internal-review.googlesource.com/441317?

That still failed on my build package. :(

=== Complete: job autotest-deps-novalegacy-1.1.5 (0m2.4s) ===
Failed chromeos-base/autotest-deps-novalegacy-1.1.5 (in 0m2.4s). Your build has failed.
Pending 10/12, [Time 16:49:22 | Elapsed 1m21.4s | Load 1.67 1.89 3.74]
Packages failed:
	chromeos-base/autotest-deps-novalegacy-1.1.5
	chromeos-base/autotest-deps-cryptomigration-0.0.1
	chromeos-base/autotest-deps-notification-0.0.5-r1
	chromeos-base/autotest-deps-cameraorientation-0.2
	chromeos-base/autotest-deps-clouddpctest-0.3-r1
	chromeos-base/autotest-deps-clipboard-0.5-r1
..


(cr) ((41af251...)) wxt@nb ~/trunk/src/scripts $ emerge-nefario autotest-deps-clouddpctest
!!! CONFIG_PROTECT is empty for '/build/nefario/'
Calculating dependencies... done!

>>> Emerging (1 of 1) chromeos-base/autotest-deps-clouddpctest-0.3-r1::cheets-private for /build/nefario/
>>> Downloading 'gs://chromeos-binaries/HOME/bcs-cheets-private/project-cheets-private/chromeos-base/autotest-deps-clouddpctest/autotest-deps-clouddpctest-0.3.tbz2'
16:55:41: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-binaries/HOME/bcs-cheets-private/project-cheets-private/chromeos-base/autotest-deps-clouddpctest/autotest-deps-clouddpctest-0.3.tbz2 /var/cache/chromeos-cache/distfiles/target/autotest-deps-clouddpctest-0.3.tbz2.tmp
16:55:48: WARNING: GS_ERROR: AccessDeniedException: 403 AccessDenied
<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>overlay.nefario.private@gmail.com does not have storage.objects.list access to bucket chromeos-binaries.</Details></Error>


Maybe I don't have the access. 
Sorry, I have to build the Gru first. (since the bob/nefario/kevin...based on Gru ).

Okay, it works now. :)


Status: Verified (was: Started)
Great! Thanks for verifying!

Sign in to add a comment