New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

intermittent failures due to Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm'

Project Member Reported by sha...@chromium.org, Mar 22 Back to list

Issue description

imageloader-0.0.1-r1250: >>> Emerging (1 of 1) chromeos-base/imageloader-0.0.1-r1250::chromiumos for /build/arm-generic/
imageloader-0.0.1-r1250:  * Running stacked hooks for pre_pkg_setup
imageloader-0.0.1-r1250:  *    sysroot_build_bin_dir ...
imageloader-0.0.1-r1250:  [ ok ]
imageloader-0.0.1-r1250:  * Running stacked hooks for post_pkg_setup
imageloader-0.0.1-r1250:  *    python_eclass_hack ...
imageloader-0.0.1-r1250:  [ ok ]
imageloader-0.0.1-r1250:  * Running stacked hooks for pre_src_unpack
imageloader-0.0.1-r1250:  *    python_multilib_setup ...
imageloader-0.0.1-r1250:  [ ok ]
imageloader-0.0.1-r1250: >>> Unpacking source...
imageloader-0.0.1-r1250:  * /mnt/host/source/src/platform2 is not at rev 88481efde4161ad9e80bfc44769804208c64da06
imageloader-0.0.1-r1250: Cloning into '/build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1'...
imageloader-0.0.1-r1250: done.
imageloader-0.0.1-r1250: >>> Source unpacked in /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work
imageloader-0.0.1-r1250:  * Running stacked hooks for post_src_unpack
imageloader-0.0.1-r1250:  *    asan_init ...
imageloader-0.0.1-r1250:  [ ok ]
imageloader-0.0.1-r1250: >>> Preparing source in /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/imageloader ...
imageloader-0.0.1-r1250: >>> Source prepared.
imageloader-0.0.1-r1250: >>> Configuring source in /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/imageloader ...
imageloader-0.0.1-r1250: /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2.py --libdir=/usr/lib --use_flags=arm cros-debug elibc_glibc kernel_linux test userland_GNU --jobs=32 --action=configure --cache_dir=/build/arm-generic/var/cache/portage/chromeos-base/imageloader /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/imageloader/imageloader.gyp
imageloader-0.0.1-r1250: >>> Source configured.
imageloader-0.0.1-r1250: >>> Compiling source in /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/imageloader ...
imageloader-0.0.1-r1250: /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2.py --libdir=/usr/lib --use_flags=arm cros-debug elibc_glibc kernel_linux test userland_GNU --jobs=32 --action=compile --cache_dir=/build/arm-generic/var/cache/portage/chromeos-base/imageloader all
imageloader-0.0.1-r1250: >>> Source compiled.
imageloader-0.0.1-r1250: >>> Test phase: chromeos-base/imageloader-0.0.1-r1250
imageloader-0.0.1-r1250: /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2_test.py --action=pre_test --sysroot=/build/arm-generic -- 
imageloader-0.0.1-r1250: ERROR: ld.so: object 'libsandbox.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
imageloader-0.0.1-r1250: /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2_test.py --action=run --sysroot=/build/arm-generic -- /build/arm-generic/var/cache/portage/chromeos-base/imageloader/out/Default/run_tests
imageloader-0.0.1-r1250: ERROR: ld.so: object 'libsandbox.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
imageloader-0.0.1-r1250: Traceback (most recent call last):
imageloader-0.0.1-r1250:   File "/build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2_test.py", line 520, in <module>
imageloader-0.0.1-r1250:     main(sys.argv[1:])
imageloader-0.0.1-r1250:   File "/build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2_test.py", line 516, in main
imageloader-0.0.1-r1250:     getattr(p2test, options.action)()
imageloader-0.0.1-r1250:   File "/build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2_test.py", line 294, in run
imageloader-0.0.1-r1250:     self.pre_test()
imageloader-0.0.1-r1250:   File "/build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2_test.py", line 284, in pre_test
imageloader-0.0.1-r1250:     self.qemu.RegisterBinfmt()
imageloader-0.0.1-r1250:   File "/usr/lib64/python2.7/site-packages/chromite/lib/qemu.py", line 295, in RegisterBinfmt
imageloader-0.0.1-r1250:     osutils.WriteFile(self.binfmt_path, '-1')
imageloader-0.0.1-r1250:   File "/usr/lib64/python2.7/site-packages/chromite/lib/osutils.py", line 130, in WriteFile
imageloader-0.0.1-r1250:     with open(write_path, mode) as f:
imageloader-0.0.1-r1250: IOError: [Errno 13] Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm'

There have not been any recent changes to this package, last ebuild uprev was about one week ago. Let's see if this paladin cycles green.
 
Summary: arm-generic-paladin failure: imageloader: Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm' (was: arm-generic-paladin failure: IOError: [Errno 13] Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm')
Components: Infra>Client>ChromeOS
If you go through the logs, there's a similar failure in the
libcontainer package:

libcontainer-0.0.1-r1274: IOError: [Errno 13] Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm'

Cc: -sha...@chromium.org
Owner: sha...@chromium.org
Sheriff, can you please find an owner for this?
The builder cycled green and I don't have a good idea about the root cause.
Cc: smbar...@chromium.org
Labels: -Pri-3 Pri-1
Again in 
Cc: sha...@chromium.org
Owner: smbar...@chromium.org
https://luci-milo.appspot.com/buildbot/chromeos/veyron_tiger-paladin/2906
Status: Assigned
Summary: intermittent failures due to Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm' (was: arm-generic-paladin failure: imageloader: Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm')
Cc: jclinton@chromium.org
Components: -Infra>Client>ChromeOS Infra>Client>ChromeOS>CI
Cc: dianwa@chromium.org
Owner: pbe...@chromium.org
Didn't make too much headway on this. I believe the builders are on kernel 3.13, so I looked upstream to see if we were missing patches or if there were known binfmt issues. I didn't dig super deep, but didn't uncover any obvious culprits.

Other things that might be worth checking:
* any new chromite changes surrounding binfmt_misc
* uprevs of qemu or other packages

Moving to this week's sheriff. Maybe worth a look from someone on the CI team as well.
I logged in via SSH and poked around and didn't see anything immediately obvious. Both of the bots that this failed on were kernel 3.13 on Trusty which is very old at this point. They will get upgraded by virtue of the transition to Swarming so perhaps we can just wait to see if this exists after that.

They had both recently been rebooted.

Sign in to add a comment