New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 824808 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jun 6
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

intermittent failures due to Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm'

Project Member Reported by sha...@chromium.org, Mar 22 2018

Issue description

imageloader-0.0.1-r1250: >>> Emerging (1 of 1) chromeos-base/imageloader-0.0.1-r1250::chromiumos for /build/arm-generic/
imageloader-0.0.1-r1250:  * Running stacked hooks for pre_pkg_setup
imageloader-0.0.1-r1250:  *    sysroot_build_bin_dir ...
imageloader-0.0.1-r1250:  [ ok ]
imageloader-0.0.1-r1250:  * Running stacked hooks for post_pkg_setup
imageloader-0.0.1-r1250:  *    python_eclass_hack ...
imageloader-0.0.1-r1250:  [ ok ]
imageloader-0.0.1-r1250:  * Running stacked hooks for pre_src_unpack
imageloader-0.0.1-r1250:  *    python_multilib_setup ...
imageloader-0.0.1-r1250:  [ ok ]
imageloader-0.0.1-r1250: >>> Unpacking source...
imageloader-0.0.1-r1250:  * /mnt/host/source/src/platform2 is not at rev 88481efde4161ad9e80bfc44769804208c64da06
imageloader-0.0.1-r1250: Cloning into '/build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1'...
imageloader-0.0.1-r1250: done.
imageloader-0.0.1-r1250: >>> Source unpacked in /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work
imageloader-0.0.1-r1250:  * Running stacked hooks for post_src_unpack
imageloader-0.0.1-r1250:  *    asan_init ...
imageloader-0.0.1-r1250:  [ ok ]
imageloader-0.0.1-r1250: >>> Preparing source in /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/imageloader ...
imageloader-0.0.1-r1250: >>> Source prepared.
imageloader-0.0.1-r1250: >>> Configuring source in /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/imageloader ...
imageloader-0.0.1-r1250: /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2.py --libdir=/usr/lib --use_flags=arm cros-debug elibc_glibc kernel_linux test userland_GNU --jobs=32 --action=configure --cache_dir=/build/arm-generic/var/cache/portage/chromeos-base/imageloader /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/imageloader/imageloader.gyp
imageloader-0.0.1-r1250: >>> Source configured.
imageloader-0.0.1-r1250: >>> Compiling source in /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/imageloader ...
imageloader-0.0.1-r1250: /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2.py --libdir=/usr/lib --use_flags=arm cros-debug elibc_glibc kernel_linux test userland_GNU --jobs=32 --action=compile --cache_dir=/build/arm-generic/var/cache/portage/chromeos-base/imageloader all
imageloader-0.0.1-r1250: >>> Source compiled.
imageloader-0.0.1-r1250: >>> Test phase: chromeos-base/imageloader-0.0.1-r1250
imageloader-0.0.1-r1250: /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2_test.py --action=pre_test --sysroot=/build/arm-generic -- 
imageloader-0.0.1-r1250: ERROR: ld.so: object 'libsandbox.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
imageloader-0.0.1-r1250: /build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2_test.py --action=run --sysroot=/build/arm-generic -- /build/arm-generic/var/cache/portage/chromeos-base/imageloader/out/Default/run_tests
imageloader-0.0.1-r1250: ERROR: ld.so: object 'libsandbox.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
imageloader-0.0.1-r1250: Traceback (most recent call last):
imageloader-0.0.1-r1250:   File "/build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2_test.py", line 520, in <module>
imageloader-0.0.1-r1250:     main(sys.argv[1:])
imageloader-0.0.1-r1250:   File "/build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2_test.py", line 516, in main
imageloader-0.0.1-r1250:     getattr(p2test, options.action)()
imageloader-0.0.1-r1250:   File "/build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2_test.py", line 294, in run
imageloader-0.0.1-r1250:     self.pre_test()
imageloader-0.0.1-r1250:   File "/build/arm-generic/tmp/portage/chromeos-base/imageloader-0.0.1-r1250/work/imageloader-0.0.1/common-mk/platform2_test.py", line 284, in pre_test
imageloader-0.0.1-r1250:     self.qemu.RegisterBinfmt()
imageloader-0.0.1-r1250:   File "/usr/lib64/python2.7/site-packages/chromite/lib/qemu.py", line 295, in RegisterBinfmt
imageloader-0.0.1-r1250:     osutils.WriteFile(self.binfmt_path, '-1')
imageloader-0.0.1-r1250:   File "/usr/lib64/python2.7/site-packages/chromite/lib/osutils.py", line 130, in WriteFile
imageloader-0.0.1-r1250:     with open(write_path, mode) as f:
imageloader-0.0.1-r1250: IOError: [Errno 13] Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm'

There have not been any recent changes to this package, last ebuild uprev was about one week ago. Let's see if this paladin cycles green.
 

Comment 2 by sha...@chromium.org, Mar 22 2018

Summary: arm-generic-paladin failure: imageloader: Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm' (was: arm-generic-paladin failure: IOError: [Errno 13] Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm')
Components: Infra>Client>ChromeOS
If you go through the logs, there's a similar failure in the
libcontainer package:

libcontainer-0.0.1-r1274: IOError: [Errno 13] Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm'

Cc: -sha...@chromium.org
Owner: sha...@chromium.org
Sheriff, can you please find an owner for this?

Comment 5 by sha...@chromium.org, Mar 23 2018

The builder cycled green and I don't have a good idea about the root cause.
Cc: smbar...@chromium.org
Labels: -Pri-3 Pri-1
Again in 
Cc: sha...@chromium.org
Owner: smbar...@chromium.org
https://luci-milo.appspot.com/buildbot/chromeos/veyron_tiger-paladin/2906
Status: Assigned (was: Untriaged)
Summary: intermittent failures due to Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm' (was: arm-generic-paladin failure: imageloader: Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm')
Cc: jclinton@chromium.org
Components: -Infra>Client>ChromeOS Infra>Client>ChromeOS>CI
Cc: dianwa@chromium.org
Owner: pbe...@chromium.org
Didn't make too much headway on this. I believe the builders are on kernel 3.13, so I looked upstream to see if we were missing patches or if there were known binfmt issues. I didn't dig super deep, but didn't uncover any obvious culprits.

Other things that might be worth checking:
* any new chromite changes surrounding binfmt_misc
* uprevs of qemu or other packages

Moving to this week's sheriff. Maybe worth a look from someone on the CI team as well.
I logged in via SSH and poked around and didn't see anything immediately obvious. Both of the bots that this failed on were kernel 3.13 on Trusty which is very old at this point. They will get upgraded by virtue of the transition to Swarming so perhaps we can just wait to see if this exists after that.

They had both recently been rebooted.

Cc: vapier@chromium.org
the binfmt/qemu logic in question was written long enough ago that i wouldn't worry about this being a "is the kernel new enough"

the code that sets up the binfmt files are also not related to qemu (the package) at all.  it's entirely written by common-mk/platform2_test.py files

i'm wondering if there's a race here somewhere.  we tried to write the code to DTRT at each step, but maybe something changed with newer versions that violate our assumptions.
Owner: vapier@chromium.org
Status: Started (was: Assigned)
I think I see a bug in the logic. I'll post a CL and at the very least, add some debug output in case it goes wrong again.
Project Member

Comment 15 by bugdroid1@chromium.org, Jun 6

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1a05b76cf1e92a2bdec4db65b9a675cf8df9c568

commit 1a05b76cf1e92a2bdec4db65b9a675cf8df9c568
Author: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Wed Jun 06 01:15:54 2018

Roll src/third_party/chromite 3c46387..784bb7f (5 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/3c46387..784bb7f


git log 3c46387..784bb7f --date=short --no-merges --format='%ad %ae %s'
2018-06-05 vapier@chromium.org cros lint: rework extension/linter lookups
2018-06-05 bmgordon@chromium.org cros_build_lib: Catch additional gethostbyaddr exception
2018-06-05 bpastene@chromium.org chromite: Roll third_party/gn_helpers forward.
2018-06-05 vapier@chromium.org qemu: fix bug in registration logic
2018-06-05 vapier@chromium.org qemu: add more logging around registration


Created with:
  gclient setdep -r src/third_party/chromite@784bb7f

The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG=chromium:None,chromium:None,chromium:824808,chromium:824808
TBR=chrome-os-gardeners@chromium.org

Change-Id: I122486e1710bce26bd16709b9d5c6d401cdb9f38
Reviewed-on: https://chromium-review.googlesource.com/1087637
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#564757}
[modify] https://crrev.com/1a05b76cf1e92a2bdec4db65b9a675cf8df9c568/DEPS

Status: Fixed (was: Started)
not sure why bugdroid didn't comment, but these two CLs should fix the issue i think:
  https://chromium-review.googlesource.com/1085859
  https://chromium-review.googlesource.com/1085860

Sign in to add a comment