New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 870357 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Aug 13
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

mosys crash: raise-ba7b1fcb (do_fatal_log)

Project Member Reported by djkurtz@chromium.org, Aug 2

Issue description

Chrome OS Version: Starting from 10925.0.0
Chrome OS Platform: many

Steps To Reproduce:
(1) Seen in logs from: https://listnr.corp.google.com/product/208/report/85579923283

Expected Result:
mosys doesn't crash

Actual Result:
mosys crashes with below stack trace.

How frequently does this problem reproduce? (Always, sometimes, hard to reproduce?)
Unknown.


What is the impact to the user, and is there a workaround? If so, what is it?
Unkown.

Please provide any additional information below. Attach a screen shot or log if possible.

List of crashes:
https://crash.corp.google.com/browse?q=product_name%3D%27ChromeOS%27+AND+EXISTS+%28SELECT+1+FROM+UNNEST%28productdata%29+WHERE+Key%3D%27exec_name%27+AND+Value%3D%27mosys%27%29+AND+stable_signature%3D%27raise-ba7b1fcb%27

Example: crash/d5db9f14ee3031cd

Magic Signature do_fatal_log
Stable Signature raise-ba7b1fcb
Report Time Wed, 01 Aug 2018 21:59:26 GMT
Process uptime 0 sec, 824 ms

Thread 0 (id: 0xc65) CRASHED [SIGABRT @ 0x00000000 ] MAGIC SIGNATURE THREAD
0x00007b481db9bdd2	(libc-2.23.so -raise.c:54 )	raise
0x00007b481db9dbf5	(libc-2.23.so -abort.c:89 )	abort
0x00007b481e75ad66	(libminijail.so -util.c:126 )	do_fatal_log
0x00007b481e756035	(libminijail.so -libminijail.c:2067 )	minijail_enter
0x00005cc926194800	(mosys + 0x00005800 )	mosys::main::hee03e99fda5baadb
0x00005cc9261b6c72	(mosys + 0x00027c72 )	std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h4c0102fcbdcfa377
0x00005cc926195dd2	(mosys + 0x00006dd2 )	main
0x00007b481db88735	(libc-2.23.so -libc-start.c:289 )	__libc_start_main
0x00005cc9261934e8	(mosys + 0x000044e8 )	_start
0x00007ffcf60f2f57		
0x00005cc9261934bf	(mosys + 0x000044bf )	_init


Suspected CL which first appears in 10924.0.0:

src/platform/mosys
d955e3de	1110337	795841	Thu Jun 14 19:23:32 2018	samanthamiller@google.com	mosys: Put mosys into a minijail
 
Cc: za...@chromium.org smbar...@chromium.org jclinton@chromium.org dgreid@chromium.org
The feedback report includes more details in its SYSLOG:

2018-08-01T13:48:11.098950-06:00 WARNING mosys[17362]: libminijail[17362]: allowing syscall: socket
2018-08-01T13:48:11.099001-06:00 WARNING mosys[17362]: libminijail[17362]: allowing syscall: connect
2018-08-01T13:48:11.099008-06:00 WARNING mosys[17362]: libminijail[17362]: allowing syscall: sendto
2018-08-01T13:48:11.099016-06:00 WARNING mosys[17362]: libminijail[17362]: allowing syscall: writev
2018-08-01T13:48:11.141964-06:00 INFO avahi-daemon[1183]: Withdrawing address record for <IPv4: 1> on wlan0.
2018-08-01T13:48:11.142005-06:00 INFO avahi-daemon[1183]: Leaving mDNS multicast group on interface wlan0.IPv4 with address <IPv4: 1>.
2018-08-01T13:48:11.145227-06:00 INFO avahi-daemon[1183]: Interface wlan0.IPv4 no longer relevant for mDNS.
2018-08-01T13:48:11.167553-06:00 ERR mosys[17362]: libminijail[17362]: prctl(PR_SET_KEEPCAPS) failed: Operation not permitted
...
2018-08-01T13:48:11.325123-06:00 WARNING crash_reporter[17375]: [user] Received crash notification for mosys[17362] sig 6, user 0 (handling)
2018-08-01T13:48:11.351055-06:00 INFO crash_reporter[17375]: State of crashed process [17362]: S (sleeping)
2018-08-01T13:48:11.390089-06:00 INFO crash_reporter[17375]: Stored minidump to /var/spool/crash/mosys.20180801.134811.17362.dmp
2018-08-01T13:48:11.391338-06:00 WARNING crash_reporter[17375]: [ARC] Received crash notification for mosys[17362] sig 6, user 0 (ignoring - crash origin is not ARC)

This is because of putting mosys into a minijail. This restricts the system calls that mosys commands are allowed to run. Since this is so dependent on the exact mosys call on the exact board, it's been impossible to catch all crashes before submitting.

I don't actually have access to the crash pages, so I can't see what's happening there.

If I can learn what mosys calls are causing failures on what boards, I can debug the issue. Without that information, it's purely guesswork. I'm have several CLs out adding system calls and logging more information about mosys. I'll debug more issues as I receive more information about each.
Here is the minidump of the crash. I would assume that has the syscall that failed.
upload_file_minidump-18541dc12afb16de.dmp
28.6 KB Download
the log also shows
2018-08-01T13:48:11.167553-06:00 ERR mosys[17362]: libminijail[17362]: prctl(PR_SET_KEEPCAPS) failed: Operation not permitted

I don't see a specific system call listed in the minidump, though it does look a lot like the minidump from  crbug.com/869971 .

The log implies that the issue comes from not allowing prctl. The CL at crrev.com/c/1160937 will hopefully fix this.
Owner: samanthamiller@chromium.org
Project Member

Comment 7 by bugdroid1@chromium.org, Aug 3

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/mosys/+/502e7577e0946779f35a4da2cf3edb330c5c854e

commit 502e7577e0946779f35a4da2cf3edb330c5c854e
Author: Samantha Miller <samanthamiller@google.com>
Date: Fri Aug 03 04:50:12 2018

mosys: Add mosys log to easy seccomp debugging

BUG= chromium:870357 
TEST=built and ran

Change-Id: Ib50cfe9de99a082679c81c8921e2648882e9f97b
Reviewed-on: https://chromium-review.googlesource.com/1160934
Commit-Ready: Samantha Miller <samanthamiller@google.com>
Tested-by: Samantha Miller <samanthamiller@google.com>
Reviewed-by: Daniel Kurtz <djkurtz@chromium.org>

[modify] https://crrev.com/502e7577e0946779f35a4da2cf3edb330c5c854e/src/lib.rs

Project Member

Comment 8 by bugdroid1@chromium.org, Aug 3

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/mosys/+/723c6df47d1c07582144d11f5d5d50186ac1dcb4

commit 723c6df47d1c07582144d11f5d5d50186ac1dcb4
Author: Samantha Miller <samanthamiller@google.com>
Date: Fri Aug 03 04:50:12 2018

mosys: Fix grunt seccomp error

TEST=none, but shouldn't break anything
BUG= chromium:870357 

Change-Id: Ie7d1303b3a2702b69e6c561a75d0830ea189694c
Reviewed-on: https://chromium-review.googlesource.com/1160937
Commit-Ready: Samantha Miller <samanthamiller@google.com>
Tested-by: Samantha Miller <samanthamiller@google.com>
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Reviewed-by: Daniel Kurtz <djkurtz@chromium.org>

[modify] https://crrev.com/723c6df47d1c07582144d11f5d5d50186ac1dcb4/seccomp/mosys-seccomp-amd64.policy

The grunt error isn't fixed, but b/112030238 has identified the root cause and is working toward a fix.

I imagine there are more problems that haven't yet reached me. Since I can't open the crash dump pages, I'm relying on people to notify me of issues through bugs. Let me know what else comes up. :)
Project Member

Comment 10 by bugdroid1@chromium.org, Aug 4

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/f1d34ca47f1ed5dfc28b601b8aa6c18f4812b024

commit f1d34ca47f1ed5dfc28b601b8aa6c18f4812b024
Author: Samantha Miller <samanthamiller@google.com>
Date: Sat Aug 04 05:14:44 2018

crash: Collect /var/log/messages for mosys debugging

BUG= chromium:870357 
TEST=I'm not sure how

Change-Id: Iea464aaa45b9d0c3fdd8c14f2ed017da0a0d52d6
Reviewed-on: https://chromium-review.googlesource.com/1161005
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Samantha Miller <samanthamiller@google.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/f1d34ca47f1ed5dfc28b601b8aa6c18f4812b024/crash-reporter/crash_reporter_logs.conf

Project Member

Comment 11 by sheriffbot@chromium.org, Aug 6

This issue is marked as a release blocker with no milestone associated. Please add an appropriate milestone.

All release blocking issues should have milestones associated to it, so that the issue can tracked and the fixes can be pushed promptly.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
From issue 872034, this log:
https://stainless.corp.google.com/browse/chromeos-autotest-results/224788468-chromeos-test/

mosys starts to crash after chrome restart and then repeatedly crash.

2018-08-07T17:10:43.488449+00:00 INFO session_manager[7109]: [INFO:upstart_signal_emitter.cc(35)] Emitting ash-initialized Upstart signal
2018-08-07T17:10:43.549209+00:00 WARNING mosys[7236]: libminijail[7236]: allowing syscall: socket
2018-08-07T17:10:43.549275+00:00 WARNING mosys[7236]: libminijail[7236]: allowing syscall: connect
2018-08-07T17:10:43.549293+00:00 WARNING mosys[7236]: libminijail[7236]: allowing syscall: gettimeofday
2018-08-07T17:10:43.549307+00:00 WARNING mosys[7236]: libminijail[7236]: allowing syscall: send
2018-08-07T17:10:43.549323+00:00 WARNING mosys[7236]: libminijail[7236]: allowing syscall: writev
2018-08-07T17:10:43.549597+00:00 ERR mosys[7236]: libminijail[7236]: unshare(CLONE_NEWNS) failed: Operation not permitted

2018-08-07T17:10:43.579162+00:00 INFO crash_reporter[7238]: libminijail[7238]: mount /dev/log -> /dev/log type ''
2018-08-07T17:10:43.702031+00:00 DEBUG kernel: [ 1081.204371] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
2018-08-07T17:10:43.741886+00:00 DEBUG kernel: [ 1081.241578] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
2018-08-07T17:10:43.778016+00:00 WARNING crash_reporter[7238]: Could not load the device policy file.
2018-08-07T17:10:43.779243+00:00 WARNING crash_reporter[7238]: [user] Received crash notification for mosys[7236] sig 6, user 1000 (developer build - not testing - always dumping)
2018-08-07T17:10:43.801341+00:00 INFO crash_reporter[7238]: State of crashed process [7236]: S (sleeping)
2018-08-07T17:10:43.804310+00:00 INFO crash_reporter[7238]: Accessing crash dir '/var/spool/crash' via symlinked handle '/proc/self/fd/5'
2018-08-07T17:10:43.809043+00:00 INFO metrics_daemon[3332]: [INFO:metrics_daemon.cc(427)] Got org.chromium.CrashReporter.UserCrash D-Bus signal
2018-08-07T17:10:43.832862+00:00 INFO crash_reporter[7238]: Stored minidump to /var/spool/crash/mosys.20180807.101043.7236.dmp
2018-08-07T17:10:43.833848+00:00 INFO crash_reporter[7238]: Leaving core file at /proc/self/fd/5/mosys.20180807.101043.7236.core due to developer image
2018-08-07T17:10:43.908397+00:00 WARNING mosys[7246]: libminijail[7246]: allowing syscall: socket
2018-08-07T17:10:43.908477+00:00 WARNING mosys[7246]: libminijail[7246]: allowing syscall: connect
2018-08-07T17:10:43.908498+00:00 WARNING mosys[7246]: libminijail[7246]: allowing syscall: gettimeofday
2018-08-07T17:10:43.908515+00:00 WARNING mosys[7246]: libminijail[7246]: allowing syscall: send
2018-08-07T17:10:43.908537+00:00 WARNING mosys[7246]: libminijail[7246]: allowing syscall: writev
2018-08-07T17:10:43.908895+00:00 ERR mosys[7246]: libminijail[7246]: unshare(CLONE_NEWNS) failed: Operation not permitted
This should be fixed by crrev.com/c/1167561.
Labels: M-70
Components: -OS>Systems>Bluetooth OS>Systems>Minijail Security

Comment 16 Deleted

Comment 17 Deleted

Hi, Is this still a Dev blocker? If so, please update here with the plan to address this. Thanks.
Status: Fixed (was: Assigned)
I don't know of anything left to address on this bug. For now, I'll mark it as fixed.

Sign in to add a comment