mosys crash: raise-ba7b1fcb (do_fatal_log) |
|||||
Issue descriptionChrome OS Version: Starting from 10925.0.0 Chrome OS Platform: many Steps To Reproduce: (1) Seen in logs from: https://listnr.corp.google.com/product/208/report/85579923283 Expected Result: mosys doesn't crash Actual Result: mosys crashes with below stack trace. How frequently does this problem reproduce? (Always, sometimes, hard to reproduce?) Unknown. What is the impact to the user, and is there a workaround? If so, what is it? Unkown. Please provide any additional information below. Attach a screen shot or log if possible. List of crashes: https://crash.corp.google.com/browse?q=product_name%3D%27ChromeOS%27+AND+EXISTS+%28SELECT+1+FROM+UNNEST%28productdata%29+WHERE+Key%3D%27exec_name%27+AND+Value%3D%27mosys%27%29+AND+stable_signature%3D%27raise-ba7b1fcb%27 Example: crash/d5db9f14ee3031cd Magic Signature do_fatal_log Stable Signature raise-ba7b1fcb Report Time Wed, 01 Aug 2018 21:59:26 GMT Process uptime 0 sec, 824 ms Thread 0 (id: 0xc65) CRASHED [SIGABRT @ 0x00000000 ] MAGIC SIGNATURE THREAD 0x00007b481db9bdd2 (libc-2.23.so -raise.c:54 ) raise 0x00007b481db9dbf5 (libc-2.23.so -abort.c:89 ) abort 0x00007b481e75ad66 (libminijail.so -util.c:126 ) do_fatal_log 0x00007b481e756035 (libminijail.so -libminijail.c:2067 ) minijail_enter 0x00005cc926194800 (mosys + 0x00005800 ) mosys::main::hee03e99fda5baadb 0x00005cc9261b6c72 (mosys + 0x00027c72 ) std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h4c0102fcbdcfa377 0x00005cc926195dd2 (mosys + 0x00006dd2 ) main 0x00007b481db88735 (libc-2.23.so -libc-start.c:289 ) __libc_start_main 0x00005cc9261934e8 (mosys + 0x000044e8 ) _start 0x00007ffcf60f2f57 0x00005cc9261934bf (mosys + 0x000044bf ) _init Suspected CL which first appears in 10924.0.0: src/platform/mosys d955e3de 1110337 795841 Thu Jun 14 19:23:32 2018 samanthamiller@google.com mosys: Put mosys into a minijail
,
Aug 2
This is because of putting mosys into a minijail. This restricts the system calls that mosys commands are allowed to run. Since this is so dependent on the exact mosys call on the exact board, it's been impossible to catch all crashes before submitting. I don't actually have access to the crash pages, so I can't see what's happening there. If I can learn what mosys calls are causing failures on what boards, I can debug the issue. Without that information, it's purely guesswork. I'm have several CLs out adding system calls and logging more information about mosys. I'll debug more issues as I receive more information about each.
,
Aug 2
Here is the minidump of the crash. I would assume that has the syscall that failed.
,
Aug 2
the log also shows 2018-08-01T13:48:11.167553-06:00 ERR mosys[17362]: libminijail[17362]: prctl(PR_SET_KEEPCAPS) failed: Operation not permitted
,
Aug 2
I don't see a specific system call listed in the minidump, though it does look a lot like the minidump from crbug.com/869971 . The log implies that the issue comes from not allowing prctl. The CL at crrev.com/c/1160937 will hopefully fix this.
,
Aug 2
,
Aug 3
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/mosys/+/502e7577e0946779f35a4da2cf3edb330c5c854e commit 502e7577e0946779f35a4da2cf3edb330c5c854e Author: Samantha Miller <samanthamiller@google.com> Date: Fri Aug 03 04:50:12 2018 mosys: Add mosys log to easy seccomp debugging BUG= chromium:870357 TEST=built and ran Change-Id: Ib50cfe9de99a082679c81c8921e2648882e9f97b Reviewed-on: https://chromium-review.googlesource.com/1160934 Commit-Ready: Samantha Miller <samanthamiller@google.com> Tested-by: Samantha Miller <samanthamiller@google.com> Reviewed-by: Daniel Kurtz <djkurtz@chromium.org> [modify] https://crrev.com/502e7577e0946779f35a4da2cf3edb330c5c854e/src/lib.rs
,
Aug 3
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/mosys/+/723c6df47d1c07582144d11f5d5d50186ac1dcb4 commit 723c6df47d1c07582144d11f5d5d50186ac1dcb4 Author: Samantha Miller <samanthamiller@google.com> Date: Fri Aug 03 04:50:12 2018 mosys: Fix grunt seccomp error TEST=none, but shouldn't break anything BUG= chromium:870357 Change-Id: Ie7d1303b3a2702b69e6c561a75d0830ea189694c Reviewed-on: https://chromium-review.googlesource.com/1160937 Commit-Ready: Samantha Miller <samanthamiller@google.com> Tested-by: Samantha Miller <samanthamiller@google.com> Reviewed-by: Jason Clinton <jclinton@chromium.org> Reviewed-by: Daniel Kurtz <djkurtz@chromium.org> [modify] https://crrev.com/723c6df47d1c07582144d11f5d5d50186ac1dcb4/seccomp/mosys-seccomp-amd64.policy
,
Aug 3
The grunt error isn't fixed, but b/112030238 has identified the root cause and is working toward a fix. I imagine there are more problems that haven't yet reached me. Since I can't open the crash dump pages, I'm relying on people to notify me of issues through bugs. Let me know what else comes up. :)
,
Aug 4
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform2/+/f1d34ca47f1ed5dfc28b601b8aa6c18f4812b024 commit f1d34ca47f1ed5dfc28b601b8aa6c18f4812b024 Author: Samantha Miller <samanthamiller@google.com> Date: Sat Aug 04 05:14:44 2018 crash: Collect /var/log/messages for mosys debugging BUG= chromium:870357 TEST=I'm not sure how Change-Id: Iea464aaa45b9d0c3fdd8c14f2ed017da0a0d52d6 Reviewed-on: https://chromium-review.googlesource.com/1161005 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Samantha Miller <samanthamiller@google.com> Reviewed-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/f1d34ca47f1ed5dfc28b601b8aa6c18f4812b024/crash-reporter/crash_reporter_logs.conf
,
Aug 6
This issue is marked as a release blocker with no milestone associated. Please add an appropriate milestone. All release blocking issues should have milestones associated to it, so that the issue can tracked and the fixes can be pushed promptly. Thanks for your time! To disable nags, add the Disable-Nags label. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Aug 8
From issue 872034, this log: https://stainless.corp.google.com/browse/chromeos-autotest-results/224788468-chromeos-test/ mosys starts to crash after chrome restart and then repeatedly crash. 2018-08-07T17:10:43.488449+00:00 INFO session_manager[7109]: [INFO:upstart_signal_emitter.cc(35)] Emitting ash-initialized Upstart signal 2018-08-07T17:10:43.549209+00:00 WARNING mosys[7236]: libminijail[7236]: allowing syscall: socket 2018-08-07T17:10:43.549275+00:00 WARNING mosys[7236]: libminijail[7236]: allowing syscall: connect 2018-08-07T17:10:43.549293+00:00 WARNING mosys[7236]: libminijail[7236]: allowing syscall: gettimeofday 2018-08-07T17:10:43.549307+00:00 WARNING mosys[7236]: libminijail[7236]: allowing syscall: send 2018-08-07T17:10:43.549323+00:00 WARNING mosys[7236]: libminijail[7236]: allowing syscall: writev 2018-08-07T17:10:43.549597+00:00 ERR mosys[7236]: libminijail[7236]: unshare(CLONE_NEWNS) failed: Operation not permitted 2018-08-07T17:10:43.579162+00:00 INFO crash_reporter[7238]: libminijail[7238]: mount /dev/log -> /dev/log type '' 2018-08-07T17:10:43.702031+00:00 DEBUG kernel: [ 1081.204371] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs 2018-08-07T17:10:43.741886+00:00 DEBUG kernel: [ 1081.241578] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs 2018-08-07T17:10:43.778016+00:00 WARNING crash_reporter[7238]: Could not load the device policy file. 2018-08-07T17:10:43.779243+00:00 WARNING crash_reporter[7238]: [user] Received crash notification for mosys[7236] sig 6, user 1000 (developer build - not testing - always dumping) 2018-08-07T17:10:43.801341+00:00 INFO crash_reporter[7238]: State of crashed process [7236]: S (sleeping) 2018-08-07T17:10:43.804310+00:00 INFO crash_reporter[7238]: Accessing crash dir '/var/spool/crash' via symlinked handle '/proc/self/fd/5' 2018-08-07T17:10:43.809043+00:00 INFO metrics_daemon[3332]: [INFO:metrics_daemon.cc(427)] Got org.chromium.CrashReporter.UserCrash D-Bus signal 2018-08-07T17:10:43.832862+00:00 INFO crash_reporter[7238]: Stored minidump to /var/spool/crash/mosys.20180807.101043.7236.dmp 2018-08-07T17:10:43.833848+00:00 INFO crash_reporter[7238]: Leaving core file at /proc/self/fd/5/mosys.20180807.101043.7236.core due to developer image 2018-08-07T17:10:43.908397+00:00 WARNING mosys[7246]: libminijail[7246]: allowing syscall: socket 2018-08-07T17:10:43.908477+00:00 WARNING mosys[7246]: libminijail[7246]: allowing syscall: connect 2018-08-07T17:10:43.908498+00:00 WARNING mosys[7246]: libminijail[7246]: allowing syscall: gettimeofday 2018-08-07T17:10:43.908515+00:00 WARNING mosys[7246]: libminijail[7246]: allowing syscall: send 2018-08-07T17:10:43.908537+00:00 WARNING mosys[7246]: libminijail[7246]: allowing syscall: writev 2018-08-07T17:10:43.908895+00:00 ERR mosys[7246]: libminijail[7246]: unshare(CLONE_NEWNS) failed: Operation not permitted
,
Aug 8
This should be fixed by crrev.com/c/1167561.
,
Aug 9
,
Aug 9
,
Aug 10
Hi, Is this still a Dev blocker? If so, please update here with the plan to address this. Thanks.
,
Aug 13
I don't know of anything left to address on this bug. For now, I'll mark it as fixed. |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by djkurtz@chromium.org
, Aug 2