Kernel panics on Cyan PFQ |
|||||||||||
Issue descriptionThe Cyan PFQ failed twice in a row due to kernel panics with similar stack traces. https://uberchromegw.corp.google.com/i/chromeos/builders/cyan-chrome-pfq/builds/63 https://uberchromegw.corp.google.com/i/chromeos/builders/cyan-chrome-pfq/builds/64 <4>[ 187.284594] Call Trace: <4>[ 187.284607] [<ffffffffbd9fcbde>] ? avc_audit_pre_callback+0x108/0x108 <4>[ 187.284622] [<ffffffffbd8b1f32>] audit_log_d_path+0x7f/0xb8 <4>[ 187.284636] [<ffffffffbda14584>] common_lsm_audit+0xdf/0x4d3 <4>[ 187.284649] [<ffffffffbd9fd91c>] slow_avc_audit+0x77/0x8f <4>[ 187.284661] [<ffffffffbd9fde8f>] avc_has_perm+0x8f/0xbc <4>[ 187.284674] [<ffffffffbda012e0>] inode_has_perm+0x62/0x6b <4>[ 187.284686] [<ffffffffbda01592>] file_has_perm+0x6f/0xa7 <4>[ 187.284698] [<ffffffffbda01732>] selinux_file_permission+0x92/0x9b <4>[ 187.284712] [<ffffffffbd9fb63e>] security_file_permission+0x1f/0x33 <4>[ 187.284726] [<ffffffffbd92ca7a>] rw_verify_area+0x80/0xd5 <4>[ 187.284737] [<ffffffffbd92cc17>] vfs_write+0x7c/0xf5 <4>[ 187.284750] [<ffffffffbd951f1c>] kernel_write+0x3f/0x56 <4>[ 187.284764] [<ffffffffbd9dffa1>] ecryptfs_write_lower+0x31/0x49 <4>[ 187.284776] [<ffffffffbd9e1102>] ecryptfs_encrypt_page+0x129/0x165 <4>[ 187.284790] [<ffffffffbd9df671>] ecryptfs_writepage+0x14/0x56 <4>[ 187.284802] [<ffffffffbd8f0ebd>] __writepage+0x17/0x36 <4>[ 187.284814] [<ffffffffbd8f1874>] write_cache_pages+0x281/0x3be <4>[ 187.284826] [<ffffffffbd8f0ea6>] ? rcu_read_unlock_sched_notrace+0x17/0x17 <4>[ 187.284843] [<ffffffffbde240b1>] ? unix_wait_for_peer+0xe8/0xe8 <4>[ 187.284857] [<ffffffffbde6aa90>] ? _raw_spin_unlock+0xe/0x21 <4>[ 187.284870] [<ffffffffbd8f1a01>] generic_writepages+0x50/0x78 <4>[ 187.284882] [<ffffffffbd8f3000>] do_writepages+0x2a/0x2c <4>[ 187.284895] [<ffffffffbd8eacfb>] __filemap_fdatawrite_range+0x62/0x78 <4>[ 187.284908] [<ffffffffbd8eb723>] filemap_fdatawrite+0x1f/0x21 <4>[ 187.284921] [<ffffffffbd8eb744>] filemap_write_and_wait+0x1f/0x43 <4>[ 187.284934] [<ffffffffbd9df363>] ecryptfs_put_lower_file+0x32/0x56 <4>[ 187.284947] [<ffffffffbd9dcd05>] ecryptfs_release+0x13/0x2c <4>[ 187.284959] [<ffffffffbd9dccf2>] ? ecryptfs_flush+0x44/0x44 <4>[ 187.284971] [<ffffffffbd92e0b3>] __fput+0x102/0x1bf <4>[ 187.284982] [<ffffffffbd92e1a8>] ____fput+0xe/0x10 <4>[ 187.284995] [<ffffffffbd857b76>] task_work_run+0x7e/0xab <4>[ 187.285008] [<ffffffffbd840de6>] do_exit+0x411/0x97d <4>[ 187.285021] [<ffffffffbda532e8>] ? timerqueue_del+0x52/0x5a <4>[ 187.285034] [<ffffffffbd8421f3>] do_group_exit+0x43/0xb1 <4>[ 187.285047] [<ffffffffbd84c4ac>] get_signal+0x570/0x5ac <4>[ 187.285059] [<ffffffffbd88bcf4>] ? hrtimer_get_res+0x44/0x44 <4>[ 187.285072] [<ffffffffbd801ff5>] do_signal+0x37/0x5f8 <4>[ 187.285085] [<ffffffffbd961904>] ? ep_poll+0x13d/0x2f5 <4>[ 187.285098] [<ffffffffbd8623ca>] ? wake_up_process+0x39/0x39 <4>[ 187.285110] [<ffffffffbd8025e2>] do_notify_resume+0x2c/0x61 <4>[ 187.285122] [<ffffffffbde6b413>] int_signal+0x12/0x17 ... <0>[ 187.290301] Kernel panic - not syncing: Fatal exception
,
Aug 18 2016
Assigning to rickyz@ for triage.
,
Aug 19 2016
,
Aug 19 2016
The issues goes something along the lines of: Process writes to ecryptfs file, keeps file open Process exits In https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.18/kernel/exit.c#758: exit_fs(tsk); // sets current->fs to NULL ... exit_task_work(tsk); // puts fds When the ecryptfs fd is closed, it attempt to write the changes to the file. This triggers a code path which ends up an selinux access check for an ecryptfs lower file. This happens to be the first time the current process's SELinux context was checked for the ecryptfs lower file, so there is no cached entry for this access check, and it goes down the slow path. This may be why this ecryptfs happens to tickle this bug. The check ends up generating an audit message for whatever reason, and in kernel/audit.c:audit_log_d_path, d_path is called, which expects current->fs to be valid: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.18/fs/dcache.c#2964 Short term, this can be worked around with a crappy kernel patch to avoid calling d_path if current->fs is gone: diff --git a/kernel/audit.c b/kernel/audit.c index 1d4df39..308336f 100644 --- a/kernel/audit.c +++ b/kernel/audit.c @@ -1628,6 +1628,12 @@ void audit_log_d_path(struct audit_buffer *ab, const char *prefix, if (prefix) audit_log_format(ab, "%s", prefix); + /* The process may be exiting. */ + if (!current->fs) { + audit_log_string(ab, "<unknown>"); + return; + } + /* We will allow 11 spaces for ' (deleted)' to be appended */ pathname = kmalloc(PATH_MAX+11, ab->gfp_mask); if (!pathname) {
,
Aug 25 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/b6e7e72f4cf33e6dd28a293c13fb8d35ea0d424d commit b6e7e72f4cf33e6dd28a293c13fb8d35ea0d424d Author: Ricky Zhou <rickyz@chromium.org> Date: Fri Aug 19 01:20:51 2016 CHROMIUM: Handle exiting process in audit_log_d_path. If the process is existing, current->fs may be NULL, so we cannot resolve paths in audit messages. BUG= chromium:639102 TEST=Ran on cyan. Change-Id: Id8f98071f5f85c05c1d4257ac46fe01b5386f42e Reviewed-on: https://chromium-review.googlesource.com/372981 Commit-Ready: Ricky Zhou <rickyz@chromium.org> Tested-by: Ricky Zhou <rickyz@chromium.org> Reviewed-by: Dylan Reid <dgreid@chromium.org> (cherry picked from commit 4c2b0a35a4dc9269ed229628f2fc536c3c1ce14a) Reviewed-on: https://chromium-review.googlesource.com/375539 [modify] https://crrev.com/b6e7e72f4cf33e6dd28a293c13fb8d35ea0d424d/kernel/audit.c
,
Aug 25 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ceba08b31289e8e7f756f7f4cb177e1fd962ae52 commit ceba08b31289e8e7f756f7f4cb177e1fd962ae52 Author: Ricky Zhou <rickyz@chromium.org> Date: Fri Aug 19 01:20:51 2016 CHROMIUM: Handle exiting process in audit_log_d_path. If the process is existing, current->fs may be NULL, so we cannot resolve paths in audit messages. BUG= chromium:639102 TEST=Ran on cyan. Change-Id: Id8f98071f5f85c05c1d4257ac46fe01b5386f42e Reviewed-on: https://chromium-review.googlesource.com/372981 Commit-Ready: Ricky Zhou <rickyz@chromium.org> Tested-by: Ricky Zhou <rickyz@chromium.org> Reviewed-by: Dylan Reid <dgreid@chromium.org> (cherry picked from commit 4c2b0a35a4dc9269ed229628f2fc536c3c1ce14a) Reviewed-on: https://chromium-review.googlesource.com/375540 Trybot-Ready: Ricky Zhou <rickyz@chromium.org> Reviewed-by: Ricky Zhou <rickyz@chromium.org> Commit-Queue: Ricky Zhou <rickyz@chromium.org> [modify] https://crrev.com/ceba08b31289e8e7f756f7f4cb177e1fd962ae52/kernel/audit.c
,
Aug 25 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c74ff55293770ba92c34a35c12dbe8d7590f1c29 commit c74ff55293770ba92c34a35c12dbe8d7590f1c29 Author: Ricky Zhou <rickyz@chromium.org> Date: Fri Aug 19 01:20:51 2016 CHROMIUM: Handle exiting process in audit_log_d_path. If the process is existing, current->fs may be NULL, so we cannot resolve paths in audit messages. BUG= chromium:639102 TEST=Ran on cyan. Change-Id: Id8f98071f5f85c05c1d4257ac46fe01b5386f42e Reviewed-on: https://chromium-review.googlesource.com/372981 Commit-Ready: Ricky Zhou <rickyz@chromium.org> Tested-by: Ricky Zhou <rickyz@chromium.org> Reviewed-by: Dylan Reid <dgreid@chromium.org> (cherry picked from commit 4c2b0a35a4dc9269ed229628f2fc536c3c1ce14a) Reviewed-on: https://chromium-review.googlesource.com/375541 Trybot-Ready: Ricky Zhou <rickyz@chromium.org> Reviewed-by: Ricky Zhou <rickyz@chromium.org> Commit-Queue: Ricky Zhou <rickyz@chromium.org> [modify] https://crrev.com/c74ff55293770ba92c34a35c12dbe8d7590f1c29/kernel/audit.c
,
Sep 2 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/348354ec4953f413f68888b1bec946dae98173c7 commit 348354ec4953f413f68888b1bec946dae98173c7 Author: Ricky Zhou <rickyz@chromium.org> Date: Fri Aug 19 01:20:51 2016 CHROMIUM: Handle exiting process in audit_log_d_path. If the process is existing, current->fs may be NULL, so we cannot resolve paths in audit messages. BUG= chromium:639102 TEST=Ran on cyan. Change-Id: Id8f98071f5f85c05c1d4257ac46fe01b5386f42e Reviewed-on: https://chromium-review.googlesource.com/372981 Commit-Ready: Ricky Zhou <rickyz@chromium.org> Tested-by: Ricky Zhou <rickyz@chromium.org> Reviewed-by: Dylan Reid <dgreid@chromium.org> (cherry picked from commit 4c2b0a35a4dc9269ed229628f2fc536c3c1ce14a) Reviewed-on: https://chromium-review.googlesource.com/375538 Reviewed-by: Ricky Zhou <rickyz@chromium.org> [modify] https://crrev.com/348354ec4953f413f68888b1bec946dae98173c7/kernel/audit.c
,
Nov 2 2016
,
Jan 21 2017
,
Mar 4 2017
,
Mar 23 2017
|
|||||||||||
►
Sign in to add a comment |
|||||||||||
Comment 1 by domlasko...@chromium.org
, Aug 18 2016