New issue
Advanced search Search tips

Issue 639102 link

Starred by 3 users

Issue metadata

Status: Verified
Owner:
Closed: Nov 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Kernel panics on Cyan PFQ

Project Member Reported by domlasko...@chromium.org, Aug 18 2016

Issue description

The Cyan PFQ failed twice in a row due to kernel panics with similar stack traces.

https://uberchromegw.corp.google.com/i/chromeos/builders/cyan-chrome-pfq/builds/63
https://uberchromegw.corp.google.com/i/chromeos/builders/cyan-chrome-pfq/builds/64


<4>[  187.284594] Call Trace:
<4>[  187.284607]  [<ffffffffbd9fcbde>] ? avc_audit_pre_callback+0x108/0x108
<4>[  187.284622]  [<ffffffffbd8b1f32>] audit_log_d_path+0x7f/0xb8
<4>[  187.284636]  [<ffffffffbda14584>] common_lsm_audit+0xdf/0x4d3
<4>[  187.284649]  [<ffffffffbd9fd91c>] slow_avc_audit+0x77/0x8f
<4>[  187.284661]  [<ffffffffbd9fde8f>] avc_has_perm+0x8f/0xbc
<4>[  187.284674]  [<ffffffffbda012e0>] inode_has_perm+0x62/0x6b
<4>[  187.284686]  [<ffffffffbda01592>] file_has_perm+0x6f/0xa7
<4>[  187.284698]  [<ffffffffbda01732>] selinux_file_permission+0x92/0x9b
<4>[  187.284712]  [<ffffffffbd9fb63e>] security_file_permission+0x1f/0x33
<4>[  187.284726]  [<ffffffffbd92ca7a>] rw_verify_area+0x80/0xd5
<4>[  187.284737]  [<ffffffffbd92cc17>] vfs_write+0x7c/0xf5
<4>[  187.284750]  [<ffffffffbd951f1c>] kernel_write+0x3f/0x56
<4>[  187.284764]  [<ffffffffbd9dffa1>] ecryptfs_write_lower+0x31/0x49
<4>[  187.284776]  [<ffffffffbd9e1102>] ecryptfs_encrypt_page+0x129/0x165
<4>[  187.284790]  [<ffffffffbd9df671>] ecryptfs_writepage+0x14/0x56
<4>[  187.284802]  [<ffffffffbd8f0ebd>] __writepage+0x17/0x36
<4>[  187.284814]  [<ffffffffbd8f1874>] write_cache_pages+0x281/0x3be
<4>[  187.284826]  [<ffffffffbd8f0ea6>] ? rcu_read_unlock_sched_notrace+0x17/0x17
<4>[  187.284843]  [<ffffffffbde240b1>] ? unix_wait_for_peer+0xe8/0xe8
<4>[  187.284857]  [<ffffffffbde6aa90>] ? _raw_spin_unlock+0xe/0x21
<4>[  187.284870]  [<ffffffffbd8f1a01>] generic_writepages+0x50/0x78
<4>[  187.284882]  [<ffffffffbd8f3000>] do_writepages+0x2a/0x2c
<4>[  187.284895]  [<ffffffffbd8eacfb>] __filemap_fdatawrite_range+0x62/0x78
<4>[  187.284908]  [<ffffffffbd8eb723>] filemap_fdatawrite+0x1f/0x21
<4>[  187.284921]  [<ffffffffbd8eb744>] filemap_write_and_wait+0x1f/0x43
<4>[  187.284934]  [<ffffffffbd9df363>] ecryptfs_put_lower_file+0x32/0x56
<4>[  187.284947]  [<ffffffffbd9dcd05>] ecryptfs_release+0x13/0x2c
<4>[  187.284959]  [<ffffffffbd9dccf2>] ? ecryptfs_flush+0x44/0x44
<4>[  187.284971]  [<ffffffffbd92e0b3>] __fput+0x102/0x1bf
<4>[  187.284982]  [<ffffffffbd92e1a8>] ____fput+0xe/0x10
<4>[  187.284995]  [<ffffffffbd857b76>] task_work_run+0x7e/0xab
<4>[  187.285008]  [<ffffffffbd840de6>] do_exit+0x411/0x97d
<4>[  187.285021]  [<ffffffffbda532e8>] ? timerqueue_del+0x52/0x5a
<4>[  187.285034]  [<ffffffffbd8421f3>] do_group_exit+0x43/0xb1
<4>[  187.285047]  [<ffffffffbd84c4ac>] get_signal+0x570/0x5ac
<4>[  187.285059]  [<ffffffffbd88bcf4>] ? hrtimer_get_res+0x44/0x44
<4>[  187.285072]  [<ffffffffbd801ff5>] do_signal+0x37/0x5f8
<4>[  187.285085]  [<ffffffffbd961904>] ? ep_poll+0x13d/0x2f5
<4>[  187.285098]  [<ffffffffbd8623ca>] ? wake_up_process+0x39/0x39
<4>[  187.285110]  [<ffffffffbd8025e2>] do_notify_resume+0x2c/0x61
<4>[  187.285122]  [<ffffffffbde6b413>] int_signal+0x12/0x17

...

<0>[  187.290301] Kernel panic - not syncing: Fatal exception
 
panic.txt
123 KB View Download
Cc: rickyz@chromium.org
Cc: -rickyz@chromium.org
Owner: rickyz@chromium.org
Assigning to rickyz@ for triage.
Status: Assigned (was: Untriaged)

Comment 4 by rickyz@chromium.org, Aug 19 2016

The issues goes something along the lines of:

Process writes to ecryptfs file, keeps file open
Process exits
In https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.18/kernel/exit.c#758:
  exit_fs(tsk);  // sets current->fs to NULL
  ...
  exit_task_work(tsk);  // puts fds

When the ecryptfs fd is closed, it attempt to write the changes to the file. This triggers a code path which ends up  an selinux access check for an ecryptfs lower file. This happens to be the first time the current process's SELinux context was checked for the ecryptfs lower file, so there is no cached entry for this access check, and it goes down the slow path. This may be why this ecryptfs happens to tickle this bug.

The check ends up generating an audit message for whatever reason, and in kernel/audit.c:audit_log_d_path, d_path is called, which expects current->fs to be valid: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.18/fs/dcache.c#2964

Short term, this can be worked around with a crappy kernel patch to avoid calling d_path if current->fs is gone:

diff --git a/kernel/audit.c b/kernel/audit.c
index 1d4df39..308336f 100644
--- a/kernel/audit.c
+++ b/kernel/audit.c
@@ -1628,6 +1628,12 @@ void audit_log_d_path(struct audit_buffer *ab, const char *prefix,
    if (prefix)
        audit_log_format(ab, "%s", prefix);
 
+   /* The process may be exiting. */
+   if (!current->fs) {
+       audit_log_string(ab, "<unknown>");
+       return;
+   }
+
    /* We will allow 11 spaces for ' (deleted)' to be appended */
    pathname = kmalloc(PATH_MAX+11, ab->gfp_mask);
    if (!pathname) {

Project Member

Comment 5 by bugdroid1@chromium.org, Aug 25 2016

Labels: merge-merged-chromeos-4.4
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/b6e7e72f4cf33e6dd28a293c13fb8d35ea0d424d

commit b6e7e72f4cf33e6dd28a293c13fb8d35ea0d424d
Author: Ricky Zhou <rickyz@chromium.org>
Date: Fri Aug 19 01:20:51 2016

CHROMIUM: Handle exiting process in audit_log_d_path.

If the process is existing, current->fs may be NULL, so we cannot
resolve paths in audit messages.

BUG= chromium:639102 
TEST=Ran on cyan.

Change-Id: Id8f98071f5f85c05c1d4257ac46fe01b5386f42e
Reviewed-on: https://chromium-review.googlesource.com/372981
Commit-Ready: Ricky Zhou <rickyz@chromium.org>
Tested-by: Ricky Zhou <rickyz@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
(cherry picked from commit 4c2b0a35a4dc9269ed229628f2fc536c3c1ce14a)
Reviewed-on: https://chromium-review.googlesource.com/375539

[modify] https://crrev.com/b6e7e72f4cf33e6dd28a293c13fb8d35ea0d424d/kernel/audit.c

Project Member

Comment 6 by bugdroid1@chromium.org, Aug 25 2016

Labels: merge-merged-release-R53-8530.B-chromeos-3.18
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ceba08b31289e8e7f756f7f4cb177e1fd962ae52

commit ceba08b31289e8e7f756f7f4cb177e1fd962ae52
Author: Ricky Zhou <rickyz@chromium.org>
Date: Fri Aug 19 01:20:51 2016

CHROMIUM: Handle exiting process in audit_log_d_path.

If the process is existing, current->fs may be NULL, so we cannot
resolve paths in audit messages.

BUG= chromium:639102 
TEST=Ran on cyan.

Change-Id: Id8f98071f5f85c05c1d4257ac46fe01b5386f42e
Reviewed-on: https://chromium-review.googlesource.com/372981
Commit-Ready: Ricky Zhou <rickyz@chromium.org>
Tested-by: Ricky Zhou <rickyz@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
(cherry picked from commit 4c2b0a35a4dc9269ed229628f2fc536c3c1ce14a)
Reviewed-on: https://chromium-review.googlesource.com/375540
Trybot-Ready: Ricky Zhou <rickyz@chromium.org>
Reviewed-by: Ricky Zhou <rickyz@chromium.org>
Commit-Queue: Ricky Zhou <rickyz@chromium.org>

[modify] https://crrev.com/ceba08b31289e8e7f756f7f4cb177e1fd962ae52/kernel/audit.c

Project Member

Comment 7 by bugdroid1@chromium.org, Aug 25 2016

Labels: merge-merged-release-R53-8530.B-chromeos-3.14
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c74ff55293770ba92c34a35c12dbe8d7590f1c29

commit c74ff55293770ba92c34a35c12dbe8d7590f1c29
Author: Ricky Zhou <rickyz@chromium.org>
Date: Fri Aug 19 01:20:51 2016

CHROMIUM: Handle exiting process in audit_log_d_path.

If the process is existing, current->fs may be NULL, so we cannot
resolve paths in audit messages.

BUG= chromium:639102 
TEST=Ran on cyan.

Change-Id: Id8f98071f5f85c05c1d4257ac46fe01b5386f42e
Reviewed-on: https://chromium-review.googlesource.com/372981
Commit-Ready: Ricky Zhou <rickyz@chromium.org>
Tested-by: Ricky Zhou <rickyz@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
(cherry picked from commit 4c2b0a35a4dc9269ed229628f2fc536c3c1ce14a)
Reviewed-on: https://chromium-review.googlesource.com/375541
Trybot-Ready: Ricky Zhou <rickyz@chromium.org>
Reviewed-by: Ricky Zhou <rickyz@chromium.org>
Commit-Queue: Ricky Zhou <rickyz@chromium.org>

[modify] https://crrev.com/c74ff55293770ba92c34a35c12dbe8d7590f1c29/kernel/audit.c

Project Member

Comment 8 by bugdroid1@chromium.org, Sep 2 2016

Labels: merge-merged-chromeos-3.14
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/348354ec4953f413f68888b1bec946dae98173c7

commit 348354ec4953f413f68888b1bec946dae98173c7
Author: Ricky Zhou <rickyz@chromium.org>
Date: Fri Aug 19 01:20:51 2016

CHROMIUM: Handle exiting process in audit_log_d_path.

If the process is existing, current->fs may be NULL, so we cannot
resolve paths in audit messages.

BUG= chromium:639102 
TEST=Ran on cyan.

Change-Id: Id8f98071f5f85c05c1d4257ac46fe01b5386f42e
Reviewed-on: https://chromium-review.googlesource.com/372981
Commit-Ready: Ricky Zhou <rickyz@chromium.org>
Tested-by: Ricky Zhou <rickyz@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
(cherry picked from commit 4c2b0a35a4dc9269ed229628f2fc536c3c1ce14a)
Reviewed-on: https://chromium-review.googlesource.com/375538
Reviewed-by: Ricky Zhou <rickyz@chromium.org>

[modify] https://crrev.com/348354ec4953f413f68888b1bec946dae98173c7/kernel/audit.c

Status: Fixed (was: Assigned)

Comment 10 by dchan@google.com, Jan 21 2017

Labels: VerifyIn-57

Comment 11 by dchan@google.com, Mar 4 2017

Labels: VerifyIn-58
Status: Verified (was: Fixed)

Sign in to add a comment