New issue
Advanced search Search tips

Issue 919675 link

Starred by 2 users
Status: Verified
Owner:
dverkamp@chromium.org
Closed: Jan 8
Cc:
smbar...@chromium.org
dgreid@chromium.org
za...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Crostini: vhost/vsock crashes on v4.14 host kernels

Project Member Reported by dverkamp@chromium.org, Jan 7 
v4.14 host kernels running Crostini (which uses vsock) crash occasionally with a general protection fault inside hash_del_rcu() called by vhost_vsock_dev_ioctl().



I believe this is the root cause of an issue uncovered by applying
"vhost/vsock: fix use-after-free in network stack callers" in these
branches. I sometimes see a crash in hash_del_rcu() with vsock in the
call stack, and that call is protected by a newly-added check of
vsock->guest_cid, which was uninitialized before this commit.

Mainline already has a fix for this: commit a72b69dc083a931422cc8a5e33841aff7d5312f2
("vhost/vsock: fix uninitialized vhost_vsock->guest_cid").  I've already sent a request to get this pulled into the relevant stable branches, and I'll also cherry pick this for our kernel branches in the meantime.

Comment 1 by dverkamp@chromium.org, Jan 7 

Example splat from this crash:

<4>[  192.857766] CPU: 1 PID: 4945 Comm: vhost_vsock Tainted: G        W       4.14.91-09275-g0e2c0794210a #1
<4>[  192.857772] Hardware name: Google Grunt/Grunt, BIOS Google_Grunt.11031.14.0 10/01/2018
<4>[  192.857778] task: ffffa0f05b445700 task.stack: ffffb5bc81614000
<4>[  192.857790] RIP: 0010:hash_del_rcu+0x10/0x23 [vhost_vsock]
<4>[  192.857795] RSP: 0018:ffffb5bc81617e00 EFLAGS: 00010282
<4>[  192.857801] RAX: fffefefefffefefe RBX: ffffa0f0c1e48c40 RCX: fffefefefffefefe
<4>[  192.857806] RDX: 0000000000000000 RSI: 00007d0d9c346758 RDI: ffffa0f0c1e48c40
<4>[  192.857811] RBP: ffffb5bc81617e00 R08: 00000000000000af R09: 0000000000000060
<4>[  192.857816] R10: 0000000000000000 R11: ffffffffc03ea256 R12: ffffa0f0c1e40000
<4>[  192.857820] R13: 0000000000000019 R14: 0000000000000004 R15: 0000000000000008
<4>[  192.857826] FS:  00007d0d9c347700(0000) GS:ffffa0f0eed00000(0000) knlGS:0000000000000000
<4>[  192.857831] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[  192.857836] CR2: 00001324dde84000 CR3: 0000000101ee0000 CR4: 00000000001406e0
<4>[  192.857840] Call Trace:
<4>[  192.857854]  vhost_vsock_dev_ioctl+0x367/0x41f [vhost_vsock]
<4>[  192.857865]  vfs_ioctl+0x1b/0x30
<4>[  192.857872]  do_vfs_ioctl+0x492/0x6c1
<4>[  192.857879]  SyS_ioctl+0x52/0x77
<4>[  192.857887]  do_syscall_64+0x6d/0x81
<4>[  192.857896]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
<4>[  192.857903] RIP: 0033:0x7d0e5248d497
<4>[  192.857908] RSP: 002b:00007d0d9c3466e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
<4>[  192.857913] RAX: ffffffffffffffda RBX: 0000000000000019 RCX: 00007d0e5248d497
<4>[  192.857918] RDX: 00007d0d9c346750 RSI: 000000004008af60 RDI: 0000000000000019
<4>[  192.857923] RBP: 0000000000000001 R08: 00000000b0d0d806 R09: 00000000b0d0d446
<4>[  192.857928] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000003
<4>[  192.857932] R13: 000000000000001a R14: 0000000000000002 R15: 000059782b400d90
<4>[  192.857937] Code: 00 4d 85 ff 75 df 49 8d b6 50 8c 00 00 4c 89 f7 5b 41 5e 41 5f 5d e9 0d 88 fc ff 55 48 89 e5 48 8b 47 08 48 85 c0 74 14 48 8b 0f <48> 89 08 48 85 c9 74 04 48 89 41 08 48 83 67 08 00 5d c3 0f 1f
<1>[  192.858004] RIP: hash_del_rcu+0x10/0x23 [vhost_vsock] RSP: ffffb5bc81617e00
<4>[  192.858043] ---[ end trace 061a2e50b215eb9e ]---
Project Member

Comment 2 by bugdroid1@chromium.org, Jan 8

Labels: merge-merged-chromeos-4.14
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/38b8f55c70895bc6b9aa622088c97e287a589687

commit 38b8f55c70895bc6b9aa622088c97e287a589687
Author: Stefan Hajnoczi <stefanha@redhat.com>
Date: Tue Jan 08 07:46:15 2019

UPSTREAM: vhost/vsock: fix uninitialized vhost_vsock->guest_cid

The vhost_vsock->guest_cid field is uninitialized when /dev/vhost-vsock
is opened until the VHOST_VSOCK_SET_GUEST_CID ioctl is called.

kvmalloc(..., GFP_KERNEL | __GFP_RETRY_MAYFAIL) does not zero memory.
All other vhost_vsock fields are initialized explicitly so just
initialize this field too.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

(cherry picked from commit a72b69dc083a931422cc8a5e33841aff7d5312f2)

BUG= chromium:919675 
TEST=Run Crostini on grunt; verify that it does not crash

Change-Id: I49029226ce53a0f51a3a76717c8feb1471396f99
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1399245
Tested-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>

[modify] https://crrev.com/38b8f55c70895bc6b9aa622088c97e287a589687/drivers/vhost/vsock.c

Comment 3 by dverkamp@chromium.org, Jan 8

Status: Verified (was: Started)
Verified on grunt with Chrome OS R73-11546.0.
►

Sign in to add a comment