New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 773474 link

Starred by 4 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Blocking:
issue 627304
issue 760007
issue 769970
issue 773816
issue 779308
issue 779567
issue 779664



Sign in to add a comment

imageloader verity mount: mapper device has dangling refcount on kernel v3.14 and older

Project Member Reported by kerrnel@chromium.org, Oct 10 2017

Issue description

When testing the new --unmount_all command in imageloader, I noticed that it works great, except on 3.14 kernels (veyron_minnie, samus, etc.) it fails to clean up the /dev/mapper/${GUID} device and the failure is 100% reproducible.

I noticed this because I was running the script (https://raw.githubusercontent.com/jay0lee/cros-scripts/master/boot-loop.sh) to check for data corruption, and I noticed that with the changes to unmount flash, stateful was still becoming corrupted after many reboots.

Looking in /var/log/messages:
imageloader[4162]: dm_task_run failed!
imageloader[4162]: Device mapper remove failed: Device or resource busy
mageloader[4162]: Failed to cleanup "/run/imageloader/PepperFlashPlayer"
imageloader[4162]: --unmount_all failed!

Running dmsetup remove by hand yields the same result:

localhost ~ # dmsetup remove /dev/mapper/8C54D270F59691AC9823D20CEFA0DB416187F07193A3AB5F1855720642A84516 
device-mapper: remove ioctl on 8C54D270F59691AC9823D20CEFA0DB416187F07193A3AB5F1855720642A84516 failed: Device or resource busy
Command failed

Note that something appears to hold an open reference to the device, even though the mount point ("/run/imageloader/PepperFlashPlayer") is indeed unmounted.

localhost ~ # dmsetup -C info
Name                                                             Maj Min Stat Open Targ Event  UUID                            
8C54D270F59691AC9823D20CEFA0DB416187F07193A3AB5F1855720642A84516 254   2 LI-r    1    1      0     

Note that this bug is also present in the original quick fix that was hard coded for Flash player. I believe there is most likely a bug in the verity or mapper code in the kernel that is not cleaning up the references properly when the mount is removed.

This is a very serious bug because it means on 3.14 devices, Flash player is still not being unmounted correctly, and so the stateful storage is still vulnerable to corruption over reboots. 

To reproduce this:

1) make rootfs writeable (/usr/share/vboot/bin/make_dev_ssd.sh ... )
2) vim /etc/chrome_dev.conf and add a line to the bottom:
--ppapi-flash-version=0.0.0.0
--component-updater=fast-update,test-request
3) restart ui
5) reboot the device

Flash player will now be mounted at /run/imageloader/PepperFlashPlayer.

Run the `stop ui` command and Flash player should unmount, and the mapper device and loop device should all be cleared. But on 3.14 you will probably see the /dev/mapper/FOO device still present, and errors about it in /var/log/messages.
 
Showing comments 46 - 145 of 145 Older
Blocking: 779664
Blocking: 779308
Project Member

Comment 48 by bugdroid1@chromium.org, Oct 31 2017

Labels: merge-merged-release-R62-9901.B-chromeos-3.14
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/74f6f72a9158f5541c00e27210f57bc7795c35f2

commit 74f6f72a9158f5541c00e27210f57bc7795c35f2
Author: Miklos Szeredi <mszeredi@suse.cz>
Date: Tue Oct 31 19:44:18 2017

UPSTREAM: vfs: add d_is_dir()

Add d_is_dir(dentry) helper which is analogous to S_ISDIR().

To avoid confusion, rename d_is_directory() to d_can_lookup().

Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>

BUG= chromium:773474 
TEST=Check with cl:668107 and cl:668107 modified to tack dm-2 that the
ref count of the dm-2 superblock reach 0 and imageloader --unmount
succed.

Reviewed-by: J. Bruce Fields <bfields@redhat.com>
(cherry picked from commit 44b1d53043c482225196e8a9cd9f35163a1b3336)
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>

Change-Id: Icd43a6f288b5e2f0ccd32419f414e261b8e62d92
Reviewed-on: https://chromium-review.googlesource.com/720086
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
(cherry picked from commit fcf3fad344e1f0ce4e48069bfb093abc5e7a85e9)
Reviewed-on: https://chromium-review.googlesource.com/746944
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
Commit-Queue: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/74f6f72a9158f5541c00e27210f57bc7795c35f2/include/linux/dcache.h
[modify] https://crrev.com/74f6f72a9158f5541c00e27210f57bc7795c35f2/fs/namei.c

Project Member

Comment 49 by bugdroid1@chromium.org, Oct 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/8534e966791de83670968f2c1b1a629df50eefbe

commit 8534e966791de83670968f2c1b1a629df50eefbe
Author: Miklos Szeredi <mszeredi@suse.cz>
Date: Tue Oct 31 19:44:24 2017

UPSTREAM: vfs: rename: move d_move() up

Move the d_move() in vfs_rename_dir() up, similarly to how it's done in
vfs_rename_other().  The next patch will consolidate these two functions
and this is the only structural difference between them.

I'm not sure if doing the d_move() after the dput is even valid.  But there
may be a logical explanation for that.  But moving the d_move() before the
dput() (and the mutex_unlock()) should definitely not hurt.

BUG= chromium:773474 
TEST=Check with cl:668107 and cl:668107 modified to tack dm-2 that the
ref count of the dm-2 superblock reach 0 and imageloader --unmount
succed.

Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Reviewed-by: J. Bruce Fields <bfields@redhat.com>
(cherry picked from commit de22a4c3720a96f1c2ebf12b0857b6db6a991f2c)
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>

Change-Id: Iaa92f494603cc97039a62e9b36ee198481aeeea8
Reviewed-on: https://chromium-review.googlesource.com/720087
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
(cherry picked from commit 0a4d19c58127adcf251a67dc6f3b167e1dbb82b8)
Reviewed-on: https://chromium-review.googlesource.com/747208
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
Commit-Queue: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/8534e966791de83670968f2c1b1a629df50eefbe/fs/namei.c

Project Member

Comment 50 by bugdroid1@chromium.org, Oct 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/edec2c60ec25753b355614237672ebb082c693b3

commit edec2c60ec25753b355614237672ebb082c693b3
Author: Miklos Szeredi <mszeredi@suse.cz>
Date: Tue Oct 31 19:44:28 2017

UPSTREAM: vfs: rename: use common code for dir and non-dir

There's actually very little difference between vfs_rename_dir() and
vfs_rename_other() so move both inline into vfs_rename() which still stays
reasonably readable.

BUG= chromium:773474 
TEST=Check with cl:668107 and cl:668107 modified to tack dm-2 that the
ref count of the dm-2 superblock reach 0 and imageloader --unmount
succed.

Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Reviewed-by: J. Bruce Fields <bfields@redhat.com>
(cherry picked from commit bc27027a73e8b80376b51a1583ad1c7445605e8a)
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>

Change-Id: I6de49c2660becea5ffb80084a5e36b4cdb2154e6
Reviewed-on: https://chromium-review.googlesource.com/720088
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
(cherry picked from commit a8b68b19a09d47639efc142b180628a9e84463da)
Reviewed-on: https://chromium-review.googlesource.com/747209
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
Commit-Queue: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/edec2c60ec25753b355614237672ebb082c693b3/fs/namei.c

Project Member

Comment 51 by bugdroid1@chromium.org, Oct 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/86d7d32b00a44fa7a320eab47b0bec100696eb8d

commit 86d7d32b00a44fa7a320eab47b0bec100696eb8d
Author: Eric W. Biederman <ebiederm@xmission.com>
Date: Tue Oct 31 19:44:32 2017

UPSTREAM: vfs: Don't allow overwriting mounts in the current mount namespace

In preparation for allowing mountpoints to be renamed and unlinked
in remote filesystems and in other mount namespaces test if on a dentry
there is a mount in the local mount namespace before allowing it to
be renamed or unlinked.

The primary motivation here are old versions of fusermount unmount
which is not safe if the a path can be renamed or unlinked while it is
verifying the mount is safe to unmount.  More recent versions are simpler
and safer by simply using UMOUNT_NOFOLLOW when unmounting a mount
in a directory owned by an arbitrary user.

Miklos Szeredi <miklos@szeredi.hu> reports this is approach is good
enough to remove concerns about new kernels mixed with old versions
of fusermount.

A secondary motivation for restrictions here is that it removing empty
directories that have non-empty mount points on them appears to
violate the rule that rmdir can not remove empty directories.  As
Linus Torvalds pointed out this is useful for programs (like git) that
test if a directory is empty with rmdir.

Therefore this patch arranges to enforce the existing mount point
semantics for local mount namespace.

v2: Rewrote the test to be a drop in replacement for d_mountpoint
v3: Use bool instead of int as the return type of is_local_mountpoint

BUG= chromium:773474 
TEST=Check with cl:668107 and cl:668107 modified to tack dm-2 that the
ref count of the dm-2 superblock reach 0 and imageloader --unmount
succed.

Reviewed-by: Miklos Szeredi <miklos@szeredi.hu>
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 7af1364ffa64db61e386628594836e13d2ef04b5)
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>

Change-Id: Ic7b62575fbd2b4b3c880467755270fa5b742ebbd
Reviewed-on: https://chromium-review.googlesource.com/720093
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
(cherry picked from commit 31a42ccd79635e238353de3dc96630dac770025b)
Reviewed-on: https://chromium-review.googlesource.com/747210
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
Commit-Queue: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/86d7d32b00a44fa7a320eab47b0bec100696eb8d/fs/namespace.c
[modify] https://crrev.com/86d7d32b00a44fa7a320eab47b0bec100696eb8d/fs/namei.c
[modify] https://crrev.com/86d7d32b00a44fa7a320eab47b0bec100696eb8d/fs/mount.h

Project Member

Comment 52 by bugdroid1@chromium.org, Oct 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/8c201161c7624c77d512fd089e869b5c9389e088

commit 8c201161c7624c77d512fd089e869b5c9389e088
Author: Eric W. Biederman <ebiederman@twitter.com>
Date: Tue Oct 31 19:44:36 2017

UPSTREAM: vfs: Keep a list of mounts on a mount point

To spot any possible problems call BUG if a mountpoint
is put when it's list of mounts is not empty.

AV: use hlist instead of list_head

BUG= chromium:773474 
TEST=Check with cl:668107 and cl:668107 modified to tack dm-2 that the
ref count of the dm-2 superblock reach 0 and imageloader --unmount
succed.

Reviewed-by: Miklos Szeredi <miklos@szeredi.hu>
Signed-off-by: Eric W. Biederman <ebiederman@twitter.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 0a5eb7c8189922e86a840972cd0b57e41de6f031)
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>

Change-Id: I91eb75117436927f348340e180b711c702d4fae0
Reviewed-on: https://chromium-review.googlesource.com/720094
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
(cherry picked from commit 9433bbe23559c570f517b5caf6a7a87a9a0a8c2e)
Reviewed-on: https://chromium-review.googlesource.com/747211
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
Commit-Queue: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/8c201161c7624c77d512fd089e869b5c9389e088/fs/namespace.c
[modify] https://crrev.com/8c201161c7624c77d512fd089e869b5c9389e088/fs/mount.h

Project Member

Comment 53 by bugdroid1@chromium.org, Oct 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/998e0a9934cdcf5719fbabf07a32afa9c8bc97cb

commit 998e0a9934cdcf5719fbabf07a32afa9c8bc97cb
Author: Eric W. Biederman <ebiederm@xmission.com>
Date: Tue Oct 31 19:44:39 2017

UPSTREAM: vfs: factor out lookup_mountpoint from new_mountpoint

I am shortly going to add a new user of struct mountpoint that
needs to look up existing entries but does not want to create
a struct mountpoint if one does not exist.  Therefore to keep
the code simple and easy to read split out lookup_mountpoint
from new_mountpoint.

BUG= chromium:773474 
TEST=Check with cl:668107 and cl:668107 modified to tack dm-2 that the
ref count of the dm-2 superblock reach 0 and imageloader --unmount
succed.

Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit e2dfa935464272395b4f35f4cc74ffcc87418b84)
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>

Change-Id: I9350dbc9f15c3bb38258d7b895a0c193ba032c75
Reviewed-on: https://chromium-review.googlesource.com/720095
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
(cherry picked from commit 178bf8c5af985b77d7cbb1dee63badce23c2e0be)
Reviewed-on: https://chromium-review.googlesource.com/747212
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
Commit-Queue: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/998e0a9934cdcf5719fbabf07a32afa9c8bc97cb/fs/namespace.c

Project Member

Comment 54 by bugdroid1@chromium.org, Oct 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/6d5864f6a6602e0f5d45d53ae421a9efb1d13505

commit 6d5864f6a6602e0f5d45d53ae421a9efb1d13505
Author: Eric W. Biederman <ebiederman@twitter.com>
Date: Tue Oct 31 19:44:43 2017

UPSTREAM: vfs: Add a function to lazily unmount all mounts from any dentry.

The new function detach_mounts comes in two pieces.  The first piece
is a static inline test of d_mounpoint that returns immediately
without taking any locks if d_mounpoint is not set.  In the common
case when mountpoints are absent this allows the vfs to continue
running with it's same cacheline foot print.

The second piece of detach_mounts __detach_mounts actually does the
work and it assumes that a mountpoint is present so it is slow and
takes namespace_sem for write, and then locks the mount hash (aka
mount_lock) after a struct mountpoint has been found.

With those two locks held each entry on the list of mounts on a
mountpoint is selected and lazily unmounted until all of the mount
have been lazily unmounted.

v7: Wrote a proper change description and removed the changelog
    documenting deleted wrong turns.

BUG= chromium:773474 
TEST=Check with cl:668107 and cl:668107 modified to tack dm-2 that the
ref count of the dm-2 superblock reach 0 and imageloader --unmount
succed.

Signed-off-by: Eric W. Biederman <ebiederman@twitter.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 80b5dce8c59b0de1ed6e403b8298e02dcb4db64b)
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>

Change-Id: I4deedb60e528dc593d9c7552c53e846b23cdde93
Reviewed-on: https://chromium-review.googlesource.com/720196
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
(cherry picked from commit 4ad7ca317990027d4acde007822cce8d0032d9f1)
Reviewed-on: https://chromium-review.googlesource.com/747213
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
Commit-Queue: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/6d5864f6a6602e0f5d45d53ae421a9efb1d13505/fs/namespace.c
[modify] https://crrev.com/6d5864f6a6602e0f5d45d53ae421a9efb1d13505/fs/mount.h

Project Member

Comment 55 by bugdroid1@chromium.org, Oct 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/4760e957c0c14846ce258673ca08127b14626494

commit 4760e957c0c14846ce258673ca08127b14626494
Author: Eric W. Biederman <ebiederman@twitter.com>
Date: Tue Oct 31 19:44:47 2017

BACKPORT: vfs: Lazily remove mounts on unlinked files and directories.

With the introduction of mount namespaces and bind mounts it became
possible to access files and directories that on some paths are mount
points but are not mount points on other paths.  It is very confusing
when rm -rf somedir returns -EBUSY simply because somedir is mounted
somewhere else.  With the addition of user namespaces allowing
unprivileged mounts this condition has gone from annoying to allowing
a DOS attack on other users in the system.

The possibility for mischief is removed by updating the vfs to support
rename, unlink and rmdir on a dentry that is a mountpoint and by
lazily unmounting mountpoints on deleted dentries.

In particular this change allows rename, unlink and rmdir system calls
on a dentry without a mountpoint in the current mount namespace to
succeed, and it allows rename, unlink, and rmdir performed on a
distributed filesystem to update the vfs cache even if when there is a
mount in some namespace on the original dentry.

There are two common patterns of maintaining mounts: Mounts on trusted
paths with the parent directory of the mount point and all ancestory
directories up to / owned by root and modifiable only by root
(i.e. /media/xxx, /dev, /dev/pts, /proc, /sys, /sys/fs/cgroup/{cpu,
cpuacct, ...}, /usr, /usr/local).  Mounts on unprivileged directories
maintained by fusermount.

In the case of mounts in trusted directories owned by root and
modifiable only by root the current parent directory permissions are
sufficient to ensure a mount point on a trusted path is not removed
or renamed by anyone other than root, even if there is a context
where the there are no mount points to prevent this.

In the case of mounts in directories owned by less privileged users
races with users modifying the path of a mount point are already a
danger.  fusermount already uses a combination of chdir,
/proc/<pid>/fd/NNN, and UMOUNT_NOFOLLOW to prevent these races.  The
removable of global rename, unlink, and rmdir protection really adds
nothing new to consider only a widening of the attack window, and
fusermount is already safe against unprivileged users modifying the
directory simultaneously.

In principle for perfect userspace programs returning -EBUSY for
unlink, rmdir, and rename of dentires that have mounts in the local
namespace is actually unnecessary.  Unfortunately not all userspace
programs are perfect so retaining -EBUSY for unlink, rmdir and rename
of dentries that have mounts in the current mount namespace plays an
important role of maintaining consistency with historical behavior and
making imperfect userspace applications hard to exploit.

v2: Remove spurious old_dentry.
v3: Optimized shrink_submounts_and_drop
    Removed unsued afs label
v4: Simplified the changes to check_submounts_and_drop
    Do not rename check_submounts_and_drop shrink_submounts_and_drop
    Document what why we need atomicity in check_submounts_and_drop
    Rely on the parent inode mutex to make d_revalidate and d_invalidate
    an atomic unit.
v5: Refcount the mountpoint to detach in case of simultaneous
    renames.

BUG= chromium:773474 
TEST=Check with cl:668107 and cl:668107 modified to tack dm-2 that the
ref count of the dm-2 superblock reach 0 and imageloader --unmount
succed.

Reviewed-by: Miklos Szeredi <miklos@szeredi.hu>
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 8ed936b5671bfb33d89bc60bdcc7cf0470ba52fe)
 Conflicts:
      fs/dcache.c: Remove extra argument used by rename2.
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>

Change-Id: I5601cd9a372e83c0be7c8931bff7387202505eff
Reviewed-on: https://chromium-review.googlesource.com/720197
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
(cherry picked from commit bffe0a3fef327b97b59fda5adeade6d77ce6ce66)
Reviewed-on: https://chromium-review.googlesource.com/747214
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
Commit-Queue: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/4760e957c0c14846ce258673ca08127b14626494/fs/dcache.c
[modify] https://crrev.com/4760e957c0c14846ce258673ca08127b14626494/fs/namei.c

Project Member

Comment 56 by bugdroid1@chromium.org, Oct 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c38078fae79007a8798067bff1d115ef54094861

commit c38078fae79007a8798067bff1d115ef54094861
Author: Eric W. Biederman <ebiederm@xmission.com>
Date: Tue Oct 31 19:44:51 2017

UPSTREAM: vfs: Document the effect of d_revalidate on d_find_alias

d_drop or check_submounts_and_drop called from d_revalidate can result
in renamed directories with child dentries being unhashed.  These
renamed and drop directory dentries can be rehashed after
d_materialise_unique uses d_find_alias to find them.

BUG= chromium:773474 
TEST=Based on https://lkml.org/lkml/2014/2/25/110 list of email.
     on samus, no mount leak.

Reviewed-by: Miklos Szeredi <miklos@szeredi.hu>
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 3ccb354d641d910309b916b9c856e2a82ced7237)
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>

Change-Id: Idecdda13920e0810a11da3be6e3b360a4914f6b9
Reviewed-on: https://chromium-review.googlesource.com/724101
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
(cherry picked from commit e49f9bfab90d4fe7657a8c4f9a69219aae30d5cb)
Reviewed-on: https://chromium-review.googlesource.com/747215
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
Commit-Queue: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/c38078fae79007a8798067bff1d115ef54094861/fs/dcache.c

General update on this -- I have a series for 3.10 now that builds but my test machine went down.  I can likely do more testing on Friday.

I started looking at 3.8 based on the 3.10 series and it looks pretty close but will need a few additional patches I think.
Good news the 3.10 backport seems to work properly -- I'm working on cleaning up the series and will upload it tonight
Thanks, Sonny! Are we planning to backport this fix to 3.8 as well? 
yes am planning on 3.8
thanks! Is it practical to merge this fix to M62? If so, I will eventually abandon this: https://chromium-review.googlesource.com/c/chromium/src/+/745303

Otherwise, I will only block 3.8 on that CL and merge it to M62.
Series for 3.10 is up now -- end of series is here:
https://chromium-review.googlesource.com/#/c/chromiumos/third_party/kernel/+/756049

I had to revert three older security fixes to get things landed properly -- I fixed one of the security fixes but still need to add the other two, but this is good enough for people to test
re #62 -- I think it may be getting a little late for 62?  This series is my own backports rather than ones from stable, so I trust it a little less -- would want to see some bake time on a dev release
In that case, it looks like we still have to disable component updater for both 3.8&3.10 in M62. Can you please +1 on this CL, Sonny? https://chromium-review.googlesource.com/c/chromium/src/+/745303
re #64 -- I actually don't seem to have permission to +1 ? but it's already +1
Thanks, Sonny! I commented on the issue: https://chromium-review.googlesource.com/c/chromium/src/+/745303

Comment 67 by uekawa@google.com, Nov 8 2017

Blocking: 779567
Updated 3.10 series is here:
https://chromium-review.googlesource.com/#/c/chromiumos/third_party/kernel/+/758190

no regressions on xfstests -g auto 
security backports are back in place
I now have a working 3.8 prototype based on the 3.10 backport that I need to clean up and upload
3.8 series is up here -- I'm still running xfstests on 3.8

https://chromium-review.googlesource.com/#/c/chromiumos/third_party/kernel/+/762646

Comment 73 by ihf@chromium.org, Nov 13 2017

Sigh, not just poppy, all of them stopped. Orthogonal problem. I will take a look at it.
Thanks. Meanwhile I'm going to make the platform_CleanShutdown test more reliable by checking the ext4 superblock directly.

Comment 75 by ihf@chromium.org, Nov 14 2017

Cc: norvez@chromium.org
The failure was  issue 784687 . Should be reverted now and we should get data again on corruption.
I've finished testing on 3.8 -- everything is ready for review
Thanks Sonny! Looking at the autotest I still see some 3.14 and 3.10 boards having failures on platform_CleanShutdown. I'll monitor these boards over the week to see how consistent the problem is and if it keeps I'll get some hardware to test on:

Kernel 3.14
-------
Auron_paine
Gandof
Soraka

Kernel 3.10
----------
Nyan-big
Nyan-blaze
Nyan-kitty

Soraka is 4.4, not 3.14.
Thanks for the correction and that's interesting, because it's showing a decent rate of failure: https://wmatrix.googleplex.com/platform/unfiltered?hide_missing=True&tests=component_UpdateFlash&releases=tot&platforms=soraka
3.10 (and 3.8) is expected since we haven't landed the fixes yet -- I think Gwendal has just finished reviewing so hopefully should land soon

3.14 and 4.4 isn't expected -- I looked at the Soraka one but there isn't much information in there other than unmount didn't succeed and we went through recovery.  

I think it would make sense to collect more information about what happened during unmount when the test fails-- don't we collect some output from the attempt to unmount and stuff like lsof? 
Project Member

Comment 81 by bugdroid1@chromium.org, Nov 21 2017

Labels: merge-merged-chromeos-3.10
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/03879e71fc1b749df1ed9b16fef6c269ee5933d2

commit 03879e71fc1b749df1ed9b16fef6c269ee5933d2
Author: Sonny Rao <sonnyrao@chromium.org>
Date: Tue Nov 21 02:13:47 2017

Revert "UPSTREAM: dcache: Fix locking bugs in backported "deal with deadlock in d_walk()""

This reverts commit 703bbb1d753eaaafc4a26ae130ae82a369bf5a94.

BUG= chromium:773474 
TEST=build/boot on 3.10 device, xfstest

Change-Id: I4186876f0a78e8aa11a9b9597ad588eaa9f4038a
[SR: Reverting this to make later cherry-picks easier, will re-land]
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/756019
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/03879e71fc1b749df1ed9b16fef6c269ee5933d2/fs/dcache.c

Project Member

Comment 82 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/8057966ef08c9d0a370d52505777e95ef0e84a47

commit 8057966ef08c9d0a370d52505777e95ef0e84a47
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue Nov 21 04:30:22 2017

UPSTREAM: vfs: use lockred "dead" flag to mark unrecoverably dead dentries

This simplifies the RCU to refcounting code in particular.

I was originally intending to leave this for later, but walking through
all the dput() logic (see previous commit), I realized that the dput()
"might_sleep()" check was misleadingly weak.  And I removed it as
misleading, both for performance profiling and for debugging.

However, the might_sleep() debugging case is actually true: the final
dput() can indeed sleep, if the inode of the dentry that you are
releasing ends up sleeping at iput time (see dentry_iput()).  So the
problem with the might_sleep() in dput() wasn't that it wasn't true, it
was that it wasn't actually testing and triggering on the interesting
case.

In particular, just about *any* dput() can indeed sleep, if you happen
to race with another thread deleting the file in question, and you then
lose the race to the be the last dput() for that file.  But because it's
a very rare race, the debugging code would never trigger it in practice.

Why is this problematic? The new d_rcu_to_refcount() (see commit
15570086b590: "vfs: reimplement d_rcu_to_refcount() using
lockref_get_or_lock()") does a dput() for the failure case, and it does
it under the RCU lock.  So potentially sleeping really is a bug.

But there's no way I'm going to fix this with the previous complicated
"lockref_get_or_lock()" interface.  And rather than revert to the old
and crufty nested dentry locking code (which did get this right by
delaying the reference count updates until they were verified to be
safe), let's make forward progress.

BUG= chromium:773474 
TEST=build/boot on 3.10 device, xfstest

Change-Id: Iccf58ee49b9986509607f09ef6961efea4419a94
Cc: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry-picked from 0d98439ea3c6ffb2af931f6de4480e744634e2c5)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/756040
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/8057966ef08c9d0a370d52505777e95ef0e84a47/fs/dcache.c
[modify] https://crrev.com/8057966ef08c9d0a370d52505777e95ef0e84a47/fs/namei.c

Project Member

Comment 83 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/572222f3e4d007204d22b23b1c7016516131c90d

commit 572222f3e4d007204d22b23b1c7016516131c90d
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue Nov 21 04:30:23 2017

UPSTREAM: vfs: fix dentry RCU to refcounting possibly sleeping dput()

This is the fix that the last two commits indirectly led up to - making
sure that we don't call dput() in a bad context on the dentries we've
looked up in RCU mode after the sequence count validation fails.

This basically expands d_rcu_to_refcount() into the callers, and then
fixes the callers to delay the dput() in the failure case until _after_
we've dropped all locks and are no longer in an RCU-locked region.

The case of 'complete_walk()' was trivial, since its failure case did
the unlock_rcu_walk() directly after the call to d_rcu_to_refcount(),
and as such that is just a pure expansion of the function with a trivial
movement of the resulting dput() to after 'unlock_rcu_walk()'.

In contrast, the unlazy_walk() case was much more complicated, because
not only does convert two different dentries from RCU to be reference
counted, but it used to not call unlock_rcu_walk() at all, and instead
just returned an error and let the caller clean everything up in
"terminate_walk()".

Happily, one of the dentries in question (called "parent" inside
unlazy_walk()) is the dentry of "nd->path", which terminate_walk() wants
a refcount to anyway for the non-RCU case.

So what the new and improved unlazy_walk() does is to first turn that
dentry into a refcounted one, and once that is set up, the error cases
can continue to use the terminate_walk() helper for cleanup, but for the
non-RCU case.  Which makes it possible to drop out of RCU mode if we
actually hit the sequence number failure case.

BUG= chromium:773474 
TEST=build/boot on 3.10 device, xfstest

Change-Id: I97e78e6ce955caf9895065c5240e2cb8ae8665cf
Acked-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from stable-3.14 commit e5c832d5558826cc6e9a24746cfdec8e7780063a)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/756041
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/572222f3e4d007204d22b23b1c7016516131c90d/fs/namei.c

Project Member

Comment 84 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/9a9b012e9877db51e611b8a430e3e95d8d54f8a3

commit 9a9b012e9877db51e611b8a430e3e95d8d54f8a3
Author: Dave Chinner <dchinner@redhat.com>
Date: Tue Nov 21 04:30:24 2017

BACKPORT: dentry: move to per-sb LRU locks

With the dentry LRUs being per-sb structures, there is no real need for
a global dentry_lru_lock. The locking can be made more fine-grained by
moving to a per-sb LRU lock, isolating the LRU operations of different
filesytsems completely from each other. The need for this is independent
of any performance consideration that may arise: in the interest of
abstracting the lru operations away, it is mandatory that each lru works
around its own lock instead of a global lock for all of them.

BUG= chromium:773474 
TEST=build/boot on 3.10 device, xfstest

Change-Id: Ie47e7f2778cf7a777f00fd47c22583256d36cc9b
[glommer@openvz.org: updated changelog ]
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Glauber Costa <glommer@openvz.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Acked-by: Mel Gorman <mgorman@suse.de>
Cc: "Theodore Ts'o" <tytso@mit.edu>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Artem Bityutskiy <artem.bityutskiy@linux.intel.com>
Cc: Arve Hjnnevg <arve@android.com>
Cc: Carlos Maiolino <cmaiolino@redhat.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Chuck Lever <chuck.lever@oracle.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: David Rientjes <rientjes@google.com>
Cc: Gleb Natapov <gleb@redhat.com>
Cc: Greg Thelen <gthelen@google.com>
Cc: J. Bruce Fields <bfields@redhat.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Jerome Glisse <jglisse@redhat.com>
Cc: John Stultz <john.stultz@linaro.org>
Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: Kent Overstreet <koverstreet@google.com>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Steven Whitehouse <swhiteho@redhat.com>
Cc: Thomas Hellstrom <thellstrom@vmware.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 19156840e33a23eeb1a749c0f991dab6588b077d)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: minor context difference]
 Conflicts:
	fs/dcache.c
Reviewed-on: https://chromium-review.googlesource.com/756042
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/9a9b012e9877db51e611b8a430e3e95d8d54f8a3/fs/super.c
[modify] https://crrev.com/9a9b012e9877db51e611b8a430e3e95d8d54f8a3/include/linux/fs.h
[modify] https://crrev.com/9a9b012e9877db51e611b8a430e3e95d8d54f8a3/fs/dcache.c

Project Member

Comment 85 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/e978498f9e8b086989d52d4d4cb9dfd896f27181

commit e978498f9e8b086989d52d4d4cb9dfd896f27181
Author: Eric W. Biederman <ebiederman@twitter.com>
Date: Tue Nov 21 04:30:28 2017

BACKPORT: vfs: Keep a list of mounts on a mount point

To spot any possible problems call BUG if a mountpoint
is put when it's list of mounts is not empty.

AV: use hlist instead of list_head

BUG= chromium:773474 
TEST=Check with cl:668107 and cl:668107 modified to tack dm-2 that the
ref count of the dm-2 superblock reach 0 and imageloader --unmount
succed.

Change-Id: I91eb75117436927f348340e180b711c702d4fae0
Reviewed-by: Miklos Szeredi <miklos@szeredi.hu>
Signed-off-by: Eric W. Biederman <ebiederman@twitter.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 0a5eb7c8189922e86a840972cd0b57e41de6f031)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: conflict from not having 0818bf27c05 resizable namespace.c hashes
  and aba809cf0944 namespace.c: get rid of mnt_ghosts]
Conflicts:
	fs/namespace.c
Reviewed-on: https://chromium-review.googlesource.com/756043
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/e978498f9e8b086989d52d4d4cb9dfd896f27181/fs/namespace.c
[modify] https://crrev.com/e978498f9e8b086989d52d4d4cb9dfd896f27181/fs/mount.h

Project Member

Comment 86 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/6c7c908e4e2342ea785b295ee04362a3cf7e6260

commit 6c7c908e4e2342ea785b295ee04362a3cf7e6260
Author: Eric W. Biederman <ebiederm@xmission.com>
Date: Tue Nov 21 04:30:29 2017

BACKPORT: vfs: factor out lookup_mountpoint from new_mountpoint

I am shortly going to add a new user of struct mountpoint that
needs to look up existing entries but does not want to create
a struct mountpoint if one does not exist.  Therefore to keep
the code simple and easy to read split out lookup_mountpoint
from new_mountpoint.

BUG= chromium:773474 
TEST=boot 3.10 device and check for clean unmount of all filesystems

Change-Id: I7a5f9d13cb1d31265682d12549a395dca1f07fa0
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit e2dfa935464272395b4f35f4cc74ffcc87418b84)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: we don't have mp_hash() from
   0818bf27c05b2d resizable namespace.c hashes
   so go back to older way of accessing hash]
Reviewed-on: https://chromium-review.googlesource.com/756044
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/6c7c908e4e2342ea785b295ee04362a3cf7e6260/fs/namespace.c

Project Member

Comment 87 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/d91e510483ef26fb482ff806d0349230a03f8f29

commit d91e510483ef26fb482ff806d0349230a03f8f29
Author: Eric W. Biederman <ebiederman@twitter.com>
Date: Tue Nov 21 04:30:30 2017

BACKPORT: vfs: Add a function to lazily unmount all mounts from any dentry.

The new function detach_mounts comes in two pieces.  The first piece
is a static inline test of d_mounpoint that returns immediately
without taking any locks if d_mounpoint is not set.  In the common
case when mountpoints are absent this allows the vfs to continue
running with it's same cacheline foot print.

The second piece of detach_mounts __detach_mounts actually does the
work and it assumes that a mountpoint is present so it is slow and
takes namespace_sem for write, and then locks the mount hash (aka
mount_lock) after a struct mountpoint has been found.

With those two locks held each entry on the list of mounts on a
mountpoint is selected and lazily unmounted until all of the mount
have been lazily unmounted.

v7: Wrote a proper change description and removed the changelog
    documenting deleted wrong turns.

BUG= chromium:773474 
TEST=reboot 3.10 device and check for clean unmount of all filesystems

Change-Id: Ia92ae9dbfd6eb924e7b9990e673237a647787793
Signed-off-by: Eric W. Biederman <ebiederman@twitter.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 80b5dce8c59b0de1ed6e403b8298e02dcb4db64b)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: swich from lock_mount_hash() helpers back to old br_lock_write]
Reviewed-on: https://chromium-review.googlesource.com/756045
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/d91e510483ef26fb482ff806d0349230a03f8f29/fs/namespace.c
[modify] https://crrev.com/d91e510483ef26fb482ff806d0349230a03f8f29/fs/mount.h

Project Member

Comment 88 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/84b651b6425ca42c0d8a37aeeea522606280a278

commit 84b651b6425ca42c0d8a37aeeea522606280a278
Author: Eric W. Biederman <ebiederm@xmission.com>
Date: Tue Nov 21 04:30:31 2017

BACKPORT: vfs: Don't allow overwriting mounts in the current mount namespace

In preparation for allowing mountpoints to be renamed and unlinked
in remote filesystems and in other mount namespaces test if on a dentry
there is a mount in the local mount namespace before allowing it to
be renamed or unlinked.

The primary motivation here are old versions of fusermount unmount
which is not safe if the a path can be renamed or unlinked while it is
verifying the mount is safe to unmount.  More recent versions are simpler
and safer by simply using UMOUNT_NOFOLLOW when unmounting a mount
in a directory owned by an arbitrary user.

Miklos Szeredi <miklos@szeredi.hu> reports this is approach is good
enough to remove concerns about new kernels mixed with old versions
of fusermount.

A secondary motivation for restrictions here is that it removing empty
directories that have non-empty mount points on them appears to
violate the rule that rmdir can not remove empty directories.  As
Linus Torvalds pointed out this is useful for programs (like git) that
test if a directory is empty with rmdir.

Therefore this patch arranges to enforce the existing mount point
semantics for local mount namespace.

v2: Rewrote the test to be a drop in replacement for d_mountpoint
v3: Use bool instead of int as the return type of is_local_mountpoint

BUG= chromium:773474 
TEST=Check with cl:668107 and cl:668107 modified to tack dm-2 that the
ref count of the dm-2 superblock reach 0 and imageloader --unmount
succed.

Change-Id: Ic7b62575fbd2b4b3c880467755270fa5b742ebbc
Reviewed-by: Miklos Szeredi <miklos@szeredi.hu>
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 7af1364ffa64db61e386628594836e13d2ef04b5)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: context conflicts]
 Conflicts:
	fs/namei.c
	fs/namespace.c
Reviewed-on: https://chromium-review.googlesource.com/756046
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/84b651b6425ca42c0d8a37aeeea522606280a278/fs/namespace.c
[modify] https://crrev.com/84b651b6425ca42c0d8a37aeeea522606280a278/fs/namei.c
[modify] https://crrev.com/84b651b6425ca42c0d8a37aeeea522606280a278/fs/mount.h

Project Member

Comment 89 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/a6f6cf3bde574fec77df97dbe70c624e8a00fa4d

commit a6f6cf3bde574fec77df97dbe70c624e8a00fa4d
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 04:30:33 2017

BACKPORT: switch shrink_dcache_for_umount() to use of d_walk()

we have too many iterators in fs/dcache.c...

BUG= chromium:773474 
TEST=build/boot on 3.10 device, xfstest

Change-Id: Ie800c11bb5989c7addfd5f260a1aa0ff56da9324
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 42c326082d8a2c91506f951ace638deae1faf083)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: context conflicts in two deleted functions
  shrink_dcache_for_subtree() and shrink_dcache_for_umount()
  also don't have d_lru_del() and d_shrink_add() so need equivalent
  code]
 Conflicts:
	fs/dcache.c
Reviewed-on: https://chromium-review.googlesource.com/756047
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/a6f6cf3bde574fec77df97dbe70c624e8a00fa4d/fs/dcache.c

Project Member

Comment 90 by bugdroid1@chromium.org, Nov 21 2017

Labels: merge-merged-chromeos-3.8
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/9485790096670f6db2d229c38c371c197df42886

commit 9485790096670f6db2d229c38c371c197df42886
Author: Sonny Rao <sonnyrao@chromium.org>
Date: Tue Nov 21 07:24:43 2017

Revert "UPSTREAM: dcache: Fix locking bugs in backported "deal with deadlock in d_walk()""

This reverts commit 373cac694676c784d78a15e20470629c39f3540a.

Reverting this so that we can more easily port fixes that landed on
later kernels.  I will re-land this or equivalent after that.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I16e17a50b0a5c4accbd8d37d4d60f53b173b31ad
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762599
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/9485790096670f6db2d229c38c371c197df42886/fs/dcache.c

Project Member

Comment 91 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/b8633413758b72660160ede035c5a6c1c449fac4

commit b8633413758b72660160ede035c5a6c1c449fac4
Author: Sonny Rao <sonnyrao@chromium.org>
Date: Tue Nov 21 07:24:45 2017

Revert "UPSTREAM: deal with deadlock in d_walk()"

This reverts commit 1da97e9ee0d087a4e4485a5b8ca87d2c38750976.

Reverting this so that we can more easily port fixes that landed on
later kernels.  I will re-land this or equivalent after that.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I3d0d865932da4134452af48b68ad8b23bc1f0352
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762600
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/b8633413758b72660160ede035c5a6c1c449fac4/fs/dcache.c

Project Member

Comment 92 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/18853960df60ea8ead16e3199cd3d382e5e39829

commit 18853960df60ea8ead16e3199cd3d382e5e39829
Author: Sonny Rao <sonnyrao@chromium.org>
Date: Tue Nov 21 07:24:46 2017

Revert "BACKPORT: move d_rcu from overlapping d_child to overlapping d_alias"

This reverts commit 9c03ae5fcad2081cc8ba40629ed9165cb52479c9.

Reverting this so that we can more easily port fixes that landed on
later kernels.  I will re-land this or equivalent after that.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I3fe36bfd36d86a5f7d833010969d8d3b0d53c7c4
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762601
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/notify/fsnotify.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/ncpfs/dir.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/affs/amigaffs.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/cifs/inode.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/dcache.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/nfs/getroot.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/ceph/inode.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/security/selinux/selinuxfs.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/exportfs/expfs.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/autofs4/root.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/kernel/cgroup.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/arch/powerpc/platforms/cell/spufs/inode.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/debugfs/inode.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/ceph/dir.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/coda/cache.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/ncpfs/ncplib_kernel.h
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/libfs.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/include/linux/dcache.h
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/ocfs2/dcache.c
[modify] https://crrev.com/18853960df60ea8ead16e3199cd3d382e5e39829/fs/autofs4/expire.c

Project Member

Comment 93 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/bba314852f88424a0e9e77ee8ca307a5ab8ff919

commit bba314852f88424a0e9e77ee8ca307a5ab8ff919
Author: Waiman Long <Waiman.Long@hp.com>
Date: Tue Nov 21 07:24:47 2017

UPSTREAM: Add new lockref infrastructure reference implementation

This introduces a new "lockref" structure that supports the concept of
lockless updates of reference counts that still honor an attached
spinlock.

NOTE! This reference implementation is not the optimized lockless
version, rather it is the fallback implementation using standard
spinlocks.  The actual optimized versions will be merged into 3.12, but
I wanted to get the infrastructure in place and document the new
interfaces.

[ Also note that this particular commit is drastically cut-down minimal
  version of the original patch by Waiman.  In order to properly credit
  the original author I'm marking Waiman as the author here, but in the
  end this patch bears little resemblance to the patch by Waiman.  So
  blame any errors on me editing things down to the point where I can
  introduce the infrastructure before the merge window for 3.12 actually
  opens.     - Linus ]

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I9ede360bd421c92853604bbdf137e7279f23fe75
Signed-off-by: Waiman Long <Waiman.Long@hp.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit 0f8f2aaaab0b0f9c13635cb02e7d19bdaa9aa1bb)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762602
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[add] https://crrev.com/bba314852f88424a0e9e77ee8ca307a5ab8ff919/include/linux/lockref.h

Project Member

Comment 94 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/af846beda2dce2194b8470d201b0342c71da0d3e

commit af846beda2dce2194b8470d201b0342c71da0d3e
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue Nov 21 07:24:48 2017

UPSTREAM: lockref: add 'lockref_get_or_lock() helper

This behaves like "lockref_get_not_zero()", but instead of doing nothing
if the count was zero, it returns with the lock held.

This allows callers to revalidate the lockref-protected data structure
if required even if the count was zero to begin with, and possibly
increment the count if it passes muster.

In particular, the dentry code wants this when it wants to turn an
RCU-protected dentry into a stable refcounted one: if the dentry count
it zero, but the sequence number still validates the dentry, we can take
a reference to it.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: If1878b1c505bf7c8f142b39ca8e6d642e30f7f8c
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit b3abd80250c13414bc258b53e57242feb159af91)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762603
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/af846beda2dce2194b8470d201b0342c71da0d3e/include/linux/lockref.h

Project Member

Comment 95 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/57f5b87b53beaaffeb61c6e94df09187f25acc86

commit 57f5b87b53beaaffeb61c6e94df09187f25acc86
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue Nov 21 07:24:49 2017

UPSTREAM: lockref: uninline lockref helper functions

They aren't very good to inline, since they already call external
functions (the spinlock code), and we're going to create rather more
complicated versions of them that can do the reference count updates
locklessly.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Ia3996db3a08f216a285753a534d27c3b68e5aa03
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry-picked-from 2f4f12e571c4e2f50f3818a3c2544929145f75dd)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762604
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/57f5b87b53beaaffeb61c6e94df09187f25acc86/lib/Makefile
[modify] https://crrev.com/57f5b87b53beaaffeb61c6e94df09187f25acc86/include/linux/lockref.h
[add] https://crrev.com/57f5b87b53beaaffeb61c6e94df09187f25acc86/lib/lockref.c

Project Member

Comment 96 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/96414e78f3a73725807f299c5c43da6aebab285a

commit 96414e78f3a73725807f299c5c43da6aebab285a
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue Nov 21 07:24:50 2017

UPSTREAM: lockref: implement lockless reference count updates using cmpxchg()

Instead of taking the spinlock, the lockless versions atomically check
that the lock is not taken, and do the reference count update using a
cmpxchg() loop.  This is semantically identical to doing the reference
count update protected by the lock, but avoids the "wait for lock"
contention that you get when accesses to the reference count are
contended.

Note that a "lockref" is absolutely _not_ equivalent to an atomic_t.
Even when the lockref reference counts are updated atomically with
cmpxchg, the fact that they also verify the state of the spinlock means
that the lockless updates can never happen while somebody else holds the
spinlock.

So while "lockref_put_or_lock()" looks a lot like just another name for
"atomic_dec_and_lock()", and both optimize to lockless updates, they are
fundamentally different: the decrement done by atomic_dec_and_lock() is
truly independent of any lock (as long as it doesn't decrement to zero),
so a locked region can still see the count change.

The lockref structure, in contrast, really is a *locked* reference
count.  If you hold the spinlock, the reference count will be stable and
you can modify the reference count without using atomics, because even
the lockless updates will see and respect the state of the lock.

In order to enable the cmpxchg lockless code, the architecture needs to
do three things:

 (1) Make sure that the "arch_spinlock_t" and an "unsigned int" can fit
     in an aligned u64, and have a "cmpxchg()" implementation that works
     on such a u64 data type.

 (2) define a helper function to test for a spinlock being unlocked
     ("arch_spin_value_unlocked()")

 (3) select the "ARCH_USE_CMPXCHG_LOCKREF" config variable in its
     Kconfig file.

This enables it for x86-64 (but not 32-bit, we'd need to make sure
cmpxchg() turns into the proper cmpxchg8b in order to enable it for
32-bit mode).

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I3fe97af934d1488bcf38bfeb9f340d9ba5ca469f
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry-picked from bc08b449ee14ace4d869adaa1bb35a44ce68d775)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762605
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/96414e78f3a73725807f299c5c43da6aebab285a/arch/x86/include/asm/spinlock.h
[modify] https://crrev.com/96414e78f3a73725807f299c5c43da6aebab285a/lib/Kconfig
[modify] https://crrev.com/96414e78f3a73725807f299c5c43da6aebab285a/arch/x86/Kconfig
[modify] https://crrev.com/96414e78f3a73725807f299c5c43da6aebab285a/lib/lockref.c
[modify] https://crrev.com/96414e78f3a73725807f299c5c43da6aebab285a/include/linux/lockref.h

Project Member

Comment 97 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c27da8850efb418935cd1e81f4ce6298b357b607

commit c27da8850efb418935cd1e81f4ce6298b357b607
Author: Luck, Tony <tony.luck@intel.com>
Date: Tue Nov 21 07:24:52 2017

UPSTREAM: lockref: Relax in cmpxchg loop

While we are likley to succeed and break out of this loop, it isn't
guaranteed.  We should be power and thread friendly if we do have to
go around for a second (or third, or more) attempt.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I8e84823548f3f955d5807b2965d3fb68b4ab00c2
Signed-off-by: Tony Luck <tony.luck@intel.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry-picked from d472d9d98b463dd7a04f2bcdeafe4261686ce6ab)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762606
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/c27da8850efb418935cd1e81f4ce6298b357b607/lib/lockref.c

Project Member

Comment 98 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/820d80973d918a9b17e6fe8040ea8f96bd4c2174

commit 820d80973d918a9b17e6fe8040ea8f96bd4c2174
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue Nov 21 07:24:53 2017

UPSTREAM: lockref: fix docbook argument names

The code got rewritten, but the comments got copied as-is from older
versions, and as a result the argument name in the comment didn't
actually match the code any more.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Ica3315cbdb93d124d2c797b0f26a6633c01febc3
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry-picked from 44a0cf92926c343366a4986808d12ab068504eed)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762607
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/820d80973d918a9b17e6fe8040ea8f96bd4c2174/lib/lockref.c

Project Member

Comment 99 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/9e2e424e5dc7e0d5cf7639abad34677809ec804d

commit 9e2e424e5dc7e0d5cf7639abad34677809ec804d
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue Nov 21 07:24:54 2017

UPSTREAM: lockref: add ability to mark lockrefs "dead"

The only actual current lockref user (dcache) uses zero reference counts
even for perfectly live dentries, because it's a cache: there may not be
any users, but that doesn't mean that we want to throw away the dentry.

At the same time, the dentry cache does have a notion of a truly "dead"
dentry that we must not even increment the reference count of, because
we have pruned it and it is not valid.

Currently that distinction is not visible in the lockref itself, and the
dentry cache validation uses "lockref_get_or_lock()" to either get a new
reference to a dentry that already had existing references (and thus
cannot be dead), or get the dentry lock so that we can then verify the
dentry and increment the reference count under the lock if that
verification was successful.

That's all somewhat complicated.

This adds the concept of being "dead" to the lockref itself, by simply
using a count that is negative.  This allows a usage scenario where we
can increment the refcount of a dentry without having to validate it,
and pushing the special "we killed it" case into the lockref code.

The dentry code itself doesn't actually use this yet, and it's probably
too late in the merge window to do that code (the dentry_kill() code
with its "should I decrement the count" logic really is pretty complex
code), but let's introduce the concept at the lockref level now.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I326fdf85792bc94d9dad5b06a74677a415e00241
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry-picked from e7d33bb5ea82922e6ddcfc6b28a630b1a4ced071)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762608
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/9e2e424e5dc7e0d5cf7639abad34677809ec804d/include/linux/lockref.h
[modify] https://crrev.com/9e2e424e5dc7e0d5cf7639abad34677809ec804d/lib/lockref.c

Project Member

Comment 100 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/4c3ebc45ecfb5231dcfb458e42632d0c5ef7ce88

commit 4c3ebc45ecfb5231dcfb458e42632d0c5ef7ce88
Author: Jeff Layton <jlayton@redhat.com>
Date: Tue Nov 21 07:24:55 2017

UPSTREAM: vfs: kill FS_REVAL_DOT by adding a d_weak_revalidate dentry op

The following set of operations on a NFS client and server will cause

    server# mkdir a
    client# cd a
    server# mv a a.bak
    client# sleep 30  # (or whatever the dir attrcache timeout is)
    client# stat .
    stat: cannot stat `.': Stale NFS file handle

Obviously, we should not be getting an ESTALE error back there since the
inode still exists on the server. The problem is that the lookup code
will call d_revalidate on the dentry that "." refers to, because NFS has
FS_REVAL_DOT set.

nfs_lookup_revalidate will see that the parent directory has changed and
will try to reverify the dentry by redoing a LOOKUP. That of course
fails, so the lookup code returns ESTALE.

The problem here is that d_revalidate is really a bad fit for this case.
What we really want to know at this point is whether the inode is still
good or not, but we don't really care what name it goes by or whether
the dcache is still valid.

Add a new d_op->d_weak_revalidate operation and have complete_walk call
that instead of d_revalidate. The intent there is to allow for a
"weaker" d_revalidate that just checks to see whether the inode is still
good. This is also gives us an opportunity to kill off the FS_REVAL_DOT
special casing.

[AV: changed method name, added note in porting, fixed confusion re
having it possibly called from RCU mode (it won't be)]

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I71c8abf04df11d3c72c4608f268b40df1c397ae1
Cc: NeilBrown <neilb@suse.de>
Signed-off-by: Jeff Layton <jlayton@redhat.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit ecf3d1f1aa74da0d632b651a2e05a911f60e92c0)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762609
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/4c3ebc45ecfb5231dcfb458e42632d0c5ef7ce88/fs/9p/vfs_super.c
[modify] https://crrev.com/4c3ebc45ecfb5231dcfb458e42632d0c5ef7ce88/fs/nfs/dir.c
[modify] https://crrev.com/4c3ebc45ecfb5231dcfb458e42632d0c5ef7ce88/fs/dcache.c
[modify] https://crrev.com/4c3ebc45ecfb5231dcfb458e42632d0c5ef7ce88/Documentation/filesystems/Locking
[modify] https://crrev.com/4c3ebc45ecfb5231dcfb458e42632d0c5ef7ce88/fs/namei.c
[modify] https://crrev.com/4c3ebc45ecfb5231dcfb458e42632d0c5ef7ce88/fs/nfs/super.c
[modify] https://crrev.com/4c3ebc45ecfb5231dcfb458e42632d0c5ef7ce88/Documentation/filesystems/vfs.txt
[modify] https://crrev.com/4c3ebc45ecfb5231dcfb458e42632d0c5ef7ce88/include/linux/fs.h
[modify] https://crrev.com/4c3ebc45ecfb5231dcfb458e42632d0c5ef7ce88/fs/9p/vfs_dentry.c
[modify] https://crrev.com/4c3ebc45ecfb5231dcfb458e42632d0c5ef7ce88/Documentation/filesystems/porting
[modify] https://crrev.com/4c3ebc45ecfb5231dcfb458e42632d0c5ef7ce88/include/linux/dcache.h
[modify] https://crrev.com/4c3ebc45ecfb5231dcfb458e42632d0c5ef7ce88/fs/nfs/nfs4super.c

Project Member

Comment 101 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/f293fc1b0fca4c510a1322d28de0fd4bf0616614

commit f293fc1b0fca4c510a1322d28de0fd4bf0616614
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue Nov 21 07:24:56 2017

BACKPORT: Don't pass inode to ->d_hash() and ->d_compare()

Instances either don't look at it at all (the majority of cases) or
only want it to find the superblock (which can be had as dentry->d_sb).
A few cases that want more are actually safe with dentry->d_inode -
the only precaution needed is the check that it hadn't been replaced with
NULL by rmdir() or by overwriting rename(), which case should be simply
treated as cache miss.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I53547056aa601febc130f1e07a7f693136eb902d
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit da53be12bbb4fabbe2e9f6f908de0cf478b5161d)
[SR: adjust context for 3.8, do not include changes for absent FS: efivarfs]
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762610
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/affs/namei.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/include/linux/dcache.h
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/fat/namei_vfat.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/namei.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/isofs/namei.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/Documentation/filesystems/vfs.txt
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/gfs2/dentry.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/hfsplus/hfsplus_fs.h
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/fat/namei_msdos.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/cifs/dir.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/ncpfs/dir.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/proc/proc_sysctl.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/Documentation/filesystems/Locking
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/hfs/string.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/hfs/hfs_fs.h
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/isofs/inode.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/sysv/namei.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/hpfs/dentry.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/dcache.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/hfsplus/unicode.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/adfs/dir.c
[modify] https://crrev.com/f293fc1b0fca4c510a1322d28de0fd4bf0616614/fs/jfs/namei.c

Project Member

Comment 102 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/6943048daff19c659b6b611d3fd36b0f32f8f232

commit 6943048daff19c659b6b611d3fd36b0f32f8f232
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:24:58 2017

BACKPORT: helper for reading ->d_count

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I2c889aa3d0be55cd8f1f05feb001ad78cfd336ab
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 84d08fa888e7c2d53b5bbc764db2ef02968b499c)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: don't have lustre]
 Conflicts:
	drivers/staging/lustre/lustre/include/linux/lustre_patchless_compat.h
Reviewed-on: https://chromium-review.googlesource.com/762611
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/6943048daff19c659b6b611d3fd36b0f32f8f232/fs/ecryptfs/inode.c
[modify] https://crrev.com/6943048daff19c659b6b611d3fd36b0f32f8f232/fs/nfs/dir.c
[modify] https://crrev.com/6943048daff19c659b6b611d3fd36b0f32f8f232/include/linux/dcache.h
[modify] https://crrev.com/6943048daff19c659b6b611d3fd36b0f32f8f232/fs/ceph/inode.c
[modify] https://crrev.com/6943048daff19c659b6b611d3fd36b0f32f8f232/fs/coda/dir.c
[modify] https://crrev.com/6943048daff19c659b6b611d3fd36b0f32f8f232/fs/autofs4/root.c
[modify] https://crrev.com/6943048daff19c659b6b611d3fd36b0f32f8f232/fs/nfs/unlink.c
[modify] https://crrev.com/6943048daff19c659b6b611d3fd36b0f32f8f232/fs/nilfs2/super.c
[modify] https://crrev.com/6943048daff19c659b6b611d3fd36b0f32f8f232/fs/locks.c
[modify] https://crrev.com/6943048daff19c659b6b611d3fd36b0f32f8f232/fs/autofs4/expire.c
[modify] https://crrev.com/6943048daff19c659b6b611d3fd36b0f32f8f232/fs/ceph/mds_client.c

Project Member

Comment 103 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/9c6387f369cae8e3e651f77b0c289d925670fa7c

commit 9c6387f369cae8e3e651f77b0c289d925670fa7c
Author: Miklos Szeredi <mszeredi@suse.cz>
Date: Tue Nov 21 07:25:04 2017

UPSTREAM: vfs: restructure d_genocide()

It shouldn't matter when we decrement the refcount during the walk as long
as we do it exactly once.

Restructure d_genocide() to do the killing on entering the dentry instead
of when leaving it.  This helps creating a common helper for tree walking.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I72886cfc174254588d96bc7b701f259d7bfd1382
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 01ddc4ede5f09cdaed015d49ab66eec179085a2e)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762616
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/9c6387f369cae8e3e651f77b0c289d925670fa7c/fs/dcache.c

Project Member

Comment 104 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/d7096015684f7b7a7496452a1648e5b00b594b66

commit d7096015684f7b7a7496452a1648e5b00b594b66
Author: Greg Thelen <gthelen@google.com>
Date: Tue Nov 21 07:25:05 2017

UPSTREAM: fs/dcache.c: add cond_resched() to shrink_dcache_parent()

Call cond_resched() in shrink_dcache_parent() to maintain interactivity.

Before this patch:

	void shrink_dcache_parent(struct dentry * parent)
	{
		while ((found = select_parent(parent, &dispose)) != 0)
			shrink_dentry_list(&dispose);
	}

select_parent() populates the dispose list with dentries which
shrink_dentry_list() then deletes.  select_parent() carefully uses
need_resched() to avoid doing too much work at once.  But neither
shrink_dcache_parent() nor its called functions call cond_resched().  So
once need_resched() is set select_parent() will return single dentry
dispose list which is then deleted by shrink_dentry_list().  This is
inefficient when there are a lot of dentry to process.  This can cause
softlockup and hurts interactivity on non preemptable kernels.

This change adds cond_resched() in shrink_dcache_parent().  The benefit
of this is that need_resched() is quickly cleared so that future calls
to select_parent() are able to efficiently return a big batch of dentry.

These additional cond_resched() do not seem to impact performance, at
least for the workload below.

Here is a program which can cause soft lockup if other system activity
sets need_resched().

	int main()
	{
	        struct rlimit rlim;
	        int i;
	        int f[100000];
	        char buf[20];
	        struct timeval t1, t2;
	        double diff;

	        /* cleanup past run */
	        system("rm -rf x");

	        /* boost nfile rlimit */
	        rlim.rlim_cur = 200000;
	        rlim.rlim_max = 200000;
	        if (setrlimit(RLIMIT_NOFILE, &rlim))
	                err(1, "setrlimit");

	        /* make directory for files */
	        if (mkdir("x", 0700))
	                err(1, "mkdir");

	        if (gettimeofday(&t1, NULL))
	                err(1, "gettimeofday");

	        /* populate directory with open files */
	        for (i = 0; i < 100000; i++) {
	                snprintf(buf, sizeof(buf), "x/%d", i);
	                f[i] = open(buf, O_CREAT);
	                if (f[i] == -1)
	                        err(1, "open");
	        }

	        /* close some of the files */
	        for (i = 0; i < 85000; i++)
	                close(f[i]);

	        /* unlink all files, even open ones */
	        system("rm -rf x");

	        if (gettimeofday(&t2, NULL))
	                err(1, "gettimeofday");

	        diff = (((double)t2.tv_sec * 1000000 + t2.tv_usec) -
	                ((double)t1.tv_sec * 1000000 + t1.tv_usec));

	        printf("done: %g elapsed\n", diff/1e6);
	        return 0;
	}

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Id1b0fb453d2d207d4753d26e81fa25aa20aa6e90
Signed-off-by: Greg Thelen <gthelen@google.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
Cc: <stable@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit 421348f1ca0bf17769dee0aed4d991845ae0536d)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762617
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/d7096015684f7b7a7496452a1648e5b00b594b66/fs/dcache.c

Project Member

Comment 105 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/0e342976ea74162c9dd9f8a723a6a1f1ed0c2756

commit 0e342976ea74162c9dd9f8a723a6a1f1ed0c2756
Author: Yan, Zheng <zheng.z.yan@intel.com>
Date: Tue Nov 21 07:25:06 2017

UPSTREAM: fs: remove dentry_lru_prune()

When pruning a dentry, its ancestor dentry can also be pruned. But
the ancestor dentry does not go through dput(), so it does not get
put on the dentry LRU. Hence associating d_prune with removing the
dentry from the LRU is the wrong.

The fix is remove dentry_lru_prune(). Call file system's d_prune()
callback directly when pruning dentries.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: If7d438e0ce95ab7d37f01534e9419bc73053a048
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from 61572bb1f40b9bec0acbb4d7bc0f5b33739f1ab1)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762618
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/0e342976ea74162c9dd9f8a723a6a1f1ed0c2756/fs/dcache.c

Project Member

Comment 106 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/f8cc712857d3efe116e648f953cf46b447a3a3a4

commit f8cc712857d3efe116e648f953cf46b447a3a3a4
Author: Miklos Szeredi <mszeredi@suse.cz>
Date: Tue Nov 21 07:25:07 2017

UPSTREAM: vfs: add d_walk()

This one replaces three instances open coded tree walking (have_submounts,
select_parent, d_genocide) with a common helper.

In addition to slightly reducing the kernel size, this simplifies the
callers and makes them less bug prone.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Ieab762b80fccab820078b2c8a00c85b0ab3e4352
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit db14fc3abcd5dcc9b32ad5b9dd5b8f9e16295a39)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762619
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/f8cc712857d3efe116e648f953cf46b447a3a3a4/fs/dcache.c

Project Member

Comment 107 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ddba1ebcba0a5eca38e8bb65503a8abe03422749

commit ddba1ebcba0a5eca38e8bb65503a8abe03422749
Author: Miklos Szeredi <mszeredi@suse.cz>
Date: Tue Nov 21 07:25:08 2017

UPSTREAM: vfs: check submounts and drop atomically

We check submounts before doing d_drop() on a non-empty directory dentry in
NFS (have_submounts()), but we do not exclude a racing mount.

 Process A: have_submounts() -> returns false
 Process B: mount() -> success
 Process A: d_drop()

This patch prepares the ground for the fix by doing the following
operations all under the same rename lock:

  have_submounts()
  shrink_dcache_parent()
  d_drop()

This is actually an optimization since have_submounts() and
shrink_dcache_parent() both traverse the same dentry tree separately.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I78d1a32a382bb478e9971fa16556276dd12dd751
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
CC: David Howells <dhowells@redhat.com>
CC: Steven Whitehouse <swhiteho@redhat.com>
CC: Trond Myklebust <Trond.Myklebust@netapp.com>
CC: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from 848ac114e847af3f1f9141c90a39ebe79bdb13b3)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762620
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/ddba1ebcba0a5eca38e8bb65503a8abe03422749/include/linux/dcache.h
[modify] https://crrev.com/ddba1ebcba0a5eca38e8bb65503a8abe03422749/fs/dcache.c

Project Member

Comment 108 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/36f542237283e7ad1d2984ab3e57a51d474a9b5e

commit 36f542237283e7ad1d2984ab3e57a51d474a9b5e
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:25:10 2017

UPSTREAM: mount: consolidate permission checks

... and ask for global CAP_SYS_ADMIN only for superblock-level remounts

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I0fae014891c7920d757caf49e5ac22245daf9a54
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 57eccb830f1cc93d4b506ba306d8dfa685e0c88f)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762621
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/36f542237283e7ad1d2984ab3e57a51d474a9b5e/fs/namespace.c

Project Member

Comment 109 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/0afbf5f4be65121337a7f17adf923a4938da0934

commit 0afbf5f4be65121337a7f17adf923a4938da0934
Author: Andrey Vagin <avagin@openvz.org>
Date: Tue Nov 21 07:25:11 2017

UPSTREAM: mnt: release locks on error path in do_loopback

do_loopback calls lock_mount(path) and forget to unlock_mount
if clone_mnt or copy_mnt fails.

[   77.661566] ================================================
[   77.662939] [ BUG: lock held when returning to user space! ]
[   77.664104] 3.9.0-rc5+ #17 Not tainted
[   77.664982] ------------------------------------------------
[   77.666488] mount/514 is leaving the kernel with locks still held!
[   77.668027] 2 locks held by mount/514:
[   77.668817]  #0:  (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff811cca22>] lock_mount+0x32/0xe0
[   77.671755]  #1:  (&namespace_sem){+++++.}, at: [<ffffffff811cca3a>] lock_mount+0x4a/0xe0

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Ie4eafb29f366f0ae7e2be8c749a27e0e8645db99
Signed-off-by: Andrey Vagin <avagin@openvz.org>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit e9c5d8a562f01b211926d70443378eb14b29a676)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762622
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/0afbf5f4be65121337a7f17adf923a4938da0934/fs/namespace.c

Project Member

Comment 110 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/8658d71e96528bcc3373ee4bbfdcadd4fd68025d

commit 8658d71e96528bcc3373ee4bbfdcadd4fd68025d
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:25:12 2017

BACKPORT: get rid of full-hash scan on detaching vfsmounts

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Iccc10a5d434e014061ca7d51225fa06ddc584bd6
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 84d17192d2afd52aeba88c71ae4959a015f56a38)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: conflict because of security fix ce15323d8e3df]
 Conflicts:
	fs/namespace.c
Reviewed-on: https://chromium-review.googlesource.com/762623
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/8658d71e96528bcc3373ee4bbfdcadd4fd68025d/fs/pnode.h
[modify] https://crrev.com/8658d71e96528bcc3373ee4bbfdcadd4fd68025d/fs/namespace.c
[modify] https://crrev.com/8658d71e96528bcc3373ee4bbfdcadd4fd68025d/fs/pnode.c
[modify] https://crrev.com/8658d71e96528bcc3373ee4bbfdcadd4fd68025d/fs/mount.h

Project Member

Comment 111 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/b6e447bc53cbd7a175297b0c7022b6849715981b

commit b6e447bc53cbd7a175297b0c7022b6849715981b
Author: Miklos Szeredi <miklos@szeredi.hu>
Date: Tue Nov 21 07:25:13 2017

BACKPORT: vfs: check unlinked ancestors before mount

We check submounts before doing d_drop() on a non-empty directory dentry in
NFS (have_submounts()), but we do not exclude a racing mount.  Nor do we
prevent mounts to be added to the disconnected subtree using relative paths
after the d_drop().

This patch fixes these issues by checking for unlinked (unhashed, non-root)
ancestors before proceeding with the mount.  This is done with rename
seqlock taken for write and with ->d_lock grabbed on each ancestor in turn,
including our dentry itself.  This ensures that the only one of
check_submounts_and_drop() or has_unlinked_ancestor() can succeed.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: If931d3280fd6ecaf3452ebf1816acfc7a7cdd08b
Signed-off-by: Miklos Szeredi <miklos@szeredi.hu>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry-picked from eed810076685c77dc9a8c5c3593e641c93caed1c)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: context fix]
 Conflicts:
	fs/internal.h
Reviewed-on: https://chromium-review.googlesource.com/762624
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/b6e447bc53cbd7a175297b0c7022b6849715981b/fs/internal.h
[modify] https://crrev.com/b6e447bc53cbd7a175297b0c7022b6849715981b/fs/namespace.c
[modify] https://crrev.com/b6e447bc53cbd7a175297b0c7022b6849715981b/fs/dcache.c

Project Member

Comment 112 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/a2a7750955068cf71f23ee2170de4a5c0f77915b

commit a2a7750955068cf71f23ee2170de4a5c0f77915b
Author: Waiman Long <Waiman.Long@hp.com>
Date: Tue Nov 21 07:25:14 2017

BACKPORT: dcache: Translating dentry into pathname without taking rename_lock

When running the AIM7's short workload, Linus' lockref patch eliminated
most of the spinlock contention. However, there were still some left:

     8.46%     reaim  [kernel.kallsyms]     [k] _raw_spin_lock
                 |--42.21%-- d_path
                 |          proc_pid_readlink
                 |          SyS_readlinkat
                 |          SyS_readlink
                 |          system_call
                 |          __GI___readlink
                 |
                 |--40.97%-- sys_getcwd
                 |          system_call
                 |          __getcwd

The big one here is the rename_lock (seqlock) contention in d_path()
and the getcwd system call. This patch will eliminate the need to take
the rename_lock while translating dentries into the full pathnames.

The need to take the rename_lock is to make sure that no rename
operation can be ongoing while the translation is in progress. However,
only one thread can take the rename_lock thus blocking all the other
threads that need it even though the translation process won't make
any change to the dentries.

This patch will replace the writer's write_seqlock/write_sequnlock
sequence of the rename_lock of the callers of the prepend_path() and
__dentry_path() functions with the reader's read_seqbegin/read_seqretry
sequence within these 2 functions. As a result, the code will have to
retry if one or more rename operations had been performed. In addition,
RCU read lock will be taken during the translation process to make sure
that no dentries will go away. To prevent live-lock from happening,
the code will switch back to take the rename_lock if read_seqretry()
fails for three times.

To further reduce spinlock contention, this patch does not take the
dentry's d_lock when copying the filename from the dentries. Instead,
it treats the name pointer and length as unreliable and just copy
the string byte-by-byte over until it hits a null byte or the end of
string as specified by the length. This should avoid stepping into
invalid memory address. The error cases are left to be handled by
the sequence number check.

The following code re-factoring are also made:
1. Move prepend('/') into prepend_name() to remove one conditional
   check.
2. Move the global root check in prepend_path() back to the top of
   the while loop.

With this patch, the _raw_spin_lock will now account for only 1.2%
of the total CPU cycles for the short workload. This patch also has
the effect of reducing the effect of running perf on its profile
since the perf command itself can be a heavy user of the d_path()
function depending on the complexity of the workload.

When taking the perf profile of the high-systime workload, the amount
of spinlock contention contributed by running perf without this patch
was about 16%. With this patch, the spinlock contention caused by
the running of perf will go away and we will have a more accurate
perf profile.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I0a223cb9d79497efd2d3d81053719ce773dcd644
Signed-off-by: Waiman Long <Waiman.Long@hp.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry-picked from 232d2d60aa5469bb097f55728f65146bd49c1d25)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: 3.8 doesn't have simple_dname()]
  Conflicts:
	fs/dcache.c
Reviewed-on: https://chromium-review.googlesource.com/762625
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/a2a7750955068cf71f23ee2170de4a5c0f77915b/fs/dcache.c

Project Member

Comment 113 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/29a1a977d0900ecf41a62b99747bfd076f818b39

commit 29a1a977d0900ecf41a62b99747bfd076f818b39
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:25:15 2017

UPSTREAM: split read_seqretry_or_unlock(), convert d_walk() to resulting primitives

Separate "check if we need to retry" from "unlock if we are done and
had seq_writelock"; that allows to use these guys in d_walk(), where
we need to recheck every time we ascend back to parent, but do *not*
want to unlock until the very end.  Lift rcu_read_lock/rcu_read_unlock
out into callers.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I4c5e91c1a8f0053742bcee8cf8702b57e5c79c2f
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry-picked from 48f5ec21d9c67e881ff35343988e290ef5cf933f)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762626
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/29a1a977d0900ecf41a62b99747bfd076f818b39/fs/dcache.c

Project Member

Comment 114 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/d4eee86e5759499d5d73348e8e14281b14df290c

commit d4eee86e5759499d5d73348e8e14281b14df290c
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue Nov 21 07:25:17 2017

UPSTREAM: vfs: reorganize dput() memory accesses

This is me being a bit OCD after all the dentry optimization work this
merge window: profiles end up showing 'dput()' as a rather expensive
operation, and there were two unrelated bad reasons for that.

The first reason was reading d_lockref.count for debugging purposes,
which touches the lockref cacheline (for reads) before really need to.
More importantly, the debugging test in question is _wrong_, and has
hidden bugs.  It's true that we can only sleep when the count goes down
to zero, but the test as-is hides the much more subtle bug that happens
if we race with somebody else deleting the file.

Anyway we _will_ touch that cacheline, but let's do it for a write and
in the right routine (ie in "lockref_put_or_lock()") which annotates the
costs better.  So remove the misleading debug code.

The other was an unnecessary access to the cacheline that contains the
d_lru list, just to check whether we already were on the LRU list or
not.  This is exactly what we have d_flags for, so that we can avoid
touching extra cache lines for the common case.  So just add another bit
for "is this dentry on the LRU".

Finally, mark the tests properly likely/unlikely, so that the common
fast-paths are dense in the instruction stream.

This makes the profiles look much saner.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Idafc3e14b2c00ca4dd82e54344aeec2087ed93dd
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry-picked from 8aab6a27332bbf2abfcb35224738394e784d940b)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762627
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/d4eee86e5759499d5d73348e8e14281b14df290c/include/linux/dcache.h
[modify] https://crrev.com/d4eee86e5759499d5d73348e8e14281b14df290c/fs/dcache.c

Project Member

Comment 115 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/63b1b2d462d8e7f6fa0f1169f86cf5fd747f7f0a

commit 63b1b2d462d8e7f6fa0f1169f86cf5fd747f7f0a
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue Nov 21 07:25:18 2017

UPSTREAM: vfs: use lockred "dead" flag to mark unrecoverably dead dentries

This simplifies the RCU to refcounting code in particular.

I was originally intending to leave this for later, but walking through
all the dput() logic (see previous commit), I realized that the dput()
"might_sleep()" check was misleadingly weak.  And I removed it as
misleading, both for performance profiling and for debugging.

However, the might_sleep() debugging case is actually true: the final
dput() can indeed sleep, if the inode of the dentry that you are
releasing ends up sleeping at iput time (see dentry_iput()).  So the
problem with the might_sleep() in dput() wasn't that it wasn't true, it
was that it wasn't actually testing and triggering on the interesting
case.

In particular, just about *any* dput() can indeed sleep, if you happen
to race with another thread deleting the file in question, and you then
lose the race to the be the last dput() for that file.  But because it's
a very rare race, the debugging code would never trigger it in practice.

Why is this problematic? The new d_rcu_to_refcount() (see commit
15570086b590: "vfs: reimplement d_rcu_to_refcount() using
lockref_get_or_lock()") does a dput() for the failure case, and it does
it under the RCU lock.  So potentially sleeping really is a bug.

But there's no way I'm going to fix this with the previous complicated
"lockref_get_or_lock()" interface.  And rather than revert to the old
and crufty nested dentry locking code (which did get this right by
delaying the reference count updates until they were verified to be
safe), let's make forward progress.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Iccf58ee49b9986509607f09ef6961efea4419a94
Cc: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry-picked from 0d98439ea3c6ffb2af931f6de4480e744634e2c5)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762628
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/63b1b2d462d8e7f6fa0f1169f86cf5fd747f7f0a/fs/dcache.c
[modify] https://crrev.com/63b1b2d462d8e7f6fa0f1169f86cf5fd747f7f0a/fs/namei.c

Project Member

Comment 116 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/49aff5f4d470b483d947def3ab97a2252c04a62b

commit 49aff5f4d470b483d947def3ab97a2252c04a62b
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue Nov 21 07:25:19 2017

UPSTREAM: vfs: fix dentry RCU to refcounting possibly sleeping dput()

This is the fix that the last two commits indirectly led up to - making
sure that we don't call dput() in a bad context on the dentries we've
looked up in RCU mode after the sequence count validation fails.

This basically expands d_rcu_to_refcount() into the callers, and then
fixes the callers to delay the dput() in the failure case until _after_
we've dropped all locks and are no longer in an RCU-locked region.

The case of 'complete_walk()' was trivial, since its failure case did
the unlock_rcu_walk() directly after the call to d_rcu_to_refcount(),
and as such that is just a pure expansion of the function with a trivial
movement of the resulting dput() to after 'unlock_rcu_walk()'.

In contrast, the unlazy_walk() case was much more complicated, because
not only does convert two different dentries from RCU to be reference
counted, but it used to not call unlock_rcu_walk() at all, and instead
just returned an error and let the caller clean everything up in
"terminate_walk()".

Happily, one of the dentries in question (called "parent" inside
unlazy_walk()) is the dentry of "nd->path", which terminate_walk() wants
a refcount to anyway for the non-RCU case.

So what the new and improved unlazy_walk() does is to first turn that
dentry into a refcounted one, and once that is set up, the error cases
can continue to use the terminate_walk() helper for cleanup, but for the
non-RCU case.  Which makes it possible to drop out of RCU mode if we
actually hit the sequence number failure case.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I97e78e6ce955caf9895065c5240e2cb8ae8665cf
Acked-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry-picked-from e5c832d5558826cc6e9a24746cfdec8e7780063a)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762629
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/49aff5f4d470b483d947def3ab97a2252c04a62b/fs/namei.c

Project Member

Comment 117 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/5da4cdf6971db0fedc7a6f81f1ac77b0eb14fecb

commit 5da4cdf6971db0fedc7a6f81f1ac77b0eb14fecb
Author: Dave Chinner <dchinner@redhat.com>
Date: Tue Nov 21 07:25:20 2017

BACKPORT: dentry: move to per-sb LRU locks

With the dentry LRUs being per-sb structures, there is no real need for
a global dentry_lru_lock. The locking can be made more fine-grained by
moving to a per-sb LRU lock, isolating the LRU operations of different
filesytsems completely from each other. The need for this is independent
of any performance consideration that may arise: in the interest of
abstracting the lru operations away, it is mandatory that each lru works
around its own lock instead of a global lock for all of them.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Ie47e7f2778cf7a777f00fd47c22583256d36cc9b
[glommer@openvz.org: updated changelog ]
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Glauber Costa <glommer@openvz.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Acked-by: Mel Gorman <mgorman@suse.de>
Cc: "Theodore Ts'o" <tytso@mit.edu>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Artem Bityutskiy <artem.bityutskiy@linux.intel.com>
Cc: Arve Hjnnevg <arve@android.com>
Cc: Carlos Maiolino <cmaiolino@redhat.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Chuck Lever <chuck.lever@oracle.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: David Rientjes <rientjes@google.com>
Cc: Gleb Natapov <gleb@redhat.com>
Cc: Greg Thelen <gthelen@google.com>
Cc: J. Bruce Fields <bfields@redhat.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Jerome Glisse <jglisse@redhat.com>
Cc: John Stultz <john.stultz@linaro.org>
Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: Kent Overstreet <koverstreet@google.com>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Steven Whitehouse <swhiteho@redhat.com>
Cc: Thomas Hellstrom <thellstrom@vmware.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 19156840e33a23eeb1a749c0f991dab6588b077d)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: minor context difference]
 Conflicts:
	fs/dcache.c
Reviewed-on: https://chromium-review.googlesource.com/762630
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/5da4cdf6971db0fedc7a6f81f1ac77b0eb14fecb/fs/super.c
[modify] https://crrev.com/5da4cdf6971db0fedc7a6f81f1ac77b0eb14fecb/include/linux/fs.h
[modify] https://crrev.com/5da4cdf6971db0fedc7a6f81f1ac77b0eb14fecb/fs/dcache.c

Project Member

Comment 118 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/5fd69433815ec21795e77a6e2078a8212c046e87

commit 5fd69433815ec21795e77a6e2078a8212c046e87
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:25:21 2017

UPSTREAM: prepend_path() needs to reinitialize dentry/vfsmount/mnt on restarts

commit ede4cebce16f5643c61aedd6d88d9070a1d23a68 upstream.

... and equivalent is needed in 3.12; it's broken there as well

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I7a2fa7b35fe794585515d139fcfad1ad57172099
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
Reported-by: Michael Marineau <michael.marineau@coreos.com>
Tested-by: Waiman Long <Waiman.Long@hp.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from stable-3.12 commit 9e5129ba8d3d65605683050f73f12dddbf3f8c00)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762631
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/5fd69433815ec21795e77a6e2078a8212c046e87/fs/dcache.c

Project Member

Comment 119 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/dda4b643bd741e9772ca5ec0b0be93cc25a5eab2

commit dda4b643bd741e9772ca5ec0b0be93cc25a5eab2
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:25:23 2017

UPSTREAM: __dentry_path() fixes

commit f6500801522c61782d4990fa1ad96154cb397cd4 upstream.

* we need to save the starting point for restarts
* reject pathologically short buffers outright

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I6d873a0e2d98db1532d28d4702d6ea6ee9d13857
Spotted-by: Denys Vlasenko <dvlasenk@redhat.com>
Spotted-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Jiri Slaby <jslaby@suse.cz>
(cherry picked from stable-3.12 commit aeaf64c5a760c88e7af7b316536980d41bf312cf)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762632
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/dda4b643bd741e9772ca5ec0b0be93cc25a5eab2/fs/dcache.c

Project Member

Comment 120 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/d8f810739462681751a7cd605ae931bfcb83cf07

commit d8f810739462681751a7cd605ae931bfcb83cf07
Author: Eric W. Biederman <ebiederman@twitter.com>
Date: Tue Nov 21 07:25:24 2017

BACKPORT: vfs: Keep a list of mounts on a mount point

To spot any possible problems call BUG if a mountpoint
is put when it's list of mounts is not empty.

AV: use hlist instead of list_head

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I91eb75117436927f348340e180b711c702d4fae0
Reviewed-by: Miklos Szeredi <miklos@szeredi.hu>
Signed-off-by: Eric W. Biederman <ebiederman@twitter.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 0a5eb7c8189922e86a840972cd0b57e41de6f031)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: conflict from not having 0818bf27c05 resizable namespace.c hashes
  and aba809cf0944 namespace.c: get rid of mnt_ghosts]
Conflicts:
	fs/namespace.c
Reviewed-on: https://chromium-review.googlesource.com/762633
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/d8f810739462681751a7cd605ae931bfcb83cf07/fs/namespace.c
[modify] https://crrev.com/d8f810739462681751a7cd605ae931bfcb83cf07/fs/mount.h

Project Member

Comment 121 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/6af229957ec7fbed1841eb4989c7f9e8e9f67692

commit 6af229957ec7fbed1841eb4989c7f9e8e9f67692
Author: Eric W. Biederman <ebiederm@xmission.com>
Date: Tue Nov 21 07:25:25 2017

BACKPORT: vfs: factor out lookup_mountpoint from new_mountpoint

I am shortly going to add a new user of struct mountpoint that
needs to look up existing entries but does not want to create
a struct mountpoint if one does not exist.  Therefore to keep
the code simple and easy to read split out lookup_mountpoint
from new_mountpoint.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I7a5f9d13cb1d31265682d12549a395dca1f07fa0
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit e2dfa935464272395b4f35f4cc74ffcc87418b84)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: we don't have mp_hash() from
   0818bf27c05b2d resizable namespace.c hashes
   so go back to older way of accessing hash]
Reviewed-on: https://chromium-review.googlesource.com/762634
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/6af229957ec7fbed1841eb4989c7f9e8e9f67692/fs/namespace.c

Project Member

Comment 122 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/1a2a3feed4f87b9ca49da1fb62fb14b98306df37

commit 1a2a3feed4f87b9ca49da1fb62fb14b98306df37
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:25:26 2017

UPSTREAM: saner umount_tree()/release_mounts(), part 1

global list of release_mounts() fodder, protected by namespace_sem;
eventually, all umount_tree() callers will use it as kill list.
Helper picking the contents of that list, releasing namespace_sem
and doing release_mounts() on what it got.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Icdf966b7a7ffe1cc69ac16b2f41dc3e599fe354e
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit e3197d83d6f5b9bd0e57a05592437ffa459ee106)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762635
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/1a2a3feed4f87b9ca49da1fb62fb14b98306df37/fs/namespace.c

Project Member

Comment 123 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/689ee66d9a86bac2ac9801c33c727ffe6cdcfa4d

commit 689ee66d9a86bac2ac9801c33c727ffe6cdcfa4d
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:25:27 2017

UPSTREAM: get rid of the second argument of shrink_submounts()

... it's always &unmounted.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Iaebba61f1ac633c86271d55b9332c7271c09b4f3
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit b54b9be7824d84158cd90305820e2c3914f74ad9)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762636
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/689ee66d9a86bac2ac9801c33c727ffe6cdcfa4d/fs/namespace.c

Project Member

Comment 124 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/25dfcbc91b17b8b5f184b256ae3239209737a28d

commit 25dfcbc91b17b8b5f184b256ae3239209737a28d
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:25:29 2017

UPSTREAM: more conversions to namespace_unlock()

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Id4df8163a2e4b5d11114e73e86144d2fa0cf604d
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 3ab6abee59ac9ca84cc4a1e31224f1dccd44394c)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762637
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/25dfcbc91b17b8b5f184b256ae3239209737a28d/fs/namespace.c

Project Member

Comment 125 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/b0dcabc84e3a6f94b2dd9bdb3fccbe21f3420a38

commit b0dcabc84e3a6f94b2dd9bdb3fccbe21f3420a38
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:25:30 2017

UPSTREAM: switch unlock_mount() to namespace_unlock(), convert all umount_tree() callers

which allows to kill the last argument of umount_tree() and make release_mounts()
static.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Ic866b654e9ddd0761edfdf53bf941e6defecda50
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 328e6d9014636afc2b3c979403b36faadb412657)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762638
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/b0dcabc84e3a6f94b2dd9bdb3fccbe21f3420a38/fs/pnode.h
[modify] https://crrev.com/b0dcabc84e3a6f94b2dd9bdb3fccbe21f3420a38/fs/namespace.c
[modify] https://crrev.com/b0dcabc84e3a6f94b2dd9bdb3fccbe21f3420a38/fs/pnode.c

Project Member

Comment 126 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/01003a54c521614ffe740f07b7dac2797b91e537

commit 01003a54c521614ffe740f07b7dac2797b91e537
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:25:31 2017

UPSTREAM: fold release_mounts() into namespace_unlock()

... and provide namespace_lock() as a trivial wrapper;
switch to those two consistently.

Result is patterned after rtnl_lock/rtnl_unlock pair.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Ia0a1949db4244cf696fab1cbe677ff790c1d722c
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 97216be09efd41414725068212e3af0f05cde11a)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762639
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/01003a54c521614ffe740f07b7dac2797b91e537/fs/namespace.c

Project Member

Comment 127 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/531c09b0e0339745602613605344cf5bbe28664d

commit 531c09b0e0339745602613605344cf5bbe28664d
Author: Eric W. Biederman <ebiederman@twitter.com>
Date: Tue Nov 21 07:25:32 2017

BACKPORT: vfs: Add a function to lazily unmount all mounts from any dentry.

The new function detach_mounts comes in two pieces.  The first piece
is a static inline test of d_mounpoint that returns immediately
without taking any locks if d_mounpoint is not set.  In the common
case when mountpoints are absent this allows the vfs to continue
running with it's same cacheline foot print.

The second piece of detach_mounts __detach_mounts actually does the
work and it assumes that a mountpoint is present so it is slow and
takes namespace_sem for write, and then locks the mount hash (aka
mount_lock) after a struct mountpoint has been found.

With those two locks held each entry on the list of mounts on a
mountpoint is selected and lazily unmounted until all of the mount
have been lazily unmounted.

v7: Wrote a proper change description and removed the changelog
    documenting deleted wrong turns.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Ia92ae9dbfd6eb924e7b9990e673237a647787793
Signed-off-by: Eric W. Biederman <ebiederman@twitter.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 80b5dce8c59b0de1ed6e403b8298e02dcb4db64b)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: swich from lock_mount_hash() helpers back to old br_lock_write]
Reviewed-on: https://chromium-review.googlesource.com/762640
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/531c09b0e0339745602613605344cf5bbe28664d/fs/namespace.c
[modify] https://crrev.com/531c09b0e0339745602613605344cf5bbe28664d/fs/mount.h

Project Member

Comment 128 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/b014b61a28b987f382f6fd60d7dae361a02a8041

commit b014b61a28b987f382f6fd60d7dae361a02a8041
Author: Eric W. Biederman <ebiederm@xmission.com>
Date: Tue Nov 21 07:25:34 2017

BACKPORT: vfs: Don't allow overwriting mounts in the current mount namespace

In preparation for allowing mountpoints to be renamed and unlinked
in remote filesystems and in other mount namespaces test if on a dentry
there is a mount in the local mount namespace before allowing it to
be renamed or unlinked.

The primary motivation here are old versions of fusermount unmount
which is not safe if the a path can be renamed or unlinked while it is
verifying the mount is safe to unmount.  More recent versions are simpler
and safer by simply using UMOUNT_NOFOLLOW when unmounting a mount
in a directory owned by an arbitrary user.

Miklos Szeredi <miklos@szeredi.hu> reports this is approach is good
enough to remove concerns about new kernels mixed with old versions
of fusermount.

A secondary motivation for restrictions here is that it removing empty
directories that have non-empty mount points on them appears to
violate the rule that rmdir can not remove empty directories.  As
Linus Torvalds pointed out this is useful for programs (like git) that
test if a directory is empty with rmdir.

Therefore this patch arranges to enforce the existing mount point
semantics for local mount namespace.

v2: Rewrote the test to be a drop in replacement for d_mountpoint
v3: Use bool instead of int as the return type of is_local_mountpoint

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Ic7b62575fbd2b4b3c880467755270fa5b742ebbc
Reviewed-by: Miklos Szeredi <miklos@szeredi.hu>
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 7af1364ffa64db61e386628594836e13d2ef04b5)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: context conflicts]
 Conflicts:
	fs/namei.c
	fs/namespace.c
Reviewed-on: https://chromium-review.googlesource.com/762641
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/b014b61a28b987f382f6fd60d7dae361a02a8041/fs/namespace.c
[modify] https://crrev.com/b014b61a28b987f382f6fd60d7dae361a02a8041/fs/namei.c
[modify] https://crrev.com/b014b61a28b987f382f6fd60d7dae361a02a8041/fs/mount.h

Project Member

Comment 129 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/e77ae2f847f76a255f5f3f7522f80a22f0aa42b8

commit e77ae2f847f76a255f5f3f7522f80a22f0aa42b8
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:25:35 2017

BACKPORT: switch shrink_dcache_for_umount() to use of d_walk()

we have too many iterators in fs/dcache.c...

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Ie800c11bb5989c7addfd5f260a1aa0ff56da9324
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 42c326082d8a2c91506f951ace638deae1faf083)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: context conflicts in two deleted functions
  shrink_dcache_for_subtree() and shrink_dcache_for_umount()
  also don't have d_lru_del() and d_shrink_add() so need equivalent
  code]
 Conflicts:
	fs/dcache.c
Reviewed-on: https://chromium-review.googlesource.com/762642
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/e77ae2f847f76a255f5f3f7522f80a22f0aa42b8/fs/dcache.c

Project Member

Comment 130 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/4c4c04552f26c8bad82807b83f8ecd291cc8e380

commit 4c4c04552f26c8bad82807b83f8ecd291cc8e380
Author: Eric W. Biederman <ebiederman@twitter.com>
Date: Tue Nov 21 07:25:36 2017

BACKPORT: vfs: Lazily remove mounts on unlinked files and directories.

With the introduction of mount namespaces and bind mounts it became
possible to access files and directories that on some paths are mount
points but are not mount points on other paths.  It is very confusing
when rm -rf somedir returns -EBUSY simply because somedir is mounted
somewhere else.  With the addition of user namespaces allowing
unprivileged mounts this condition has gone from annoying to allowing
a DOS attack on other users in the system.

The possibility for mischief is removed by updating the vfs to support
rename, unlink and rmdir on a dentry that is a mountpoint and by
lazily unmounting mountpoints on deleted dentries.

In particular this change allows rename, unlink and rmdir system calls
on a dentry without a mountpoint in the current mount namespace to
succeed, and it allows rename, unlink, and rmdir performed on a
distributed filesystem to update the vfs cache even if when there is a
mount in some namespace on the original dentry.

There are two common patterns of maintaining mounts: Mounts on trusted
paths with the parent directory of the mount point and all ancestory
directories up to / owned by root and modifiable only by root
(i.e. /media/xxx, /dev, /dev/pts, /proc, /sys, /sys/fs/cgroup/{cpu,
cpuacct, ...}, /usr, /usr/local).  Mounts on unprivileged directories
maintained by fusermount.

In the case of mounts in trusted directories owned by root and
modifiable only by root the current parent directory permissions are
sufficient to ensure a mount point on a trusted path is not removed
or renamed by anyone other than root, even if there is a context
where the there are no mount points to prevent this.

In the case of mounts in directories owned by less privileged users
races with users modifying the path of a mount point are already a
danger.  fusermount already uses a combination of chdir,
/proc/<pid>/fd/NNN, and UMOUNT_NOFOLLOW to prevent these races.  The
removable of global rename, unlink, and rmdir protection really adds
nothing new to consider only a widening of the attack window, and
fusermount is already safe against unprivileged users modifying the
directory simultaneously.

In principle for perfect userspace programs returning -EBUSY for
unlink, rmdir, and rename of dentires that have mounts in the local
namespace is actually unnecessary.  Unfortunately not all userspace
programs are perfect so retaining -EBUSY for unlink, rmdir and rename
of dentries that have mounts in the current mount namespace plays an
important role of maintaining consistency with historical behavior and
making imperfect userspace applications hard to exploit.

v2: Remove spurious old_dentry.
v3: Optimized shrink_submounts_and_drop
    Removed unsued afs label
v4: Simplified the changes to check_submounts_and_drop
    Do not rename check_submounts_and_drop shrink_submounts_and_drop
    Document what why we need atomicity in check_submounts_and_drop
    Rely on the parent inode mutex to make d_revalidate and d_invalidate
    an atomic unit.
v5: Refcount the mountpoint to detach in case of simultaneous
    renames.

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Ice8da7c924c989d219001a44e6acb4aa001b4d33
Reviewed-by: Miklos Szeredi <miklos@szeredi.hu>
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 8ed936b5671bfb33d89bc60bdcc7cf0470ba52fe)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: remove extra argument to __d_move() in dcache and context conflict
  in namei.c for changes to mutex lock]
 Conflicts:
	fs/dcache.c
	fs/namei.c
Reviewed-on: https://chromium-review.googlesource.com/762643
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/4c4c04552f26c8bad82807b83f8ecd291cc8e380/fs/dcache.c
[modify] https://crrev.com/4c4c04552f26c8bad82807b83f8ecd291cc8e380/fs/namei.c

Project Member

Comment 131 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/99627e7f19d2727f768e58bb345a00a59b8df417

commit 99627e7f19d2727f768e58bb345a00a59b8df417
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:25:38 2017

BACKPORT: move d_rcu from overlapping d_child to overlapping d_alias

commit 946e51f2bf37f1656916eb75bd0742ba33983c28 upstream.

Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Ben Hutchings <ben@decadent.org.uk>

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Idc8a85c34556803320e66a92e3a0d3b72ce2e30d
Signed-off-by: Kevin Cernekee <cernekee@chromium.org>
[cernekee: Backported to 3.8 using 026181647a6262 from 3.2.y as a reference.
 - Apply name changes in all the different places we use d_alias and d_child
 - Move the WARN_ON() in __d_free() to d_free() as we don't have dentry_free()]
Reviewed-on: https://chromium-review.googlesource.com/264863
Reviewed-by: Kees Cook <keescook@chromium.org>
(cherry picked from 9c03ae5fcad2081cc8ba40629ed9165cb52479c9)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: re-land this and fix up conflicts after umount series fixes applied]
 Conflicts:
	fs/dcache.c
Reviewed-on: https://chromium-review.googlesource.com/762644
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/notify/fsnotify.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/ncpfs/dir.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/affs/amigaffs.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/cifs/inode.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/dcache.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/nfs/getroot.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/ceph/inode.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/security/selinux/selinuxfs.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/exportfs/expfs.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/autofs4/root.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/kernel/cgroup.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/arch/powerpc/platforms/cell/spufs/inode.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/debugfs/inode.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/ceph/dir.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/coda/cache.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/ncpfs/ncplib_kernel.h
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/libfs.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/include/linux/dcache.h
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/ocfs2/dcache.c
[modify] https://crrev.com/99627e7f19d2727f768e58bb345a00a59b8df417/fs/autofs4/expire.c

Project Member

Comment 132 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/0e7f1a6b7d752aedd6ad6753b4b0d02457fe28b7

commit 0e7f1a6b7d752aedd6ad6753b4b0d02457fe28b7
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:25:39 2017

BACKPORT: fold try_to_ascend() into the sole remaining caller

commit 31dec1327e377b6d91a8a6c92b5cd8513939a233 upstream.

There used to be a bunch of tree-walkers in dcache.c, all alike.
try_to_ascend() had been introduced to abstract a piece of logics
duplicated in all of them.  These days all these tree-walkers are
implemented via the same iterator (d_walk()), which is the only
remaining caller of try_to_ascend(), so let's fold it back...

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: Iaff67d1bb2d954388cf6b1851f86400a5f60cbb4
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Jiri Slaby <jslaby@suse.cz>
(cherry picked from stable-3.12 commit 316845a0eb891d1705917d3139a1005910563e5b)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
[SR: need to use child->d_child instead of child->d_u.d_child]
Conflicts:
	fs/dcache.c
Reviewed-on: https://chromium-review.googlesource.com/762645
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/0e7f1a6b7d752aedd6ad6753b4b0d02457fe28b7/fs/dcache.c

Project Member

Comment 133 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/bf4ac7070b4ae6409b6bd80c8ea4cb8943fd33c2

commit bf4ac7070b4ae6409b6bd80c8ea4cb8943fd33c2
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 21 07:25:40 2017

UPSTREAM: deal with deadlock in d_walk()

commit ca5358ef75fc69fee5322a38a340f5739d997c10 upstream.

... by not hitting rename_retry for reasons other than rename having
happened.  In other words, do _not_ restart when finding that
between unlocking the child and locking the parent the former got
into __dentry_kill().  Skip the killed siblings instead...

BUG= chromium:773474 
TEST=build/boot on link, run xfsFilesystemTestSuiteExt4Plain

Change-Id: I57c3695ad95d366b0d198b979c455a449753ae47
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
Cc: Ben Hutchings <ben@decadent.org.uk>
[hujianyang: Backported to 3.14 refer to the work of Ben Hutchings in 3.2:
 - Adjust context to make __dentry_kill() apply to d_kill()]
Signed-off-by: hujianyang <hujianyang@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from stable-3.14 commit 9cf2961ba6adaa14e06e1aa48fc65dadea49f9e2)
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/762646
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/bf4ac7070b4ae6409b6bd80c8ea4cb8943fd33c2/fs/dcache.c

> I think it would make sense to collect more information about what happened during unmount when the test fails-- don't we collect some output from the attempt to unmount and stuff like lsof? 

The tricky part there is that, at least with the bug addressed by these patches, lsof and the mount table aren't helpful because it's the underlying mapper device that leaked. The unmount of the Flash disk image succeeded. 

I think we should give all the patches some 'bake time' over the Holiday break, and next week I'll look at where all the test results stand. If we still see unexpected failures, I'll add whatever logging you think will help Sonny (although I think recording all the mapper devices is most useful - to know if the device still leaked), and I'll get some test hardware as well.
re #134 -- the issue is that we don't really know why we're seeing unclean shutdown on kernels that are already supposed to be fixed like 4.4 -- so lsof and other stuff would be useful in narrowing it down.
Status: Fixed (was: Assigned)
I think everything is landed now
Some boards are still failing. It's not anything alarming but it is curious to still see intermittent failures, especially on 4.4. I want to add more logging to also collect lsof output, as well the current /dev/dm-* mapper devices so that we can try to narrow down what's going wrong.

4.4
----
soraka

3.14
----
Paine
Veyron-[fievel|jerry|mighty]

3.18
----
asuka

3.8
----
panther
be careful with lsof ... the default can suck up a lot of data on all processes, and can hit the network.  you'll need to restrict on both fronts.
Project Member

Comment 139 by bugdroid1@chromium.org, Nov 30 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/b500559d5792811ebfd2067545baf7229422a20f

commit b500559d5792811ebfd2067545baf7229422a20f
Author: Greg Kerr <kerrnel@chromium.org>
Date: Thu Nov 30 02:04:37 2017

init: record mapper devices when umount of stateful fails.

This records the mapper devices when umount of the stateful partition
fails. This will help understand why some devices still see sporadic
umount failures.

BUG= chromium:773474 
TEST=install and reboot chromeos-init

Change-Id: Icf3f522c84e2747dca0046c35b035e25d31c3970
Reviewed-on: https://chromium-review.googlesource.com/794893
Commit-Ready: Greg Kerr <kerrnel@chromium.org>
Tested-by: Greg Kerr <kerrnel@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/b500559d5792811ebfd2067545baf7229422a20f/init/chromeos_shutdown

Project Member

Comment 140 by bugdroid1@chromium.org, Dec 2 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c961fb5853b3a57d7ddb453fd8aba3c3b33ce562

commit c961fb5853b3a57d7ddb453fd8aba3c3b33ce562
Author: Greg Kerr <kerrnel@chromium.org>
Date: Sat Dec 02 01:57:53 2017

platform_CleanShutdown: Log contents of stateful umount failure errors.

This logs the contents of shutdown_stateful_umount_failure to the
debug logging. This is necessary to understand why some lab machines
still fail to perform clean shutdown.

BUG= chromium:773474 
TEST=test_that -b ${BOARD} ${DUT_IP} platform_CleanShutdown

Change-Id: I14de9290883da4a393c96b6f067d567af2997493
Reviewed-on: https://chromium-review.googlesource.com/798025
Commit-Ready: Greg Kerr <kerrnel@chromium.org>
Tested-by: Greg Kerr <kerrnel@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/c961fb5853b3a57d7ddb453fd8aba3c3b33ce562/client/site_tests/platform_CleanShutdown/platform_CleanShutdown.py

Project Member

Comment 141 by bugdroid1@chromium.org, Dec 8 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/363f20f5fd802093198dbab572928b4e2d8a5aaf

commit 363f20f5fd802093198dbab572928b4e2d8a5aaf
Author: Greg Kerr <kerrnel@chromium.org>
Date: Fri Dec 08 11:59:10 2017

init: Record lsof output on unclean shutdown.

This records the output of lsof when stateful is not
unmounted properly on shutdown.

BUG= chromium:773474 
TEST=reboot a device uncleanly and read the log

Change-Id: Id6698d87163a6cabc5a2f8a6710b05bb3a6a2d5e
Reviewed-on: https://chromium-review.googlesource.com/813091
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Greg Kerr <kerrnel@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>

[modify] https://crrev.com/363f20f5fd802093198dbab572928b4e2d8a5aaf/init/chromeos_shutdown

Project Member

Comment 142 by bugdroid1@chromium.org, Dec 19 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/3ec925e0e113791eee61b59c3298b08b2f87f855

commit 3ec925e0e113791eee61b59c3298b08b2f87f855
Author: Eric W. Biederman <ebiederm@xmission.com>
Date: Tue Dec 19 18:16:25 2017

UPSTREAM: vfs: Document the effect of d_revalidate on d_find_alias

d_drop or check_submounts_and_drop called from d_revalidate can result
in renamed directories with child dentries being unhashed.  These
renamed and drop directory dentries can be rehashed after
d_materialise_unique uses d_find_alias to find them.

BUG= chromium:773474 
TEST=Based on https://lkml.org/lkml/2014/2/25/110 list of email.
     on samus, no mount leak.

Reviewed-by: Miklos Szeredi <miklos@szeredi.hu>
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
(cherry picked from commit 3ccb354d641d910309b916b9c856e2a82ced7237)
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>

Change-Id: Idecdda13920e0810a11da3be6e3b360a4914f6b9
Reviewed-on: https://chromium-review.googlesource.com/724101
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Tested-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
(cherry picked from commit e49f9bfab90d4fe7657a8c4f9a69219aae30d5cb)
Reviewed-on: https://chromium-review.googlesource.com/834410
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
Commit-Queue: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/3ec925e0e113791eee61b59c3298b08b2f87f855/fs/dcache.c

Project Member

Comment 143 by bugdroid1@chromium.org, Jan 2 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/a3f8f8c807472518967d4bc80befc20de1641dc6

commit a3f8f8c807472518967d4bc80befc20de1641dc6
Author: Sonny Rao <sonnyrao@chromium.org>
Date: Tue Jan 02 21:19:37 2018

CHROMIUM: cifs: fix compilation of cifs after dcache changes

In one of the dcache fixes: commit f293fc1b0fca
("BACKPORT: Don't pass inode to ->d_hash() and ->d_compare()")
We didn't fix up a call to d_hash in CIFS -- so let's fix that.

BUG= chromium:773474 
TEST=build for link with CONFIG_CIFS

Change-Id: I10972cac4ed7da3d948ad7532ecbd9ca15a9eabc
Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/844237
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/a3f8f8c807472518967d4bc80befc20de1641dc6/fs/cifs/readdir.c

Status: archived (was: Fixed)
Status: Fixed (was: Archived)
Showing comments 46 - 145 of 145 Older

Sign in to add a comment