I think this could just use vmalloc instead. There might be even a patch
already upstream that changes this.

2017/05/13 0:55 "dianders via monorail" <monorail+v2.2711505450@chromium.org

Hmm, apparently upstream is sill using kzalloc() in v4l2_ctrl_new(). Possibly the patch I saw before was changing some other allocation.

Anyway, let me try to make this fall back to vzalloc() if the allocation is bigger than PAGE_SIZE and use kvfree() for freeing this. As long as the control buffers are not supposed to be directly DMA-able (they're not in case of our drivers), it should be fine. I can send it to upstream and get some comments.

On the other hand, are we sure there isn't something not working properly inside the mm code? The order >0 allocation functionality is there for some reason and there are things other than OOM killer that should be able to reclaim enough memory to fulfill the request (page cache reclaim, compaction).

@2: Compaction is turned off in all of our kernels besides 4.4.  I tried to turn it on but the graphics team turned it back off.  See <https://chromium-review.googlesource.com/339812>.  By that time I was too deep into Kevin to really track down the issues and we had no test cases showing the benefit of compaction, so it stayed off.

Due to the lack of compaction, there's really nothing that can help make larger chunks of memory.  

I agree that there may be just "the next thing" that will try to allocate a big chunk of memory, but it still seems sane to at least make this particular thing behave better.  :)

CL as per #2 at crosreview.com/509090.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/3cdaeb0273d8d8ac868325f062edb1af203a3178

commit 3cdaeb0273d8d8ac868325f062edb1af203a3178
Author: Michal Hocko <mhocko@suse.com>
Date: Fri Jun 02 06:49:19 2017

BACKPORT: mm: introduce kv[mz]alloc helpers

Patch series "kvmalloc", v5.

There are many open coded kmalloc with vmalloc fallback instances in the
tree.  Most of them are not careful enough or simply do not care about
the underlying semantic of the kmalloc/page allocator which means that
a) some vmalloc fallbacks are basically unreachable because the kmalloc
part will keep retrying until it succeeds b) the page allocator can
invoke a really disruptive steps like the OOM killer to move forward
which doesn't sound appropriate when we consider that the vmalloc
fallback is available.

As it can be seen implementing kvmalloc requires quite an intimate
knowledge if the page allocator and the memory reclaim internals which
strongly suggests that a helper should be implemented in the memory
subsystem proper.

Most callers, I could find, have been converted to use the helper
instead.  This is patch 6.  There are some more relying on __GFP_REPEAT
in the networking stack which I have converted as well and Eric Dumazet
was not opposed [2] to convert them as well.

[1] http://lkml.kernel.org/r/20170130094940.13546-1-mhocko@kernel.org
[2] http://lkml.kernel.org/r/1485273626.16328.301.camel@edumazet-glaptop3.roam.corp.google.com

This patch (of 9):

Using kmalloc with the vmalloc fallback for larger allocations is a
common pattern in the kernel code.  Yet we do not have any common helper
for that and so users have invented their own helpers.  Some of them are
really creative when doing so.  Let's just add kv[mz]alloc and make sure
it is implemented properly.  This implementation makes sure to not make
a large memory pressure for > PAGE_SZE requests (__GFP_NORETRY) and also
to not warn about allocation failures.  This also rules out the OOM
killer as the vmalloc is a more approapriate fallback than a disruptive
user visible action.

This patch also changes some existing users and removes helpers which
are specific for them.  In some cases this is not possible (e.g.
ext4_kvmalloc, libcfs_kvzalloc) because those seems to be broken and
require GFP_NO{FS,IO} context which is not vmalloc compatible in general
(note that the page table allocation is GFP_KERNEL).  Those need to be
fixed separately.

While we are at it, document that __vmalloc{_node} about unsupported gfp
mask because there seems to be a lot of confusion out there.
kvmalloc_node will warn about GFP_KERNEL incompatible (which are not
superset) flags to catch new abusers.  Existing ones would have to die
slowly.

[sfr@canb.auug.org.au: f2fs fixup]
  Link: http://lkml.kernel.org/r/20170320163735.332e64b7@canb.auug.org.au
Link: http://lkml.kernel.org/r/20170306103032.2540-2-mhocko@kernel.org
Signed-off-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au>
Reviewed-by: Andreas Dilger <adilger@dilger.ca>	[ext4 part]
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: David Miller <davem@davemloft.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from a7c3e901a46ff54c016d040847eda598a9e3e653)
[tfiga: Removed changes for users of the API from the commit. Kept only
the new API itself.]

BUG= chromium:721786 
TEST=compile

Change-Id: I6f4ea7da69285f07470a58b1b92970c24af2f365
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/509088
Reviewed-by: Douglas Anderson <dianders@chromium.org>

[modify] https://crrev.com/3cdaeb0273d8d8ac868325f062edb1af203a3178/include/linux/mm.h
[modify] https://crrev.com/3cdaeb0273d8d8ac868325f062edb1af203a3178/mm/util.c
[modify] https://crrev.com/3cdaeb0273d8d8ac868325f062edb1af203a3178/mm/vmalloc.c
[modify] https://crrev.com/3cdaeb0273d8d8ac868325f062edb1af203a3178/mm/nommu.c
[modify] https://crrev.com/3cdaeb0273d8d8ac868325f062edb1af203a3178/include/linux/vmalloc.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/acdb14324ec0edc581bef217a6edc84f04629491

commit acdb14324ec0edc581bef217a6edc84f04629491
Author: Tomasz Figa <tfiga@chromium.org>
Date: Fri Jun 02 06:49:20 2017

CHROMIUM: mm: Take kvmalloc_array() from upstream

Comes from upstream commit 752ade68cbd8 ("treewide: use kv[mz]alloc*
rather than opencoded variants"), which also converted existing code to
use it, but it doesn't apply cleanly to 3.14. Original commit message:

There are many code paths opencoding kvmalloc.  Let's use the helper
instead.  The main difference to kvmalloc is that those users are
usually not considering all the aspects of the memory allocator.  E.g.
allocation requests <= 32kB (with 4kB pages) are basically never failing
and invoke OOM killer to satisfy the allocation.  This sounds too
disruptive for something that has a reasonable fallback - the vmalloc.
On the other hand those requests might fallback to vmalloc even when the
memory allocator would succeed after several more reclaim/compaction
attempts previously.  There is no guarantee something like that happens
though.

This patch converts many of those places to kv[mz]alloc* helpers because
they are more conservative.

Link: http://lkml.kernel.org/r/20170306103327.2766-2-mhocko@kernel.org
Signed-off-by: Michal Hocko <mhocko@suse.com>
Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com> # Xen bits
Acked-by: Kees Cook <keescook@chromium.org>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Acked-by: Andreas Dilger <andreas.dilger@intel.com> # Lustre
Acked-by: Christian Borntraeger <borntraeger@de.ibm.com> # KVM/s390
Acked-by: Dan Williams <dan.j.williams@intel.com> # nvdim
Acked-by: David Sterba <dsterba@suse.com> # btrfs
Acked-by: Ilya Dryomov <idryomov@gmail.com> # Ceph
Acked-by: Tariq Toukan <tariqt@mellanox.com> # mlx4
Acked-by: Leon Romanovsky <leonro@mellanox.com> # mlx5
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Anton Vorontsov <anton@enomsg.org>
Cc: Colin Cross <ccross@android.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>
Cc: Ben Skeggs <bskeggs@redhat.com>
Cc: Kent Overstreet <kent.overstreet@gmail.com>
Cc: Santosh Raspatur <santosh@chelsio.com>
Cc: Hariprasad S <hariprasad@chelsio.com>
Cc: Yishai Hadas <yishaih@mellanox.com>
Cc: Oleg Drokin <oleg.drokin@intel.com>
Cc: "Yan, Zheng" <zyan@redhat.com>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: David Miller <davem@davemloft.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

BUG= chromium:721786 
TEST=compile

Change-Id: I093d657ced5b6e1105829c67b48ada6170c8544e
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/509089
Reviewed-by: Douglas Anderson <dianders@chromium.org>

[modify] https://crrev.com/acdb14324ec0edc581bef217a6edc84f04629491/include/linux/mm.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/749978e9f613bbee0fbae4078cdfc072355d1e39

commit 749978e9f613bbee0fbae4078cdfc072355d1e39
Author: Tomasz Figa <tfiga@chromium.org>
Date: Fri Jun 02 06:49:23 2017

CHROMIUM: [media] v4l2-core: Use kvmalloc() for potentially big allocations

There are multiple places where arrays or otherwise variable sized
buffer are allocated through V4L2 core code, including things like
controls, memory pages, staging buffers for ioctls and so on. Such
allocations can potentially require an order > 0 allocation from the
page allocator, which is not guaranteed to be fulfilled and is likely to
fail on a system with severe memory fragmentation (e.g. a system with
very long uptime).

Since the memory being allocated is intended to be used by the CPU
exclusively, we can consider using vmalloc() as a fallback and this is
exactly what the recently merged kvmalloc() helpers do. A kmalloc() call
is still attempted, even for order > 0 allocations, but it is done
with __GFP_NORETRY and __GFP_NOWARN, with expectation of failing if
requested memory is not available instantly. Only then the vmalloc()
fallback is used. This should give us fast and more reliable allocations
even on systems with higher memory pressure and/or more fragmentation,
while still retaining the same performance level on systems not
suffering from such conditions.

While at it, replace explicit array size calculations on changed
allocations with kvmalloc_array().

(An equivalent patch was sent upstream separately due to a big number of
merge conflicts: https://patchwork.linuxtv.org/patch/41547/)

BUG= chromium:721786 
TEST=test_that video_VideoDecodeAccelerator.vp8 on veyron_minnie
TEST=test_that video_VideoEncodeAccelerator.vp8 on veyron_minnie

Change-Id: I690bd14bc15e60187bae6058bc2d7c91dcf232e5
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/509090
Reviewed-by: Pawel Osciak <posciak@chromium.org>

[modify] https://crrev.com/749978e9f613bbee0fbae4078cdfc072355d1e39/drivers/media/v4l2-core/videobuf2-dma-contig.c
[modify] https://crrev.com/749978e9f613bbee0fbae4078cdfc072355d1e39/drivers/media/v4l2-core/v4l2-ioctl.c
[modify] https://crrev.com/749978e9f613bbee0fbae4078cdfc072355d1e39/drivers/media/v4l2-core/v4l2-subdev.c
[modify] https://crrev.com/749978e9f613bbee0fbae4078cdfc072355d1e39/drivers/media/v4l2-core/videobuf2-vmalloc.c
[modify] https://crrev.com/749978e9f613bbee0fbae4078cdfc072355d1e39/drivers/media/v4l2-core/v4l2-async.c
[modify] https://crrev.com/749978e9f613bbee0fbae4078cdfc072355d1e39/drivers/media/v4l2-core/v4l2-ctrls.c
[modify] https://crrev.com/749978e9f613bbee0fbae4078cdfc072355d1e39/drivers/media/v4l2-core/v4l2-event.c
[modify] https://crrev.com/749978e9f613bbee0fbae4078cdfc072355d1e39/drivers/media/v4l2-core/videobuf2-dma-sg.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/02efde95af2b753963ec1de00937f7ebb6001bf9

commit 02efde95af2b753963ec1de00937f7ebb6001bf9
Author: Michal Hocko <mhocko@suse.com>
Date: Fri Jun 02 06:49:21 2017

BACKPORT: mm, vmalloc: properly track vmalloc users

__vmalloc_node_flags used to be static inline but this has changed by
"mm: introduce kv[mz]alloc helpers" because kvmalloc_node needs to use
it as well and the code is outside of the vmalloc proper.  I haven't
realized that changing this will lead to a subtle bug though.  The
function is responsible to track the caller as well.  This caller is
then printed by /proc/vmallocinfo.  If __vmalloc_node_flags is not
inline then we would get only direct users of __vmalloc_node_flags as
callers (e.g.  v[mz]alloc) which reduces usefulness of this debugging
feature considerably.  It simply doesn't help to see that the given
range belongs to vmalloc as a caller:

  0xffffc90002c79000-0xffffc90002c7d000   16384 vmalloc+0x16/0x18 pages=3 vmalloc N0=3
  0xffffc90002c81000-0xffffc90002c85000   16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3
  0xffffc90002c8d000-0xffffc90002c91000   16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3
  0xffffc90002c95000-0xffffc90002c99000   16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3

We really want to catch the _caller_ of the vmalloc function.  Fix this
issue by making __vmalloc_node_flags static inline again.

Link: http://lkml.kernel.org/r/20170502134657.12381-1-mhocko@kernel.org
Signed-off-by: Michal Hocko <mhocko@suse.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit 1f5307b1e094bfffa83c65c40ac6e3415c108780)

BUG= chromium:721786 
TEST=compile

Change-Id: I992497279e346cdce8357ab55ada6c6d45d758b1
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/512422
Reviewed-by: Douglas Anderson <dianders@chromium.org>

[modify] https://crrev.com/02efde95af2b753963ec1de00937f7ebb6001bf9/mm/vmalloc.c
[modify] https://crrev.com/02efde95af2b753963ec1de00937f7ebb6001bf9/include/linux/vmalloc.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/bf63686b65c825db6c923b887b6a8bbfa95854fd

commit bf63686b65c825db6c923b887b6a8bbfa95854fd
Author: Michal Hocko <mhocko@suse.com>
Date: Fri Jun 02 06:49:22 2017

BACKPORT: mm, vmalloc: fix vmalloc users tracking properly

Commit 1f5307b1e094 ("mm, vmalloc: properly track vmalloc users") has
pulled asm/pgtable.h include dependency to linux/vmalloc.h and that
turned out to be a bad idea for some architectures.  E.g.  m68k fails
with

   In file included from arch/m68k/include/asm/pgtable_mm.h:145:0,
                    from arch/m68k/include/asm/pgtable.h:4,
                    from include/linux/vmalloc.h:9,
                    from arch/m68k/kernel/module.c:9:
   arch/m68k/include/asm/mcf_pgtable.h: In function 'nocache_page':
>> arch/m68k/include/asm/mcf_pgtable.h:339:43: error: 'init_mm' undeclared (first use in this function)
    #define pgd_offset_k(address) pgd_offset(&init_mm, address)

as spotted by kernel build bot. nios2 fails for other reason

  In file included from include/asm-generic/io.h:767:0,
                   from arch/nios2/include/asm/io.h:61,
                   from include/linux/io.h:25,
                   from arch/nios2/include/asm/pgtable.h:18,
                   from include/linux/mm.h:70,
                   from include/linux/pid_namespace.h:6,
                   from include/linux/ptrace.h:9,
                   from arch/nios2/include/uapi/asm/elf.h:23,
                   from arch/nios2/include/asm/elf.h:22,
                   from include/linux/elf.h:4,
                   from include/linux/module.h:15,
                   from init/main.c:16:
  include/linux/vmalloc.h: In function '__vmalloc_node_flags':
  include/linux/vmalloc.h:99:40: error: 'PAGE_KERNEL' undeclared (first use in this function); did you mean 'GFP_KERNEL'?

which is due to the newly added #include <asm/pgtable.h>, which on nios2
includes <linux/io.h> and thus <asm/io.h> and <asm-generic/io.h> which
again includes <linux/vmalloc.h>.

Tweaking that around just turns out a bigger headache than necessary.
This patch reverts 1f5307b1e094 and reimplements the original fix in a
different way.  __vmalloc_node_flags can stay static inline which will
cover vmalloc* functions.  We only have one external user
(kvmalloc_node) and we can export __vmalloc_node_flags_caller and
provide the caller directly.  This is much simpler and it doesn't really
need any games with header files.

[akpm@linux-foundation.org: coding-style fixes]
[mhocko@kernel.org: revert old comment]
  Link: http://lkml.kernel.org/r/20170509211054.GB16325@dhcp22.suse.cz
Fixes: 1f5307b1e094 ("mm, vmalloc: properly track vmalloc users")
Link: http://lkml.kernel.org/r/20170509153702.GR6481@dhcp22.suse.cz
Signed-off-by: Michal Hocko <mhocko@suse.com>
Cc: Tobias Klauser <tklauser@distanz.ch>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit 8594a21cf7a8318baedbedc3fcd2967a17ddeec0)

BUG= chromium:721786 
TEST=compile

Change-Id: Ic7486d53e101995f71b5fc6e43bb588d512610aa
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/512423
Reviewed-by: Douglas Anderson <dianders@chromium.org>

[modify] https://crrev.com/bf63686b65c825db6c923b887b6a8bbfa95854fd/mm/util.c
[modify] https://crrev.com/bf63686b65c825db6c923b887b6a8bbfa95854fd/mm/vmalloc.c
[modify] https://crrev.com/bf63686b65c825db6c923b887b6a8bbfa95854fd/include/linux/vmalloc.h

Should be fixed in ToT now.

Thanks!  This landed in M-61, so adding a milestone for validation.  This isn't a new regression and has been there forever.  By default, not suggesting a merge to M-60.

To me it seems like we probably want this on other kernels.  Opened  bug #729038  to track.

Verified using https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/testDetails?testName=video_VideoDecodeAccelerator.vp8&suite=&daysBack=30&board=&architecture=&boardFamily=&buildConfig=&reason=&version=&milestone=61&dut=

Also verified by running videos in loop on minnie device.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/51227ec5e4bbccd43e8a84c5db12745d0bf8a640

commit 51227ec5e4bbccd43e8a84c5db12745d0bf8a640
Author: Michal Hocko <mhocko@suse.com>
Date: Wed Aug 30 20:04:11 2017

BACKPORT: mm: introduce kv[mz]alloc helpers

Patch series "kvmalloc", v5.

There are many open coded kmalloc with vmalloc fallback instances in the
tree.  Most of them are not careful enough or simply do not care about
the underlying semantic of the kmalloc/page allocator which means that
a) some vmalloc fallbacks are basically unreachable because the kmalloc
part will keep retrying until it succeeds b) the page allocator can
invoke a really disruptive steps like the OOM killer to move forward
which doesn't sound appropriate when we consider that the vmalloc
fallback is available.

As it can be seen implementing kvmalloc requires quite an intimate
knowledge if the page allocator and the memory reclaim internals which
strongly suggests that a helper should be implemented in the memory
subsystem proper.

Most callers, I could find, have been converted to use the helper
instead.  This is patch 6.  There are some more relying on __GFP_REPEAT
in the networking stack which I have converted as well and Eric Dumazet
was not opposed [2] to convert them as well.

[1] http://lkml.kernel.org/r/20170130094940.13546-1-mhocko@kernel.org
[2] http://lkml.kernel.org/r/1485273626.16328.301.camel@edumazet-glaptop3.roam.corp.google.com

This patch (of 9):

Using kmalloc with the vmalloc fallback for larger allocations is a
common pattern in the kernel code.  Yet we do not have any common helper
for that and so users have invented their own helpers.  Some of them are
really creative when doing so.  Let's just add kv[mz]alloc and make sure
it is implemented properly.  This implementation makes sure to not make
a large memory pressure for > PAGE_SZE requests (__GFP_NORETRY) and also
to not warn about allocation failures.  This also rules out the OOM
killer as the vmalloc is a more approapriate fallback than a disruptive
user visible action.

This patch also changes some existing users and removes helpers which
are specific for them.  In some cases this is not possible (e.g.
ext4_kvmalloc, libcfs_kvzalloc) because those seems to be broken and
require GFP_NO{FS,IO} context which is not vmalloc compatible in general
(note that the page table allocation is GFP_KERNEL).  Those need to be
fixed separately.

While we are at it, document that __vmalloc{_node} about unsupported gfp
mask because there seems to be a lot of confusion out there.
kvmalloc_node will warn about GFP_KERNEL incompatible (which are not
superset) flags to catch new abusers.  Existing ones would have to die
slowly.

[sfr@canb.auug.org.au: f2fs fixup]
  Link: http://lkml.kernel.org/r/20170320163735.332e64b7@canb.auug.org.au
Link: http://lkml.kernel.org/r/20170306103032.2540-2-mhocko@kernel.org
Signed-off-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au>
Reviewed-by: Andreas Dilger <adilger@dilger.ca>	[ext4 part]
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: David Miller <davem@davemloft.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit a7c3e901a46ff54c016d040847eda598a9e3e653)
[tfiga: Removed changes for users of the API from the commit. Kept only
the new API itself.]

BUG= chromium:721786 
TEST=compile

Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/509088
Reviewed-by: Douglas Anderson <dianders@chromium.org>
(cherry picked from commit 3cdaeb0273d8d8ac868325f062edb1af203a3178)

BUG= chromium:729038 
TEST=compile

Change-Id: I6f4ea7da69285f07470a58b1b92970c24af2f365
Reviewed-on: https://chromium-review.googlesource.com/608007
Commit-Ready: Tomasz Figa <tfiga@chromium.org>
Tested-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>

[modify] https://crrev.com/51227ec5e4bbccd43e8a84c5db12745d0bf8a640/include/linux/mm.h
[modify] https://crrev.com/51227ec5e4bbccd43e8a84c5db12745d0bf8a640/mm/util.c
[modify] https://crrev.com/51227ec5e4bbccd43e8a84c5db12745d0bf8a640/mm/vmalloc.c
[modify] https://crrev.com/51227ec5e4bbccd43e8a84c5db12745d0bf8a640/mm/nommu.c
[modify] https://crrev.com/51227ec5e4bbccd43e8a84c5db12745d0bf8a640/include/linux/vmalloc.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ddceb3d384580c2c0a70b3e26fc6e45ad434342d

commit ddceb3d384580c2c0a70b3e26fc6e45ad434342d
Author: Tomasz Figa <tfiga@chromium.org>
Date: Wed Aug 30 20:04:12 2017

CHROMIUM: mm: Take kvmalloc_array() from upstream

Comes from upstream commit 752ade68cbd8 ("treewide: use kv[mz]alloc*
rather than opencoded variants"), which also converted existing code to
use it, but it doesn't apply cleanly to 3.14. Original commit message:

There are many code paths opencoding kvmalloc.  Let's use the helper
instead.  The main difference to kvmalloc is that those users are
usually not considering all the aspects of the memory allocator.  E.g.
allocation requests <= 32kB (with 4kB pages) are basically never failing
and invoke OOM killer to satisfy the allocation.  This sounds too
disruptive for something that has a reasonable fallback - the vmalloc.
On the other hand those requests might fallback to vmalloc even when the
memory allocator would succeed after several more reclaim/compaction
attempts previously.  There is no guarantee something like that happens
though.

This patch converts many of those places to kv[mz]alloc* helpers because
they are more conservative.

Link: http://lkml.kernel.org/r/20170306103327.2766-2-mhocko@kernel.org
Signed-off-by: Michal Hocko <mhocko@suse.com>
Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com> # Xen bits
Acked-by: Kees Cook <keescook@chromium.org>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Acked-by: Andreas Dilger <andreas.dilger@intel.com> # Lustre
Acked-by: Christian Borntraeger <borntraeger@de.ibm.com> # KVM/s390
Acked-by: Dan Williams <dan.j.williams@intel.com> # nvdim
Acked-by: David Sterba <dsterba@suse.com> # btrfs
Acked-by: Ilya Dryomov <idryomov@gmail.com> # Ceph
Acked-by: Tariq Toukan <tariqt@mellanox.com> # mlx4
Acked-by: Leon Romanovsky <leonro@mellanox.com> # mlx5
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Anton Vorontsov <anton@enomsg.org>
Cc: Colin Cross <ccross@android.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>
Cc: Ben Skeggs <bskeggs@redhat.com>
Cc: Kent Overstreet <kent.overstreet@gmail.com>
Cc: Santosh Raspatur <santosh@chelsio.com>
Cc: Hariprasad S <hariprasad@chelsio.com>
Cc: Yishai Hadas <yishaih@mellanox.com>
Cc: Oleg Drokin <oleg.drokin@intel.com>
Cc: "Yan, Zheng" <zyan@redhat.com>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: David Miller <davem@davemloft.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

BUG= chromium:721786 
TEST=compile

Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/509089
Reviewed-by: Douglas Anderson <dianders@chromium.org>
(cherry picked from commit acdb14324ec0edc581bef217a6edc84f04629491)

BUG= chromium:729038 
TEST=compile

Change-Id: I093d657ced5b6e1105829c67b48ada6170c8544e
Reviewed-on: https://chromium-review.googlesource.com/608008
Commit-Ready: Tomasz Figa <tfiga@chromium.org>
Tested-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>

[modify] https://crrev.com/ddceb3d384580c2c0a70b3e26fc6e45ad434342d/include/linux/mm.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/641f4f4717d8e8e8a2f74350e418d8e16a5272c0

commit 641f4f4717d8e8e8a2f74350e418d8e16a5272c0
Author: Michal Hocko <mhocko@suse.com>
Date: Wed Aug 30 20:04:13 2017

UPSTREAM: mm, vmalloc: properly track vmalloc users

__vmalloc_node_flags used to be static inline but this has changed by
"mm: introduce kv[mz]alloc helpers" because kvmalloc_node needs to use
it as well and the code is outside of the vmalloc proper.  I haven't
realized that changing this will lead to a subtle bug though.  The
function is responsible to track the caller as well.  This caller is
then printed by /proc/vmallocinfo.  If __vmalloc_node_flags is not
inline then we would get only direct users of __vmalloc_node_flags as
callers (e.g.  v[mz]alloc) which reduces usefulness of this debugging
feature considerably.  It simply doesn't help to see that the given
range belongs to vmalloc as a caller:

  0xffffc90002c79000-0xffffc90002c7d000   16384 vmalloc+0x16/0x18 pages=3 vmalloc N0=3
  0xffffc90002c81000-0xffffc90002c85000   16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3
  0xffffc90002c8d000-0xffffc90002c91000   16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3
  0xffffc90002c95000-0xffffc90002c99000   16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3

We really want to catch the _caller_ of the vmalloc function.  Fix this
issue by making __vmalloc_node_flags static inline again.

Link: http://lkml.kernel.org/r/20170502134657.12381-1-mhocko@kernel.org
Signed-off-by: Michal Hocko <mhocko@suse.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit 1f5307b1e094bfffa83c65c40ac6e3415c108780)

BUG= chromium:721786 
TEST=compile

Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/512422
Reviewed-by: Douglas Anderson <dianders@chromium.org>
(cherry picked from commit 02efde95af2b753963ec1de00937f7ebb6001bf9)

BUG= chromium:729038 
TEST=compile

Change-Id: I992497279e346cdce8357ab55ada6c6d45d758b1
Reviewed-on: https://chromium-review.googlesource.com/608009
Commit-Ready: Tomasz Figa <tfiga@chromium.org>
Tested-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>

[modify] https://crrev.com/641f4f4717d8e8e8a2f74350e418d8e16a5272c0/mm/vmalloc.c
[modify] https://crrev.com/641f4f4717d8e8e8a2f74350e418d8e16a5272c0/include/linux/vmalloc.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c90cac68a003989d7ba64ea661e1687bff2a7204

commit c90cac68a003989d7ba64ea661e1687bff2a7204
Author: Michal Hocko <mhocko@suse.com>
Date: Wed Aug 30 20:04:14 2017

BACKPORT: mm, vmalloc: fix vmalloc users tracking properly

Commit 1f5307b1e094 ("mm, vmalloc: properly track vmalloc users") has
pulled asm/pgtable.h include dependency to linux/vmalloc.h and that
turned out to be a bad idea for some architectures.  E.g.  m68k fails
with

   In file included from arch/m68k/include/asm/pgtable_mm.h:145:0,
                    from arch/m68k/include/asm/pgtable.h:4,
                    from include/linux/vmalloc.h:9,
                    from arch/m68k/kernel/module.c:9:
   arch/m68k/include/asm/mcf_pgtable.h: In function 'nocache_page':
>> arch/m68k/include/asm/mcf_pgtable.h:339:43: error: 'init_mm' undeclared (first use in this function)
    #define pgd_offset_k(address) pgd_offset(&init_mm, address)

as spotted by kernel build bot. nios2 fails for other reason

  In file included from include/asm-generic/io.h:767:0,
                   from arch/nios2/include/asm/io.h:61,
                   from include/linux/io.h:25,
                   from arch/nios2/include/asm/pgtable.h:18,
                   from include/linux/mm.h:70,
                   from include/linux/pid_namespace.h:6,
                   from include/linux/ptrace.h:9,
                   from arch/nios2/include/uapi/asm/elf.h:23,
                   from arch/nios2/include/asm/elf.h:22,
                   from include/linux/elf.h:4,
                   from include/linux/module.h:15,
                   from init/main.c:16:
  include/linux/vmalloc.h: In function '__vmalloc_node_flags':
  include/linux/vmalloc.h:99:40: error: 'PAGE_KERNEL' undeclared (first use in this function); did you mean 'GFP_KERNEL'?

which is due to the newly added #include <asm/pgtable.h>, which on nios2
includes <linux/io.h> and thus <asm/io.h> and <asm-generic/io.h> which
again includes <linux/vmalloc.h>.

Tweaking that around just turns out a bigger headache than necessary.
This patch reverts 1f5307b1e094 and reimplements the original fix in a
different way.  __vmalloc_node_flags can stay static inline which will
cover vmalloc* functions.  We only have one external user
(kvmalloc_node) and we can export __vmalloc_node_flags_caller and
provide the caller directly.  This is much simpler and it doesn't really
need any games with header files.

[akpm@linux-foundation.org: coding-style fixes]
[mhocko@kernel.org: revert old comment]
  Link: http://lkml.kernel.org/r/20170509211054.GB16325@dhcp22.suse.cz
Fixes: 1f5307b1e094 ("mm, vmalloc: properly track vmalloc users")
Link: http://lkml.kernel.org/r/20170509153702.GR6481@dhcp22.suse.cz
Signed-off-by: Michal Hocko <mhocko@suse.com>
Cc: Tobias Klauser <tklauser@distanz.ch>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit 8594a21cf7a8318baedbedc3fcd2967a17ddeec0)

BUG= chromium:721786 
TEST=compile

Change-Id: Ic7486d53e101995f71b5fc6e43bb588d512610aa
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/512423
Reviewed-by: Douglas Anderson <dianders@chromium.org>
(cherry picked from commit bf63686b65c825db6c923b887b6a8bbfa95854fd)

 Conflicts:
	mm/util.c

BUG= chromium:729038 
TEST=compile

Change-Id: Ic7486d53e101995f71b5fc6e43bb588d512610aa
Reviewed-on: https://chromium-review.googlesource.com/608010
Commit-Ready: Tomasz Figa <tfiga@chromium.org>
Tested-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>

[modify] https://crrev.com/c90cac68a003989d7ba64ea661e1687bff2a7204/mm/util.c
[modify] https://crrev.com/c90cac68a003989d7ba64ea661e1687bff2a7204/mm/vmalloc.c
[modify] https://crrev.com/c90cac68a003989d7ba64ea661e1687bff2a7204/include/linux/vmalloc.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/52f62342a711809e0ae91124ed94cb9b899d1560

commit 52f62342a711809e0ae91124ed94cb9b899d1560
Author: Tomasz Figa <tfiga@chromium.org>
Date: Wed Aug 30 20:04:16 2017

CHROMIUM: [media] v4l2-core: Use kvmalloc() for potentially big allocations

There are multiple places where arrays or otherwise variable sized
buffer are allocated through V4L2 core code, including things like
controls, memory pages, staging buffers for ioctls and so on. Such
allocations can potentially require an order > 0 allocation from the
page allocator, which is not guaranteed to be fulfilled and is likely to
fail on a system with severe memory fragmentation (e.g. a system with
very long uptime).

Since the memory being allocated is intended to be used by the CPU
exclusively, we can consider using vmalloc() as a fallback and this is
exactly what the recently merged kvmalloc() helpers do. A kmalloc() call
is still attempted, even for order > 0 allocations, but it is done
with __GFP_NORETRY and __GFP_NOWARN, with expectation of failing if
requested memory is not available instantly. Only then the vmalloc()
fallback is used. This should give us fast and more reliable allocations
even on systems with higher memory pressure and/or more fragmentation,
while still retaining the same performance level on systems not
suffering from such conditions.

While at it, replace explicit array size calculations on changed
allocations with kvmalloc_array().

(An equivalent patch was sent upstream separately due to a big number of
merge conflicts: https://patchwork.linuxtv.org/patch/41547/)

BUG= chromium:721786 
TEST=test_that video_VideoDecodeAccelerator.vp8 on veyron_minnie
TEST=test_that video_VideoEncodeAccelerator.vp8 on veyron_minnie

Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/509090
Reviewed-by: Pawel Osciak <posciak@chromium.org>
(cherry picked from commit 749978e9f613bbee0fbae4078cdfc072355d1e39)

BUG= chromium:729038 
TEST=compile

Change-Id: I690bd14bc15e60187bae6058bc2d7c91dcf232e5
Reviewed-on: https://chromium-review.googlesource.com/608011
Commit-Ready: Tomasz Figa <tfiga@chromium.org>
Tested-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>

[modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/videobuf2-dma-contig.c
[modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/v4l2-ioctl.c
[modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/v4l2-subdev.c
[modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/videobuf2-vmalloc.c
[modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/v4l2-async.c
[modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/v4l2-ctrls.c
[modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/v4l2-event.c
[modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/videobuf2-dma-sg.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/5597af9e8f7c33dae55657320e4a754ab3d949fe

commit 5597af9e8f7c33dae55657320e4a754ab3d949fe
Author: Michal Hocko <mhocko@suse.com>
Date: Fri Sep 29 18:24:30 2017

BACKPORT: mm: introduce kv[mz]alloc helpers

Patch series "kvmalloc", v5.

There are many open coded kmalloc with vmalloc fallback instances in the
tree.  Most of them are not careful enough or simply do not care about
the underlying semantic of the kmalloc/page allocator which means that
a) some vmalloc fallbacks are basically unreachable because the kmalloc
part will keep retrying until it succeeds b) the page allocator can
invoke a really disruptive steps like the OOM killer to move forward
which doesn't sound appropriate when we consider that the vmalloc
fallback is available.

As it can be seen implementing kvmalloc requires quite an intimate
knowledge if the page allocator and the memory reclaim internals which
strongly suggests that a helper should be implemented in the memory
subsystem proper.

Most callers, I could find, have been converted to use the helper
instead.  This is patch 6.  There are some more relying on __GFP_REPEAT
in the networking stack which I have converted as well and Eric Dumazet
was not opposed [2] to convert them as well.

[1] http://lkml.kernel.org/r/20170130094940.13546-1-mhocko@kernel.org
[2] http://lkml.kernel.org/r/1485273626.16328.301.camel@edumazet-glaptop3.roam.corp.google.com

This patch (of 9):

Using kmalloc with the vmalloc fallback for larger allocations is a
common pattern in the kernel code.  Yet we do not have any common helper
for that and so users have invented their own helpers.  Some of them are
really creative when doing so.  Let's just add kv[mz]alloc and make sure
it is implemented properly.  This implementation makes sure to not make
a large memory pressure for > PAGE_SZE requests (__GFP_NORETRY) and also
to not warn about allocation failures.  This also rules out the OOM
killer as the vmalloc is a more approapriate fallback than a disruptive
user visible action.

This patch also changes some existing users and removes helpers which
are specific for them.  In some cases this is not possible (e.g.
ext4_kvmalloc, libcfs_kvzalloc) because those seems to be broken and
require GFP_NO{FS,IO} context which is not vmalloc compatible in general
(note that the page table allocation is GFP_KERNEL).  Those need to be
fixed separately.

While we are at it, document that __vmalloc{_node} about unsupported gfp
mask because there seems to be a lot of confusion out there.
kvmalloc_node will warn about GFP_KERNEL incompatible (which are not
superset) flags to catch new abusers.  Existing ones would have to die
slowly.

[sfr@canb.auug.org.au: f2fs fixup]
  Link: http://lkml.kernel.org/r/20170320163735.332e64b7@canb.auug.org.au
Link: http://lkml.kernel.org/r/20170306103032.2540-2-mhocko@kernel.org
Signed-off-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au>
Reviewed-by: Andreas Dilger <adilger@dilger.ca>	[ext4 part]
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: David Miller <davem@davemloft.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit a7c3e901a46ff54c016d040847eda598a9e3e653)
[tfiga: Removed changes for users of the API from the commit. Kept only
the new API itself.]

BUG= chromium:721786 
TEST=compile

Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/509088
Reviewed-by: Douglas Anderson <dianders@chromium.org>
(cherry picked from commit 3cdaeb0273d8d8ac868325f062edb1af203a3178)

BUG= chromium:729038 
TEST=compile

Change-Id: I6f4ea7da69285f07470a58b1b92970c24af2f365
Reviewed-on: https://chromium-review.googlesource.com/606752
Commit-Ready: Tomasz Figa <tfiga@chromium.org>
Tested-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>

[modify] https://crrev.com/5597af9e8f7c33dae55657320e4a754ab3d949fe/include/linux/mm.h
[modify] https://crrev.com/5597af9e8f7c33dae55657320e4a754ab3d949fe/mm/util.c
[modify] https://crrev.com/5597af9e8f7c33dae55657320e4a754ab3d949fe/mm/vmalloc.c
[modify] https://crrev.com/5597af9e8f7c33dae55657320e4a754ab3d949fe/mm/nommu.c
[modify] https://crrev.com/5597af9e8f7c33dae55657320e4a754ab3d949fe/include/linux/vmalloc.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/3f9139a047b7303ffd48baec47e0ec940fedc2dc

commit 3f9139a047b7303ffd48baec47e0ec940fedc2dc
Author: Tomasz Figa <tfiga@chromium.org>
Date: Fri Sep 29 18:24:31 2017

CHROMIUM: mm: Take kvmalloc_array() from upstream

Comes from upstream commit 752ade68cbd8 ("treewide: use kv[mz]alloc*
rather than opencoded variants"), which also converted existing code to
use it, but it doesn't apply cleanly to 3.14. Original commit message:

There are many code paths opencoding kvmalloc.  Let's use the helper
instead.  The main difference to kvmalloc is that those users are
usually not considering all the aspects of the memory allocator.  E.g.
allocation requests <= 32kB (with 4kB pages) are basically never failing
and invoke OOM killer to satisfy the allocation.  This sounds too
disruptive for something that has a reasonable fallback - the vmalloc.
On the other hand those requests might fallback to vmalloc even when the
memory allocator would succeed after several more reclaim/compaction
attempts previously.  There is no guarantee something like that happens
though.

This patch converts many of those places to kv[mz]alloc* helpers because
they are more conservative.

Link: http://lkml.kernel.org/r/20170306103327.2766-2-mhocko@kernel.org
Signed-off-by: Michal Hocko <mhocko@suse.com>
Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com> # Xen bits
Acked-by: Kees Cook <keescook@chromium.org>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Acked-by: Andreas Dilger <andreas.dilger@intel.com> # Lustre
Acked-by: Christian Borntraeger <borntraeger@de.ibm.com> # KVM/s390
Acked-by: Dan Williams <dan.j.williams@intel.com> # nvdim
Acked-by: David Sterba <dsterba@suse.com> # btrfs
Acked-by: Ilya Dryomov <idryomov@gmail.com> # Ceph
Acked-by: Tariq Toukan <tariqt@mellanox.com> # mlx4
Acked-by: Leon Romanovsky <leonro@mellanox.com> # mlx5
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Anton Vorontsov <anton@enomsg.org>
Cc: Colin Cross <ccross@android.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>
Cc: Ben Skeggs <bskeggs@redhat.com>
Cc: Kent Overstreet <kent.overstreet@gmail.com>
Cc: Santosh Raspatur <santosh@chelsio.com>
Cc: Hariprasad S <hariprasad@chelsio.com>
Cc: Yishai Hadas <yishaih@mellanox.com>
Cc: Oleg Drokin <oleg.drokin@intel.com>
Cc: "Yan, Zheng" <zyan@redhat.com>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: David Miller <davem@davemloft.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

BUG= chromium:721786 
TEST=compile

Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/509089
Reviewed-by: Douglas Anderson <dianders@chromium.org>
(cherry picked from commit acdb14324ec0edc581bef217a6edc84f04629491)

BUG= chromium:729038 
TEST=compile

Change-Id: I093d657ced5b6e1105829c67b48ada6170c8544e
Reviewed-on: https://chromium-review.googlesource.com/606753
Commit-Ready: Tomasz Figa <tfiga@chromium.org>
Tested-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>

[modify] https://crrev.com/3f9139a047b7303ffd48baec47e0ec940fedc2dc/include/linux/mm.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c382b44e1afdc4903141da88161933cef71dd215

commit c382b44e1afdc4903141da88161933cef71dd215
Author: Michal Hocko <mhocko@suse.com>
Date: Fri Sep 29 18:24:32 2017

UPSTREAM: mm, vmalloc: properly track vmalloc users

__vmalloc_node_flags used to be static inline but this has changed by
"mm: introduce kv[mz]alloc helpers" because kvmalloc_node needs to use
it as well and the code is outside of the vmalloc proper.  I haven't
realized that changing this will lead to a subtle bug though.  The
function is responsible to track the caller as well.  This caller is
then printed by /proc/vmallocinfo.  If __vmalloc_node_flags is not
inline then we would get only direct users of __vmalloc_node_flags as
callers (e.g.  v[mz]alloc) which reduces usefulness of this debugging
feature considerably.  It simply doesn't help to see that the given
range belongs to vmalloc as a caller:

  0xffffc90002c79000-0xffffc90002c7d000   16384 vmalloc+0x16/0x18 pages=3 vmalloc N0=3
  0xffffc90002c81000-0xffffc90002c85000   16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3
  0xffffc90002c8d000-0xffffc90002c91000   16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3
  0xffffc90002c95000-0xffffc90002c99000   16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3

We really want to catch the _caller_ of the vmalloc function.  Fix this
issue by making __vmalloc_node_flags static inline again.

Link: http://lkml.kernel.org/r/20170502134657.12381-1-mhocko@kernel.org
Signed-off-by: Michal Hocko <mhocko@suse.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit 1f5307b1e094bfffa83c65c40ac6e3415c108780)

BUG= chromium:721786 
TEST=compile

Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/512422
Reviewed-by: Douglas Anderson <dianders@chromium.org>
(cherry picked from commit 02efde95af2b753963ec1de00937f7ebb6001bf9)

BUG= chromium:729038 
TEST=compile

Change-Id: I992497279e346cdce8357ab55ada6c6d45d758b1
Reviewed-on: https://chromium-review.googlesource.com/606754
Commit-Ready: Tomasz Figa <tfiga@chromium.org>
Tested-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>

[modify] https://crrev.com/c382b44e1afdc4903141da88161933cef71dd215/mm/vmalloc.c
[modify] https://crrev.com/c382b44e1afdc4903141da88161933cef71dd215/include/linux/vmalloc.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/2414300989160aaabc0ec20913b0be2133d73f9e

commit 2414300989160aaabc0ec20913b0be2133d73f9e
Author: Michal Hocko <mhocko@suse.com>
Date: Fri Sep 29 18:24:33 2017

BACKPORT: mm, vmalloc: fix vmalloc users tracking properly

Commit 1f5307b1e094 ("mm, vmalloc: properly track vmalloc users") has
pulled asm/pgtable.h include dependency to linux/vmalloc.h and that
turned out to be a bad idea for some architectures.  E.g.  m68k fails
with

   In file included from arch/m68k/include/asm/pgtable_mm.h:145:0,
                    from arch/m68k/include/asm/pgtable.h:4,
                    from include/linux/vmalloc.h:9,
                    from arch/m68k/kernel/module.c:9:
   arch/m68k/include/asm/mcf_pgtable.h: In function 'nocache_page':
>> arch/m68k/include/asm/mcf_pgtable.h:339:43: error: 'init_mm' undeclared (first use in this function)
    #define pgd_offset_k(address) pgd_offset(&init_mm, address)

as spotted by kernel build bot. nios2 fails for other reason

  In file included from include/asm-generic/io.h:767:0,
                   from arch/nios2/include/asm/io.h:61,
                   from include/linux/io.h:25,
                   from arch/nios2/include/asm/pgtable.h:18,
                   from include/linux/mm.h:70,
                   from include/linux/pid_namespace.h:6,
                   from include/linux/ptrace.h:9,
                   from arch/nios2/include/uapi/asm/elf.h:23,
                   from arch/nios2/include/asm/elf.h:22,
                   from include/linux/elf.h:4,
                   from include/linux/module.h:15,
                   from init/main.c:16:
  include/linux/vmalloc.h: In function '__vmalloc_node_flags':
  include/linux/vmalloc.h:99:40: error: 'PAGE_KERNEL' undeclared (first use in this function); did you mean 'GFP_KERNEL'?

which is due to the newly added #include <asm/pgtable.h>, which on nios2
includes <linux/io.h> and thus <asm/io.h> and <asm-generic/io.h> which
again includes <linux/vmalloc.h>.

Tweaking that around just turns out a bigger headache than necessary.
This patch reverts 1f5307b1e094 and reimplements the original fix in a
different way.  __vmalloc_node_flags can stay static inline which will
cover vmalloc* functions.  We only have one external user
(kvmalloc_node) and we can export __vmalloc_node_flags_caller and
provide the caller directly.  This is much simpler and it doesn't really
need any games with header files.

[akpm@linux-foundation.org: coding-style fixes]
[mhocko@kernel.org: revert old comment]
  Link: http://lkml.kernel.org/r/20170509211054.GB16325@dhcp22.suse.cz
Fixes: 1f5307b1e094 ("mm, vmalloc: properly track vmalloc users")
Link: http://lkml.kernel.org/r/20170509153702.GR6481@dhcp22.suse.cz
Signed-off-by: Michal Hocko <mhocko@suse.com>
Cc: Tobias Klauser <tklauser@distanz.ch>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit 8594a21cf7a8318baedbedc3fcd2967a17ddeec0)

BUG= chromium:721786 
TEST=compile

Change-Id: Ic7486d53e101995f71b5fc6e43bb588d512610aa
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/512423
Reviewed-by: Douglas Anderson <dianders@chromium.org>
(cherry picked from commit bf63686b65c825db6c923b887b6a8bbfa95854fd)

 Conflicts:
	mm/util.c

BUG= chromium:729038 
TEST=compile

Change-Id: Ic7486d53e101995f71b5fc6e43bb588d512610aa
Reviewed-on: https://chromium-review.googlesource.com/606755
Commit-Ready: Tomasz Figa <tfiga@chromium.org>
Tested-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>

[modify] https://crrev.com/2414300989160aaabc0ec20913b0be2133d73f9e/mm/util.c
[modify] https://crrev.com/2414300989160aaabc0ec20913b0be2133d73f9e/mm/vmalloc.c
[modify] https://crrev.com/2414300989160aaabc0ec20913b0be2133d73f9e/include/linux/vmalloc.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/d8a6732eaca277aff853e38f0057c92b4436812d

commit d8a6732eaca277aff853e38f0057c92b4436812d
Author: Tomasz Figa <tfiga@chromium.org>
Date: Fri Sep 29 18:24:35 2017

CHROMIUM: [media] v4l2-core: Use kvmalloc() for potentially big allocations

There are multiple places where arrays or otherwise variable sized
buffer are allocated through V4L2 core code, including things like
controls, memory pages, staging buffers for ioctls and so on. Such
allocations can potentially require an order > 0 allocation from the
page allocator, which is not guaranteed to be fulfilled and is likely to
fail on a system with severe memory fragmentation (e.g. a system with
very long uptime).

Since the memory being allocated is intended to be used by the CPU
exclusively, we can consider using vmalloc() as a fallback and this is
exactly what the recently merged kvmalloc() helpers do. A kmalloc() call
is still attempted, even for order > 0 allocations, but it is done
with __GFP_NORETRY and __GFP_NOWARN, with expectation of failing if
requested memory is not available instantly. Only then the vmalloc()
fallback is used. This should give us fast and more reliable allocations
even on systems with higher memory pressure and/or more fragmentation,
while still retaining the same performance level on systems not
suffering from such conditions.

While at it, replace explicit array size calculations on changed
allocations with kvmalloc_array().

(An equivalent patch was sent upstream separately due to a big number of
merge conflicts: https://patchwork.linuxtv.org/patch/41547/)

BUG= chromium:721786 
TEST=test_that video_VideoDecodeAccelerator.vp8 on veyron_minnie
TEST=test_that video_VideoEncodeAccelerator.vp8 on veyron_minnie

Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/509090
Reviewed-by: Pawel Osciak <posciak@chromium.org>
(cherry picked from commit 749978e9f613bbee0fbae4078cdfc072355d1e39)

BUG= chromium:729038 
TEST=compile

Change-Id: I690bd14bc15e60187bae6058bc2d7c91dcf232e5
Reviewed-on: https://chromium-review.googlesource.com/606756
Commit-Ready: Tomasz Figa <tfiga@chromium.org>
Tested-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>

[modify] https://crrev.com/d8a6732eaca277aff853e38f0057c92b4436812d/drivers/media/v4l2-core/v4l2-ioctl.c
[modify] https://crrev.com/d8a6732eaca277aff853e38f0057c92b4436812d/drivers/media/v4l2-core/v4l2-subdev.c
[modify] https://crrev.com/d8a6732eaca277aff853e38f0057c92b4436812d/drivers/media/v4l2-core/v4l2-async.c
[modify] https://crrev.com/d8a6732eaca277aff853e38f0057c92b4436812d/drivers/media/v4l2-core/v4l2-ctrls.c
[modify] https://crrev.com/d8a6732eaca277aff853e38f0057c92b4436812d/drivers/media/v4l2-core/v4l2-event.c
[modify] https://crrev.com/d8a6732eaca277aff853e38f0057c92b4436812d/drivers/media/v4l2-core/videobuf2-dma-sg.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/5b3cc1774b21b59bd46302e8904b05af4178f79d

commit 5b3cc1774b21b59bd46302e8904b05af4178f79d
Author: Guenter Roeck <groeck@chromium.org>
Date: Sat May 12 08:44:28 2018

FIXUP: CHROMIUM: [media] v4l2-core: Use kvmalloc() for potentially big allocations

0day reports:

drivers/media/v4l2-core/v4l2-subdev.c:38:12: error: implicit declaration of function 'kvmalloc_array'
drivers/media/v4l2-core/v4l2-subdev.c:49:2: error: implicit declaration of function 'kvfree'

Add the missing include file. Note that the matching upstream commit
does include that file.

BUG= chromium:721786 
TEST=Test build various configurations

Change-Id: I3722f4227361ccc48310a636e219d6f88964a10f
Signed-off-by: Guenter Roeck <groeck@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1055767
Reviewed-by: Tomasz Figa <tfiga@chromium.org>

[modify] https://crrev.com/5b3cc1774b21b59bd46302e8904b05af4178f79d/drivers/media/v4l2-core/v4l2-subdev.c