Should we port v4l2 kvmalloc change from v3.14 to v3.18, v4.4 kernels? |
|||||||
Issue descriptionIn bug #721786 we changed v4l2 to use kvmalloc for allocating some bigger buffers. This helps avoid certain OOM cases and in general should make v4l2 behave better in low memory situations. Instead of v4l2 causing things to be paged out or starting compaction (both of which are slow and also impact the rest of the system), when memory is tight v4l2 can just bundle together a few smaller pages. We originally made the change in v3.14 because Sonny pointed out to me that veyron (running v3.14) OOM reports often showed that v4l2 allocations were triggering the OOM. Even though the original report was on 3.14, it seems like we should apply this to other kernels as well. ...by default, I'd tend to suggest v3.18 and v4.4 and skip v3.8 and v3.10. Mostly this is because I'm not aware of many people doing active development on those kernels, so if we accidentally break something it's less likely anyone will notice. --- In any case, since Tomasz has posted his patch upstream and it's been ACKed, presumable we'll get this fix automatically for future kernels. --- What do folks think?
,
Jun 5 2017
Okay, let's try.
,
Aug 9 2017
Took me long enough. 3.18: https://chromium-review.googlesource.com/q/topic:%22v4l2-kvmalloc%22 4.4: https://chromium-review.googlesource.com/q/topic:%22v4l2-kvmalloc-4_4%22
,
Aug 30 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/51227ec5e4bbccd43e8a84c5db12745d0bf8a640 commit 51227ec5e4bbccd43e8a84c5db12745d0bf8a640 Author: Michal Hocko <mhocko@suse.com> Date: Wed Aug 30 20:04:11 2017 BACKPORT: mm: introduce kv[mz]alloc helpers Patch series "kvmalloc", v5. There are many open coded kmalloc with vmalloc fallback instances in the tree. Most of them are not careful enough or simply do not care about the underlying semantic of the kmalloc/page allocator which means that a) some vmalloc fallbacks are basically unreachable because the kmalloc part will keep retrying until it succeeds b) the page allocator can invoke a really disruptive steps like the OOM killer to move forward which doesn't sound appropriate when we consider that the vmalloc fallback is available. As it can be seen implementing kvmalloc requires quite an intimate knowledge if the page allocator and the memory reclaim internals which strongly suggests that a helper should be implemented in the memory subsystem proper. Most callers, I could find, have been converted to use the helper instead. This is patch 6. There are some more relying on __GFP_REPEAT in the networking stack which I have converted as well and Eric Dumazet was not opposed [2] to convert them as well. [1] http://lkml.kernel.org/r/20170130094940.13546-1-mhocko@kernel.org [2] http://lkml.kernel.org/r/1485273626.16328.301.camel@edumazet-glaptop3.roam.corp.google.com This patch (of 9): Using kmalloc with the vmalloc fallback for larger allocations is a common pattern in the kernel code. Yet we do not have any common helper for that and so users have invented their own helpers. Some of them are really creative when doing so. Let's just add kv[mz]alloc and make sure it is implemented properly. This implementation makes sure to not make a large memory pressure for > PAGE_SZE requests (__GFP_NORETRY) and also to not warn about allocation failures. This also rules out the OOM killer as the vmalloc is a more approapriate fallback than a disruptive user visible action. This patch also changes some existing users and removes helpers which are specific for them. In some cases this is not possible (e.g. ext4_kvmalloc, libcfs_kvzalloc) because those seems to be broken and require GFP_NO{FS,IO} context which is not vmalloc compatible in general (note that the page table allocation is GFP_KERNEL). Those need to be fixed separately. While we are at it, document that __vmalloc{_node} about unsupported gfp mask because there seems to be a lot of confusion out there. kvmalloc_node will warn about GFP_KERNEL incompatible (which are not superset) flags to catch new abusers. Existing ones would have to die slowly. [sfr@canb.auug.org.au: f2fs fixup] Link: http://lkml.kernel.org/r/20170320163735.332e64b7@canb.auug.org.au Link: http://lkml.kernel.org/r/20170306103032.2540-2-mhocko@kernel.org Signed-off-by: Michal Hocko <mhocko@suse.com> Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au> Reviewed-by: Andreas Dilger <adilger@dilger.ca> [ext4 part] Acked-by: Vlastimil Babka <vbabka@suse.cz> Cc: John Hubbard <jhubbard@nvidia.com> Cc: David Miller <davem@davemloft.net> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> (cherry picked from commit a7c3e901a46ff54c016d040847eda598a9e3e653) [tfiga: Removed changes for users of the API from the commit. Kept only the new API itself.] BUG= chromium:721786 TEST=compile Signed-off-by: Tomasz Figa <tfiga@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/509088 Reviewed-by: Douglas Anderson <dianders@chromium.org> (cherry picked from commit 3cdaeb0273d8d8ac868325f062edb1af203a3178) BUG= chromium:729038 TEST=compile Change-Id: I6f4ea7da69285f07470a58b1b92970c24af2f365 Reviewed-on: https://chromium-review.googlesource.com/608007 Commit-Ready: Tomasz Figa <tfiga@chromium.org> Tested-by: Tomasz Figa <tfiga@chromium.org> Reviewed-by: Sonny Rao <sonnyrao@chromium.org> [modify] https://crrev.com/51227ec5e4bbccd43e8a84c5db12745d0bf8a640/include/linux/mm.h [modify] https://crrev.com/51227ec5e4bbccd43e8a84c5db12745d0bf8a640/mm/util.c [modify] https://crrev.com/51227ec5e4bbccd43e8a84c5db12745d0bf8a640/mm/vmalloc.c [modify] https://crrev.com/51227ec5e4bbccd43e8a84c5db12745d0bf8a640/mm/nommu.c [modify] https://crrev.com/51227ec5e4bbccd43e8a84c5db12745d0bf8a640/include/linux/vmalloc.h
,
Aug 30 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ddceb3d384580c2c0a70b3e26fc6e45ad434342d commit ddceb3d384580c2c0a70b3e26fc6e45ad434342d Author: Tomasz Figa <tfiga@chromium.org> Date: Wed Aug 30 20:04:12 2017 CHROMIUM: mm: Take kvmalloc_array() from upstream Comes from upstream commit 752ade68cbd8 ("treewide: use kv[mz]alloc* rather than opencoded variants"), which also converted existing code to use it, but it doesn't apply cleanly to 3.14. Original commit message: There are many code paths opencoding kvmalloc. Let's use the helper instead. The main difference to kvmalloc is that those users are usually not considering all the aspects of the memory allocator. E.g. allocation requests <= 32kB (with 4kB pages) are basically never failing and invoke OOM killer to satisfy the allocation. This sounds too disruptive for something that has a reasonable fallback - the vmalloc. On the other hand those requests might fallback to vmalloc even when the memory allocator would succeed after several more reclaim/compaction attempts previously. There is no guarantee something like that happens though. This patch converts many of those places to kv[mz]alloc* helpers because they are more conservative. Link: http://lkml.kernel.org/r/20170306103327.2766-2-mhocko@kernel.org Signed-off-by: Michal Hocko <mhocko@suse.com> Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com> # Xen bits Acked-by: Kees Cook <keescook@chromium.org> Acked-by: Vlastimil Babka <vbabka@suse.cz> Acked-by: Andreas Dilger <andreas.dilger@intel.com> # Lustre Acked-by: Christian Borntraeger <borntraeger@de.ibm.com> # KVM/s390 Acked-by: Dan Williams <dan.j.williams@intel.com> # nvdim Acked-by: David Sterba <dsterba@suse.com> # btrfs Acked-by: Ilya Dryomov <idryomov@gmail.com> # Ceph Acked-by: Tariq Toukan <tariqt@mellanox.com> # mlx4 Acked-by: Leon Romanovsky <leonro@mellanox.com> # mlx5 Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: Anton Vorontsov <anton@enomsg.org> Cc: Colin Cross <ccross@android.com> Cc: Tony Luck <tony.luck@intel.com> Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net> Cc: Ben Skeggs <bskeggs@redhat.com> Cc: Kent Overstreet <kent.overstreet@gmail.com> Cc: Santosh Raspatur <santosh@chelsio.com> Cc: Hariprasad S <hariprasad@chelsio.com> Cc: Yishai Hadas <yishaih@mellanox.com> Cc: Oleg Drokin <oleg.drokin@intel.com> Cc: "Yan, Zheng" <zyan@redhat.com> Cc: Alexander Viro <viro@zeniv.linux.org.uk> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Eric Dumazet <eric.dumazet@gmail.com> Cc: David Miller <davem@davemloft.net> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> BUG= chromium:721786 TEST=compile Signed-off-by: Tomasz Figa <tfiga@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/509089 Reviewed-by: Douglas Anderson <dianders@chromium.org> (cherry picked from commit acdb14324ec0edc581bef217a6edc84f04629491) BUG= chromium:729038 TEST=compile Change-Id: I093d657ced5b6e1105829c67b48ada6170c8544e Reviewed-on: https://chromium-review.googlesource.com/608008 Commit-Ready: Tomasz Figa <tfiga@chromium.org> Tested-by: Tomasz Figa <tfiga@chromium.org> Reviewed-by: Sonny Rao <sonnyrao@chromium.org> [modify] https://crrev.com/ddceb3d384580c2c0a70b3e26fc6e45ad434342d/include/linux/mm.h
,
Aug 30 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/641f4f4717d8e8e8a2f74350e418d8e16a5272c0 commit 641f4f4717d8e8e8a2f74350e418d8e16a5272c0 Author: Michal Hocko <mhocko@suse.com> Date: Wed Aug 30 20:04:13 2017 UPSTREAM: mm, vmalloc: properly track vmalloc users __vmalloc_node_flags used to be static inline but this has changed by "mm: introduce kv[mz]alloc helpers" because kvmalloc_node needs to use it as well and the code is outside of the vmalloc proper. I haven't realized that changing this will lead to a subtle bug though. The function is responsible to track the caller as well. This caller is then printed by /proc/vmallocinfo. If __vmalloc_node_flags is not inline then we would get only direct users of __vmalloc_node_flags as callers (e.g. v[mz]alloc) which reduces usefulness of this debugging feature considerably. It simply doesn't help to see that the given range belongs to vmalloc as a caller: 0xffffc90002c79000-0xffffc90002c7d000 16384 vmalloc+0x16/0x18 pages=3 vmalloc N0=3 0xffffc90002c81000-0xffffc90002c85000 16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3 0xffffc90002c8d000-0xffffc90002c91000 16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3 0xffffc90002c95000-0xffffc90002c99000 16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3 We really want to catch the _caller_ of the vmalloc function. Fix this issue by making __vmalloc_node_flags static inline again. Link: http://lkml.kernel.org/r/20170502134657.12381-1-mhocko@kernel.org Signed-off-by: Michal Hocko <mhocko@suse.com> Cc: Vlastimil Babka <vbabka@suse.cz> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> (cherry picked from commit 1f5307b1e094bfffa83c65c40ac6e3415c108780) BUG= chromium:721786 TEST=compile Signed-off-by: Tomasz Figa <tfiga@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/512422 Reviewed-by: Douglas Anderson <dianders@chromium.org> (cherry picked from commit 02efde95af2b753963ec1de00937f7ebb6001bf9) BUG= chromium:729038 TEST=compile Change-Id: I992497279e346cdce8357ab55ada6c6d45d758b1 Reviewed-on: https://chromium-review.googlesource.com/608009 Commit-Ready: Tomasz Figa <tfiga@chromium.org> Tested-by: Tomasz Figa <tfiga@chromium.org> Reviewed-by: Sonny Rao <sonnyrao@chromium.org> [modify] https://crrev.com/641f4f4717d8e8e8a2f74350e418d8e16a5272c0/mm/vmalloc.c [modify] https://crrev.com/641f4f4717d8e8e8a2f74350e418d8e16a5272c0/include/linux/vmalloc.h
,
Aug 30 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c90cac68a003989d7ba64ea661e1687bff2a7204 commit c90cac68a003989d7ba64ea661e1687bff2a7204 Author: Michal Hocko <mhocko@suse.com> Date: Wed Aug 30 20:04:14 2017 BACKPORT: mm, vmalloc: fix vmalloc users tracking properly Commit 1f5307b1e094 ("mm, vmalloc: properly track vmalloc users") has pulled asm/pgtable.h include dependency to linux/vmalloc.h and that turned out to be a bad idea for some architectures. E.g. m68k fails with In file included from arch/m68k/include/asm/pgtable_mm.h:145:0, from arch/m68k/include/asm/pgtable.h:4, from include/linux/vmalloc.h:9, from arch/m68k/kernel/module.c:9: arch/m68k/include/asm/mcf_pgtable.h: In function 'nocache_page': >> arch/m68k/include/asm/mcf_pgtable.h:339:43: error: 'init_mm' undeclared (first use in this function) #define pgd_offset_k(address) pgd_offset(&init_mm, address) as spotted by kernel build bot. nios2 fails for other reason In file included from include/asm-generic/io.h:767:0, from arch/nios2/include/asm/io.h:61, from include/linux/io.h:25, from arch/nios2/include/asm/pgtable.h:18, from include/linux/mm.h:70, from include/linux/pid_namespace.h:6, from include/linux/ptrace.h:9, from arch/nios2/include/uapi/asm/elf.h:23, from arch/nios2/include/asm/elf.h:22, from include/linux/elf.h:4, from include/linux/module.h:15, from init/main.c:16: include/linux/vmalloc.h: In function '__vmalloc_node_flags': include/linux/vmalloc.h:99:40: error: 'PAGE_KERNEL' undeclared (first use in this function); did you mean 'GFP_KERNEL'? which is due to the newly added #include <asm/pgtable.h>, which on nios2 includes <linux/io.h> and thus <asm/io.h> and <asm-generic/io.h> which again includes <linux/vmalloc.h>. Tweaking that around just turns out a bigger headache than necessary. This patch reverts 1f5307b1e094 and reimplements the original fix in a different way. __vmalloc_node_flags can stay static inline which will cover vmalloc* functions. We only have one external user (kvmalloc_node) and we can export __vmalloc_node_flags_caller and provide the caller directly. This is much simpler and it doesn't really need any games with header files. [akpm@linux-foundation.org: coding-style fixes] [mhocko@kernel.org: revert old comment] Link: http://lkml.kernel.org/r/20170509211054.GB16325@dhcp22.suse.cz Fixes: 1f5307b1e094 ("mm, vmalloc: properly track vmalloc users") Link: http://lkml.kernel.org/r/20170509153702.GR6481@dhcp22.suse.cz Signed-off-by: Michal Hocko <mhocko@suse.com> Cc: Tobias Klauser <tklauser@distanz.ch> Cc: Geert Uytterhoeven <geert@linux-m68k.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> (cherry picked from commit 8594a21cf7a8318baedbedc3fcd2967a17ddeec0) BUG= chromium:721786 TEST=compile Change-Id: Ic7486d53e101995f71b5fc6e43bb588d512610aa Signed-off-by: Tomasz Figa <tfiga@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/512423 Reviewed-by: Douglas Anderson <dianders@chromium.org> (cherry picked from commit bf63686b65c825db6c923b887b6a8bbfa95854fd) Conflicts: mm/util.c BUG= chromium:729038 TEST=compile Change-Id: Ic7486d53e101995f71b5fc6e43bb588d512610aa Reviewed-on: https://chromium-review.googlesource.com/608010 Commit-Ready: Tomasz Figa <tfiga@chromium.org> Tested-by: Tomasz Figa <tfiga@chromium.org> Reviewed-by: Sonny Rao <sonnyrao@chromium.org> [modify] https://crrev.com/c90cac68a003989d7ba64ea661e1687bff2a7204/mm/util.c [modify] https://crrev.com/c90cac68a003989d7ba64ea661e1687bff2a7204/mm/vmalloc.c [modify] https://crrev.com/c90cac68a003989d7ba64ea661e1687bff2a7204/include/linux/vmalloc.h
,
Aug 30 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/52f62342a711809e0ae91124ed94cb9b899d1560 commit 52f62342a711809e0ae91124ed94cb9b899d1560 Author: Tomasz Figa <tfiga@chromium.org> Date: Wed Aug 30 20:04:16 2017 CHROMIUM: [media] v4l2-core: Use kvmalloc() for potentially big allocations There are multiple places where arrays or otherwise variable sized buffer are allocated through V4L2 core code, including things like controls, memory pages, staging buffers for ioctls and so on. Such allocations can potentially require an order > 0 allocation from the page allocator, which is not guaranteed to be fulfilled and is likely to fail on a system with severe memory fragmentation (e.g. a system with very long uptime). Since the memory being allocated is intended to be used by the CPU exclusively, we can consider using vmalloc() as a fallback and this is exactly what the recently merged kvmalloc() helpers do. A kmalloc() call is still attempted, even for order > 0 allocations, but it is done with __GFP_NORETRY and __GFP_NOWARN, with expectation of failing if requested memory is not available instantly. Only then the vmalloc() fallback is used. This should give us fast and more reliable allocations even on systems with higher memory pressure and/or more fragmentation, while still retaining the same performance level on systems not suffering from such conditions. While at it, replace explicit array size calculations on changed allocations with kvmalloc_array(). (An equivalent patch was sent upstream separately due to a big number of merge conflicts: https://patchwork.linuxtv.org/patch/41547/) BUG= chromium:721786 TEST=test_that video_VideoDecodeAccelerator.vp8 on veyron_minnie TEST=test_that video_VideoEncodeAccelerator.vp8 on veyron_minnie Signed-off-by: Tomasz Figa <tfiga@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/509090 Reviewed-by: Pawel Osciak <posciak@chromium.org> (cherry picked from commit 749978e9f613bbee0fbae4078cdfc072355d1e39) BUG= chromium:729038 TEST=compile Change-Id: I690bd14bc15e60187bae6058bc2d7c91dcf232e5 Reviewed-on: https://chromium-review.googlesource.com/608011 Commit-Ready: Tomasz Figa <tfiga@chromium.org> Tested-by: Tomasz Figa <tfiga@chromium.org> Reviewed-by: Sonny Rao <sonnyrao@chromium.org> [modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/videobuf2-dma-contig.c [modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/v4l2-ioctl.c [modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/v4l2-subdev.c [modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/videobuf2-vmalloc.c [modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/v4l2-async.c [modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/v4l2-ctrls.c [modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/v4l2-event.c [modify] https://crrev.com/52f62342a711809e0ae91124ed94cb9b899d1560/drivers/media/v4l2-core/videobuf2-dma-sg.c
,
Sep 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/5597af9e8f7c33dae55657320e4a754ab3d949fe commit 5597af9e8f7c33dae55657320e4a754ab3d949fe Author: Michal Hocko <mhocko@suse.com> Date: Fri Sep 29 18:24:30 2017 BACKPORT: mm: introduce kv[mz]alloc helpers Patch series "kvmalloc", v5. There are many open coded kmalloc with vmalloc fallback instances in the tree. Most of them are not careful enough or simply do not care about the underlying semantic of the kmalloc/page allocator which means that a) some vmalloc fallbacks are basically unreachable because the kmalloc part will keep retrying until it succeeds b) the page allocator can invoke a really disruptive steps like the OOM killer to move forward which doesn't sound appropriate when we consider that the vmalloc fallback is available. As it can be seen implementing kvmalloc requires quite an intimate knowledge if the page allocator and the memory reclaim internals which strongly suggests that a helper should be implemented in the memory subsystem proper. Most callers, I could find, have been converted to use the helper instead. This is patch 6. There are some more relying on __GFP_REPEAT in the networking stack which I have converted as well and Eric Dumazet was not opposed [2] to convert them as well. [1] http://lkml.kernel.org/r/20170130094940.13546-1-mhocko@kernel.org [2] http://lkml.kernel.org/r/1485273626.16328.301.camel@edumazet-glaptop3.roam.corp.google.com This patch (of 9): Using kmalloc with the vmalloc fallback for larger allocations is a common pattern in the kernel code. Yet we do not have any common helper for that and so users have invented their own helpers. Some of them are really creative when doing so. Let's just add kv[mz]alloc and make sure it is implemented properly. This implementation makes sure to not make a large memory pressure for > PAGE_SZE requests (__GFP_NORETRY) and also to not warn about allocation failures. This also rules out the OOM killer as the vmalloc is a more approapriate fallback than a disruptive user visible action. This patch also changes some existing users and removes helpers which are specific for them. In some cases this is not possible (e.g. ext4_kvmalloc, libcfs_kvzalloc) because those seems to be broken and require GFP_NO{FS,IO} context which is not vmalloc compatible in general (note that the page table allocation is GFP_KERNEL). Those need to be fixed separately. While we are at it, document that __vmalloc{_node} about unsupported gfp mask because there seems to be a lot of confusion out there. kvmalloc_node will warn about GFP_KERNEL incompatible (which are not superset) flags to catch new abusers. Existing ones would have to die slowly. [sfr@canb.auug.org.au: f2fs fixup] Link: http://lkml.kernel.org/r/20170320163735.332e64b7@canb.auug.org.au Link: http://lkml.kernel.org/r/20170306103032.2540-2-mhocko@kernel.org Signed-off-by: Michal Hocko <mhocko@suse.com> Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au> Reviewed-by: Andreas Dilger <adilger@dilger.ca> [ext4 part] Acked-by: Vlastimil Babka <vbabka@suse.cz> Cc: John Hubbard <jhubbard@nvidia.com> Cc: David Miller <davem@davemloft.net> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> (cherry picked from commit a7c3e901a46ff54c016d040847eda598a9e3e653) [tfiga: Removed changes for users of the API from the commit. Kept only the new API itself.] BUG= chromium:721786 TEST=compile Signed-off-by: Tomasz Figa <tfiga@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/509088 Reviewed-by: Douglas Anderson <dianders@chromium.org> (cherry picked from commit 3cdaeb0273d8d8ac868325f062edb1af203a3178) BUG= chromium:729038 TEST=compile Change-Id: I6f4ea7da69285f07470a58b1b92970c24af2f365 Reviewed-on: https://chromium-review.googlesource.com/606752 Commit-Ready: Tomasz Figa <tfiga@chromium.org> Tested-by: Tomasz Figa <tfiga@chromium.org> Reviewed-by: Sonny Rao <sonnyrao@chromium.org> [modify] https://crrev.com/5597af9e8f7c33dae55657320e4a754ab3d949fe/include/linux/mm.h [modify] https://crrev.com/5597af9e8f7c33dae55657320e4a754ab3d949fe/mm/util.c [modify] https://crrev.com/5597af9e8f7c33dae55657320e4a754ab3d949fe/mm/vmalloc.c [modify] https://crrev.com/5597af9e8f7c33dae55657320e4a754ab3d949fe/mm/nommu.c [modify] https://crrev.com/5597af9e8f7c33dae55657320e4a754ab3d949fe/include/linux/vmalloc.h
,
Sep 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/3f9139a047b7303ffd48baec47e0ec940fedc2dc commit 3f9139a047b7303ffd48baec47e0ec940fedc2dc Author: Tomasz Figa <tfiga@chromium.org> Date: Fri Sep 29 18:24:31 2017 CHROMIUM: mm: Take kvmalloc_array() from upstream Comes from upstream commit 752ade68cbd8 ("treewide: use kv[mz]alloc* rather than opencoded variants"), which also converted existing code to use it, but it doesn't apply cleanly to 3.14. Original commit message: There are many code paths opencoding kvmalloc. Let's use the helper instead. The main difference to kvmalloc is that those users are usually not considering all the aspects of the memory allocator. E.g. allocation requests <= 32kB (with 4kB pages) are basically never failing and invoke OOM killer to satisfy the allocation. This sounds too disruptive for something that has a reasonable fallback - the vmalloc. On the other hand those requests might fallback to vmalloc even when the memory allocator would succeed after several more reclaim/compaction attempts previously. There is no guarantee something like that happens though. This patch converts many of those places to kv[mz]alloc* helpers because they are more conservative. Link: http://lkml.kernel.org/r/20170306103327.2766-2-mhocko@kernel.org Signed-off-by: Michal Hocko <mhocko@suse.com> Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com> # Xen bits Acked-by: Kees Cook <keescook@chromium.org> Acked-by: Vlastimil Babka <vbabka@suse.cz> Acked-by: Andreas Dilger <andreas.dilger@intel.com> # Lustre Acked-by: Christian Borntraeger <borntraeger@de.ibm.com> # KVM/s390 Acked-by: Dan Williams <dan.j.williams@intel.com> # nvdim Acked-by: David Sterba <dsterba@suse.com> # btrfs Acked-by: Ilya Dryomov <idryomov@gmail.com> # Ceph Acked-by: Tariq Toukan <tariqt@mellanox.com> # mlx4 Acked-by: Leon Romanovsky <leonro@mellanox.com> # mlx5 Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: Anton Vorontsov <anton@enomsg.org> Cc: Colin Cross <ccross@android.com> Cc: Tony Luck <tony.luck@intel.com> Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net> Cc: Ben Skeggs <bskeggs@redhat.com> Cc: Kent Overstreet <kent.overstreet@gmail.com> Cc: Santosh Raspatur <santosh@chelsio.com> Cc: Hariprasad S <hariprasad@chelsio.com> Cc: Yishai Hadas <yishaih@mellanox.com> Cc: Oleg Drokin <oleg.drokin@intel.com> Cc: "Yan, Zheng" <zyan@redhat.com> Cc: Alexander Viro <viro@zeniv.linux.org.uk> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Eric Dumazet <eric.dumazet@gmail.com> Cc: David Miller <davem@davemloft.net> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> BUG= chromium:721786 TEST=compile Signed-off-by: Tomasz Figa <tfiga@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/509089 Reviewed-by: Douglas Anderson <dianders@chromium.org> (cherry picked from commit acdb14324ec0edc581bef217a6edc84f04629491) BUG= chromium:729038 TEST=compile Change-Id: I093d657ced5b6e1105829c67b48ada6170c8544e Reviewed-on: https://chromium-review.googlesource.com/606753 Commit-Ready: Tomasz Figa <tfiga@chromium.org> Tested-by: Tomasz Figa <tfiga@chromium.org> Reviewed-by: Sonny Rao <sonnyrao@chromium.org> [modify] https://crrev.com/3f9139a047b7303ffd48baec47e0ec940fedc2dc/include/linux/mm.h
,
Sep 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c382b44e1afdc4903141da88161933cef71dd215 commit c382b44e1afdc4903141da88161933cef71dd215 Author: Michal Hocko <mhocko@suse.com> Date: Fri Sep 29 18:24:32 2017 UPSTREAM: mm, vmalloc: properly track vmalloc users __vmalloc_node_flags used to be static inline but this has changed by "mm: introduce kv[mz]alloc helpers" because kvmalloc_node needs to use it as well and the code is outside of the vmalloc proper. I haven't realized that changing this will lead to a subtle bug though. The function is responsible to track the caller as well. This caller is then printed by /proc/vmallocinfo. If __vmalloc_node_flags is not inline then we would get only direct users of __vmalloc_node_flags as callers (e.g. v[mz]alloc) which reduces usefulness of this debugging feature considerably. It simply doesn't help to see that the given range belongs to vmalloc as a caller: 0xffffc90002c79000-0xffffc90002c7d000 16384 vmalloc+0x16/0x18 pages=3 vmalloc N0=3 0xffffc90002c81000-0xffffc90002c85000 16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3 0xffffc90002c8d000-0xffffc90002c91000 16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3 0xffffc90002c95000-0xffffc90002c99000 16384 vmalloc+0x16/0x18 pages=3 vmalloc N1=3 We really want to catch the _caller_ of the vmalloc function. Fix this issue by making __vmalloc_node_flags static inline again. Link: http://lkml.kernel.org/r/20170502134657.12381-1-mhocko@kernel.org Signed-off-by: Michal Hocko <mhocko@suse.com> Cc: Vlastimil Babka <vbabka@suse.cz> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> (cherry picked from commit 1f5307b1e094bfffa83c65c40ac6e3415c108780) BUG= chromium:721786 TEST=compile Signed-off-by: Tomasz Figa <tfiga@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/512422 Reviewed-by: Douglas Anderson <dianders@chromium.org> (cherry picked from commit 02efde95af2b753963ec1de00937f7ebb6001bf9) BUG= chromium:729038 TEST=compile Change-Id: I992497279e346cdce8357ab55ada6c6d45d758b1 Reviewed-on: https://chromium-review.googlesource.com/606754 Commit-Ready: Tomasz Figa <tfiga@chromium.org> Tested-by: Tomasz Figa <tfiga@chromium.org> Reviewed-by: Sonny Rao <sonnyrao@chromium.org> [modify] https://crrev.com/c382b44e1afdc4903141da88161933cef71dd215/mm/vmalloc.c [modify] https://crrev.com/c382b44e1afdc4903141da88161933cef71dd215/include/linux/vmalloc.h
,
Sep 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/2414300989160aaabc0ec20913b0be2133d73f9e commit 2414300989160aaabc0ec20913b0be2133d73f9e Author: Michal Hocko <mhocko@suse.com> Date: Fri Sep 29 18:24:33 2017 BACKPORT: mm, vmalloc: fix vmalloc users tracking properly Commit 1f5307b1e094 ("mm, vmalloc: properly track vmalloc users") has pulled asm/pgtable.h include dependency to linux/vmalloc.h and that turned out to be a bad idea for some architectures. E.g. m68k fails with In file included from arch/m68k/include/asm/pgtable_mm.h:145:0, from arch/m68k/include/asm/pgtable.h:4, from include/linux/vmalloc.h:9, from arch/m68k/kernel/module.c:9: arch/m68k/include/asm/mcf_pgtable.h: In function 'nocache_page': >> arch/m68k/include/asm/mcf_pgtable.h:339:43: error: 'init_mm' undeclared (first use in this function) #define pgd_offset_k(address) pgd_offset(&init_mm, address) as spotted by kernel build bot. nios2 fails for other reason In file included from include/asm-generic/io.h:767:0, from arch/nios2/include/asm/io.h:61, from include/linux/io.h:25, from arch/nios2/include/asm/pgtable.h:18, from include/linux/mm.h:70, from include/linux/pid_namespace.h:6, from include/linux/ptrace.h:9, from arch/nios2/include/uapi/asm/elf.h:23, from arch/nios2/include/asm/elf.h:22, from include/linux/elf.h:4, from include/linux/module.h:15, from init/main.c:16: include/linux/vmalloc.h: In function '__vmalloc_node_flags': include/linux/vmalloc.h:99:40: error: 'PAGE_KERNEL' undeclared (first use in this function); did you mean 'GFP_KERNEL'? which is due to the newly added #include <asm/pgtable.h>, which on nios2 includes <linux/io.h> and thus <asm/io.h> and <asm-generic/io.h> which again includes <linux/vmalloc.h>. Tweaking that around just turns out a bigger headache than necessary. This patch reverts 1f5307b1e094 and reimplements the original fix in a different way. __vmalloc_node_flags can stay static inline which will cover vmalloc* functions. We only have one external user (kvmalloc_node) and we can export __vmalloc_node_flags_caller and provide the caller directly. This is much simpler and it doesn't really need any games with header files. [akpm@linux-foundation.org: coding-style fixes] [mhocko@kernel.org: revert old comment] Link: http://lkml.kernel.org/r/20170509211054.GB16325@dhcp22.suse.cz Fixes: 1f5307b1e094 ("mm, vmalloc: properly track vmalloc users") Link: http://lkml.kernel.org/r/20170509153702.GR6481@dhcp22.suse.cz Signed-off-by: Michal Hocko <mhocko@suse.com> Cc: Tobias Klauser <tklauser@distanz.ch> Cc: Geert Uytterhoeven <geert@linux-m68k.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> (cherry picked from commit 8594a21cf7a8318baedbedc3fcd2967a17ddeec0) BUG= chromium:721786 TEST=compile Change-Id: Ic7486d53e101995f71b5fc6e43bb588d512610aa Signed-off-by: Tomasz Figa <tfiga@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/512423 Reviewed-by: Douglas Anderson <dianders@chromium.org> (cherry picked from commit bf63686b65c825db6c923b887b6a8bbfa95854fd) Conflicts: mm/util.c BUG= chromium:729038 TEST=compile Change-Id: Ic7486d53e101995f71b5fc6e43bb588d512610aa Reviewed-on: https://chromium-review.googlesource.com/606755 Commit-Ready: Tomasz Figa <tfiga@chromium.org> Tested-by: Tomasz Figa <tfiga@chromium.org> Reviewed-by: Sonny Rao <sonnyrao@chromium.org> [modify] https://crrev.com/2414300989160aaabc0ec20913b0be2133d73f9e/mm/util.c [modify] https://crrev.com/2414300989160aaabc0ec20913b0be2133d73f9e/mm/vmalloc.c [modify] https://crrev.com/2414300989160aaabc0ec20913b0be2133d73f9e/include/linux/vmalloc.h
,
Sep 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/d8a6732eaca277aff853e38f0057c92b4436812d commit d8a6732eaca277aff853e38f0057c92b4436812d Author: Tomasz Figa <tfiga@chromium.org> Date: Fri Sep 29 18:24:35 2017 CHROMIUM: [media] v4l2-core: Use kvmalloc() for potentially big allocations There are multiple places where arrays or otherwise variable sized buffer are allocated through V4L2 core code, including things like controls, memory pages, staging buffers for ioctls and so on. Such allocations can potentially require an order > 0 allocation from the page allocator, which is not guaranteed to be fulfilled and is likely to fail on a system with severe memory fragmentation (e.g. a system with very long uptime). Since the memory being allocated is intended to be used by the CPU exclusively, we can consider using vmalloc() as a fallback and this is exactly what the recently merged kvmalloc() helpers do. A kmalloc() call is still attempted, even for order > 0 allocations, but it is done with __GFP_NORETRY and __GFP_NOWARN, with expectation of failing if requested memory is not available instantly. Only then the vmalloc() fallback is used. This should give us fast and more reliable allocations even on systems with higher memory pressure and/or more fragmentation, while still retaining the same performance level on systems not suffering from such conditions. While at it, replace explicit array size calculations on changed allocations with kvmalloc_array(). (An equivalent patch was sent upstream separately due to a big number of merge conflicts: https://patchwork.linuxtv.org/patch/41547/) BUG= chromium:721786 TEST=test_that video_VideoDecodeAccelerator.vp8 on veyron_minnie TEST=test_that video_VideoEncodeAccelerator.vp8 on veyron_minnie Signed-off-by: Tomasz Figa <tfiga@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/509090 Reviewed-by: Pawel Osciak <posciak@chromium.org> (cherry picked from commit 749978e9f613bbee0fbae4078cdfc072355d1e39) BUG= chromium:729038 TEST=compile Change-Id: I690bd14bc15e60187bae6058bc2d7c91dcf232e5 Reviewed-on: https://chromium-review.googlesource.com/606756 Commit-Ready: Tomasz Figa <tfiga@chromium.org> Tested-by: Tomasz Figa <tfiga@chromium.org> Reviewed-by: Sonny Rao <sonnyrao@chromium.org> [modify] https://crrev.com/d8a6732eaca277aff853e38f0057c92b4436812d/drivers/media/v4l2-core/v4l2-ioctl.c [modify] https://crrev.com/d8a6732eaca277aff853e38f0057c92b4436812d/drivers/media/v4l2-core/v4l2-subdev.c [modify] https://crrev.com/d8a6732eaca277aff853e38f0057c92b4436812d/drivers/media/v4l2-core/v4l2-async.c [modify] https://crrev.com/d8a6732eaca277aff853e38f0057c92b4436812d/drivers/media/v4l2-core/v4l2-ctrls.c [modify] https://crrev.com/d8a6732eaca277aff853e38f0057c92b4436812d/drivers/media/v4l2-core/v4l2-event.c [modify] https://crrev.com/d8a6732eaca277aff853e38f0057c92b4436812d/drivers/media/v4l2-core/videobuf2-dma-sg.c
,
Sep 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/a8666e9cdfd2bb42a0f1dcee1cf4974383081f66 commit a8666e9cdfd2bb42a0f1dcee1cf4974383081f66 Author: Tomasz Figa <tfiga@chromium.org> Date: Fri Sep 29 18:24:29 2017 CHROMIUM: security/apparmor: Fix naming clash of kvmalloc()/kvzalloc() Looks like AppArmor has its own implementation of kvmalloc() and kvzalloc(). This is causing a naming clash with the generic functions being backported. Since the local versions are implemented in a slightly different way that generic ones, let's rename them to aa_kvmalloc() and aa_kvzalloc() respectively, so that no behavior change is introduced. BUG= chromium:729038 TEST=cbuildbot --remote kevin-release poppy-release lakitu-gpu-release Change-Id: I1e30eaa2c8a1c8f2ebe1323d272594d323d92f3c Signed-off-by: Tomasz Figa <tfiga@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/691735 Reviewed-by: Sonny Rao <sonnyrao@chromium.org> [modify] https://crrev.com/a8666e9cdfd2bb42a0f1dcee1cf4974383081f66/security/apparmor/match.c [modify] https://crrev.com/a8666e9cdfd2bb42a0f1dcee1cf4974383081f66/security/apparmor/include/apparmor.h [modify] https://crrev.com/a8666e9cdfd2bb42a0f1dcee1cf4974383081f66/security/apparmor/apparmorfs.c
,
Sep 30 2017
,
Oct 9 2017
Verified on 9765.81.0, 61.0.3163.120 on minnie by running videos in loop. |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by sonnyrao@chromium.org
, Jun 2 2017