iv/linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
linux/mm
History
Johannes Weiner 836adaddc6 mm: page_alloc: fix CMA and HIGHATOMIC landing on the wrong buddy list 
[ Upstream commit 7b086755fb8cdbb6b3e45a1bbddc00e7f9b1dc03 ]

Commit 4b23a68f9536 ("mm/page_alloc: protect PCP lists with a spinlock")
bypasses the pcplist on lock contention and returns the page directly to
the buddy list of the page's migratetype.

For pages that don't have their own pcplist, such as CMA and HIGHATOMIC,
the migratetype is temporarily updated such that the page can hitch a ride
on the MOVABLE pcplist.  Their true type is later reassessed when flushing
in free_pcppages_bulk().  However, when lock contention is detected after
the type was already overridden, the bypass will then put the page on the
wrong buddy list.

Once on the MOVABLE buddy list, the page becomes eligible for fallbacks
and even stealing.  In the case of HIGHATOMIC, otherwise ineligible
allocations can dip into the highatomic reserves.  In the case of CMA, the
page can be lost from the CMA region permanently.

Use a separate pcpmigratetype variable for the pcplist override.  Use the
original migratetype when going directly to the buddy.  This fixes the bug
and should make the intentions more obvious in the code.

Originally sent here to address the HIGHATOMIC case:
https://lore.kernel.org/lkml/20230821183733.106619-4-hannes@cmpxchg.org/

Changelog updated in response to the CMA-specific bug report.

[mgorman@techsingularity.net: updated changelog]
Link: https://lkml.kernel.org/r/20230911181108.GA104295@cmpxchg.org
Fixes: 4b23a68f9536 ("mm/page_alloc: protect PCP lists with a spinlock")
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reported-by: Joe Liu <joe.liu@mediatek.com>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2023-10-10 22:00:36 +02:00
..
damon
mm/damon/vaddr-test: fix memory leak in damon_do_test_apply_three_regions()
2023-10-06 14:57:03 +02:00
kasan
kasan: add kasan_tag_mismatch prototype
2023-07-23 13:49:32 +02:00
kfence
mm: kfence: fix handling discontiguous page
2023-04-13 16:55:30 +02:00
kmsan
mm: kmsan: handle alloc failures in kmsan_vmap_pages_range_noflush()
2023-04-26 14:28:41 +02:00
backing-dev.c
writeback, cgroup: fix null-ptr-deref write in bdi_split_work_to_wbs
2023-04-26 14:28:39 +02:00
balloon_compaction.c
mm: Convert all PageMovable users to movable_operations
2022-08-02 12:34:03 -04:00
bootmem_info.c
bootmem: remove the vmemmap pages from kmemleak in put_page_bootmem
2022-08-28 14:02:45 -07:00
cma_debug.c
mm/cma_debug: show complete cma name in debugfs directories
2022-09-11 20:25:50 -07:00
cma_sysfs.c
…
cma.c
Revert "mm/cma.c: remove redundant cma_mutex lock"
2022-05-13 15:11:26 -07:00
cma.h
…
compaction.c
Revert "mm/compaction: fix set skip in fast_find_migrateblock"
2023-02-01 08:34:49 +01:00
debug_page_ref.c
…
debug_vm_pgtable.c
docs: rename Documentation/vm to Documentation/mm
2022-06-27 12:52:53 -07:00
debug.c
mm: remove the vma linked list
2022-09-26 19:46:26 -07:00
dmapool.c
…
early_ioremap.c
…
fadvise.c
riscv: compat: syscall: Add compat_sys_call_table implementation
2022-04-26 13:36:25 -07:00
failslab.c
mm: fix unexpected changes to {failslab|fail_page_alloc}.attr
2022-11-22 18:50:44 -08:00
filemap.c
mm/filemap: fix page end in filemap_get_read_batch
2023-02-22 12:59:49 +01:00
folio-compat.c
mm: remove try_to_free_swap()
2022-10-03 14:02:53 -07:00
frontswap.c
frontswap: don't call ->init if no ops are registered
2022-09-26 12:14:34 -07:00
gup_test.c
mm: rename is_pinnable_page() to is_longterm_pinnable_page()
2022-07-17 17:14:27 -07:00
gup_test.h
…
gup.c
mm: always expand the stack with the mmap write lock held
2023-07-01 13:16:25 +02:00
highmem.c
highmem: fix kmap_to_page() for kmap_local_page() addresses
2022-10-12 18:51:51 -07:00
hmm.c
mm/swap: add swp_offset_pfn() to fetch PFN from swap entry
2022-09-26 19:46:05 -07:00
huge_memory.c
mm/huge_memory.c: warn with pr_warn_ratelimited instead of VM_WARN_ON_ONCE_FOLIO
2023-04-26 14:28:41 +02:00
hugetlb_cgroup.c
hugetlb_cgroup: use helper for_each_hstate and hstate_index
2022-09-11 20:25:53 -07:00
hugetlb_vmemmap.c
mm: hugetlb_vmemmap: fix a race between vmemmap pmd split
2023-09-19 12:27:56 +02:00
hugetlb_vmemmap.h
mm: hugetlb_vmemmap: improve hugetlb_vmemmap code readability
2022-08-08 18:06:43 -07:00
hugetlb.c
hugetlb: do not clear hugetlb dtor until allocating vmemmap
2023-08-23 17:52:41 +02:00
hwpoison-inject.c
mm/hwpoison: add __init/__exit annotations to module init/exit funcs
2022-10-03 14:03:05 -07:00
init-mm.c
mm: remove rb tree.
2022-09-26 19:46:16 -07:00
internal.h
mm/page_alloc: make boot_nodestats static
2022-10-03 14:03:30 -07:00
interval_tree.c
…
io-mapping.c
…
ioremap.c
mm: ioremap: Add ioremap/iounmap_allowed()
2022-06-27 12:22:31 +01:00
Kconfig
mm: introduce new 'lock_mm_and_find_vma()' page fault helper
2023-07-01 13:16:24 +02:00
Kconfig.debug
mm: page_table_check: Make it dependent on EXCLUSIVE_SYSTEM_RAM
2023-06-14 11:15:29 +02:00
khugepaged.c
mm/khugepaged: check again on anon uffd-wp during isolation
2023-04-26 14:28:41 +02:00
kmemleak.c
mm/kmemleak: prevent soft lockup in kmemleak_scan()'s object iteration loops
2022-10-28 13:37:22 -07:00
ksm.c
mm/ksm: fix race with VMA iteration and mm_struct teardown
2023-03-30 12:49:29 +02:00
list_lru.c
mm: kmem: make mem_cgroup_from_obj() vmalloc()-safe
2022-06-16 19:48:31 -07:00
maccess.c
mm: Fix copy_from_user_nofault().
2023-06-28 11:12:17 +02:00
madvise.c
madvise:madvise_free_pte_range(): don't use mapcount() against large folio for sharing check
2023-08-30 16:11:11 +02:00
Makefile
mm: memcontrol: drop dead CONFIG_MEMCG_SWAP config symbol
2022-10-03 14:03:36 -07:00
mapping_dirty_helpers.c
…
memblock.c
Revert "mm: Always release pages to the buddy allocator in memblock_free_late()."
2023-02-22 12:59:50 +01:00
memcontrol.c
mm, memcg: reconsider kmem.limit_in_bytes deprecation
2023-10-06 14:57:06 +02:00
memfd.c
memfd: check for non-NULL file_seals in memfd_create() syscall
2023-06-28 11:12:27 +02:00
memory_hotplug.c
mm: add pageblock_aligned() macro
2022-10-03 14:03:04 -07:00
memory-failure.c
mm: memory-failure: fix unexpected return value in soft_offline_page()
2023-08-30 16:11:06 +02:00
memory-tiers.c
memory tier: release the new_memtier in find_create_memory_tier()
2023-03-10 09:34:27 +01:00
memory.c
mm/memory: add vm_normal_folio()
2023-10-10 22:00:35 +02:00
mempolicy.c
mm: mempolicy: keep VMA walk if both MPOL_MF_STRICT and MPOL_MF_MOVE are specified
2023-10-10 22:00:35 +02:00
mempool.c
mm/mempool: use might_alloc()
2022-06-16 19:48:30 -07:00
memremap.c
mm/memremap.c: map FS_DAX device memory as decrypted
2022-11-08 15:57:23 -08:00
memtest.c
…
migrate_device.c
mm/migrate_device: return number of migrating pages in args->cpages
2022-11-22 18:50:43 -08:00
migrate.c
mm/migrate: fix wrongly apply write bit after mkdirty on sparc64
2023-02-22 12:59:49 +01:00
mincore.c
mm: teach mincore_hugetlb about pte markers
2023-03-22 13:34:03 +01:00
mlock.c
mm/mlock: drop dead code in count_mm_mlocked_page_nr()
2022-09-26 19:46:27 -07:00
mm_init.c
mm: multi-gen LRU: groundwork
2022-09-26 19:46:09 -07:00
mm_slot.h
mm: introduce common struct mm_slot
2022-10-03 14:02:43 -07:00
mmap_lock.c
…
mmap.c
mm/mmap: Fix extra maple tree write
2023-07-19 16:22:16 +02:00
mmu_gather.c
mm/khugepaged: fix GUP-fast interaction by sending IPI
2022-11-30 14:49:42 -08:00
mmu_notifier.c
mm/mmu_notifier.c: fix race in mmu_interval_notifier_remove()
2022-04-21 20:01:10 -07:00
mmzone.c
mm: multi-gen LRU: groundwork
2022-09-26 19:46:09 -07:00
mprotect.c
mm/uffd: fix warning without PTE_MARKER_UFFD_WP compiled in
2022-10-12 15:56:46 -07:00
mremap.c
mm, mremap: fix mremap() expanding for vma's with vm_ops->close()
2023-02-09 11:28:22 +01:00
msync.c
mm/msync: use vma_find() instead of vma linked list
2022-09-26 19:46:25 -07:00
nommu.c
xtensa: fix lock_mm_and_find_vma in case VMA not found
2023-07-05 18:27:37 +01:00
oom_kill.c
mm: reduce noise in show_mem for lowmem allocations
2022-09-26 19:46:29 -07:00
page_alloc.c
mm: page_alloc: fix CMA and HIGHATOMIC landing on the wrong buddy list
2023-10-10 22:00:36 +02:00
page_counter.c
mm: page_counter: remove unneeded atomic ops for low/min
2022-09-11 20:26:01 -07:00
page_ext.c
mm/page_exit: fix kernel doc warning in page_ext_put()
2022-11-22 18:50:41 -08:00
page_idle.c
mm: don't be stuck to rmap lock on reclaim path
2022-05-19 14:08:54 -07:00
page_io.c
use less confusing names for iov_iter direction initializers
2023-02-09 11:28:04 +01:00
page_isolation.c
mm/page_isolation: fix clang deadcode warning
2022-10-28 13:37:22 -07:00
page_owner.c
mm: reuse pageblock_start/end_pfn() macro
2022-10-03 14:03:03 -07:00
page_poison.c
…
page_reporting.c
…
page_reporting.h
…
page_table_check.c
mm: page_table_check: Ensure user pages are not slab pages
2023-06-14 11:15:29 +02:00
page_vma_mapped.c
mm/swap: add swp_offset_pfn() to fetch PFN from swap entry
2022-09-26 19:46:05 -07:00
page-writeback.c
mm: export balance_dirty_pages_ratelimited_flags()
2022-09-26 12:28:07 +02:00
pagewalk.c
- Yu Zhao's Multi-Gen LRU patches are here. They've been under test in
2022-10-10 17:53:04 -07:00
percpu-internal.h
percpu: improve percpu_alloc_percpu event trace
2022-05-13 07:20:18 -07:00
percpu-km.c
…
percpu-stats.c
…
percpu-vm.c
…
percpu.c
mm: percpu: use kmemleak_ignore_phys() instead of kmemleak_free()
2022-07-17 17:14:47 -07:00
pgalloc-track.h
…
pgtable-generic.c
mm: avoid unnecessary flush on change_huge_pmd()
2022-05-13 07:20:05 -07:00
process_vm_access.c
use less confusing names for iov_iter direction initializers
2023-02-09 11:28:04 +01:00
ptdump.c
mm: pagewalk: Fix race between unmap and page walker
2022-09-03 10:13:13 -07:00
readahead.c
mm: add PSI accounting around ->read_folio and ->readahead calls
2022-09-20 08:24:38 -06:00
rmap.c
mm/hwpoison: convert TTU_IGNORE_HWPOISON to TTU_HWPOISON
2023-03-10 09:34:25 +01:00
rodata_test.c
mm/rodata_test: use PAGE_ALIGNED() helper
2022-10-03 14:03:05 -07:00
secretmem.c
mm/secretmem: remove reduntant return value
2022-10-03 14:03:36 -07:00
shmem.c
tmpfs: verify {g,u}id mount options correctly
2023-09-13 09:42:27 +02:00
shrinker_debug.c
mm: shrinkers: fix deadlock in shrinker debugfs
2023-02-22 12:59:46 +01:00
shuffle.c
mm/shuffle: convert module_param_call to module_param_cb
2022-10-03 14:03:07 -07:00
shuffle.h
…
slab_common.c
mm/slab_common: fix slab_caches list corruption after kmem_cache_destroy()
2023-10-06 14:57:03 +02:00
slab.c
mm/slab: Fix undefined init_cache_node_node() for NUMA and !SMP
2023-03-30 12:49:23 +02:00
slab.h
- Yu Zhao's Multi-Gen LRU patches are here. They've been under test in
2022-10-10 17:53:04 -07:00
slob.c
Merge branch 'slab/for-6.1/kmalloc_size_roundup' into slab/for-next
2022-09-29 11:30:55 +02:00
slub.c
treewide: use prandom_u32_max() when possible, part 1
2022-10-11 17:42:55 -06:00
sparse-vmemmap.c
mm: hugetlb_vmemmap: move vmemmap code related to HugeTLB to hugetlb_vmemmap.c
2022-08-08 18:06:42 -07:00
sparse.c
mm: memory_hotplug: enumerate all supported section flags
2022-07-03 18:08:49 -07:00
swap_cgroup.c
mm: memcontrol: don't allocate cgroup swap arrays when memcg is disabled
2022-10-03 14:03:36 -07:00
swap_slots.c
mm/swap: convert put_swap_page() to put_swap_folio()
2022-10-03 14:02:46 -07:00
swap_state.c
swap_state: convert free_swap_cache() to use a folio
2022-10-03 14:02:51 -07:00
swap.c
mm: add folio_add_lru_vma()
2022-10-03 14:02:45 -07:00
swap.h
mm: remove lookup_swap_cache()
2022-10-03 14:02:51 -07:00
swapfile.c
mm/swap: fix swap_info_struct race between swapoff and get_swap_pages()
2023-04-13 16:55:36 +02:00
truncate.c
mm: add split_folio()
2022-10-03 14:02:45 -07:00
usercopy.c
mm: Fix copy_from_user_nofault().
2023-06-28 11:12:17 +02:00
userfaultfd.c
mm/shmem: use page_mapping() to detect page cache for uffd continue
2022-11-08 15:57:23 -08:00
util.c
rcu: dump vmalloc memory info safely
2023-09-13 09:42:59 +02:00
vmalloc.c
mm/vmalloc: add a safer version of find_vm_area() for debug
2023-09-13 09:43:00 +02:00
vmpressure.c
net-memcg: Fix scope of sockmem pressure indicators
2023-09-13 09:42:33 +02:00
vmscan.c
Multi-gen LRU: avoid race in inc_min_seq()
2023-09-19 12:28:04 +02:00
vmstat.c
- Yu Zhao's Multi-Gen LRU patches are here. They've been under test in
2022-10-10 17:53:04 -07:00
workingset.c
mm: multi-gen LRU: minimal implementation
2022-09-26 19:46:09 -07:00
z3fold.c
mm: Convert all PageMovable users to movable_operations
2022-08-02 12:34:03 -04:00
zbud.c
…
zpool.c
…
zsmalloc.c
zsmalloc: allow only one active pool compaction context
2023-08-23 17:52:40 +02:00
zswap.c
zswap: do not shrink if cgroup may not zswap
2023-06-21 16:00:54 +02:00