iv/linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
linux/mm
History
Jiexun Wang b2f557a21b mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range() 
I conducted real-time testing and observed that
madvise_cold_or_pageout_pte_range() causes significant latency under
memory pressure, which can be effectively reduced by adding cond_resched()
within the loop.

I tested on the LicheePi 4A board using Cylictest for latency testing and
Ftrace for latency tracing.  The board uses TH1520 processor and has a
memory size of 8GB.  The kernel version is 6.5.0 with the PREEMPT_RT patch
applied.

The script I tested is as follows:

echo wakeup_rt > /sys/kernel/tracing/current_tracer
echo 1 > /sys/kernel/tracing/tracing_on
echo 0 > /sys/kernel/tracing/tracing_max_latency
stress-ng --vm 8 --vm-bytes 2G &
cyclictest --mlockall --smp --priority=99 --distance=0 --duration=30m
echo 0 > /sys/kernel/tracing/tracing_on
cat /sys/kernel/tracing/trace 

The tracing results before modification are as follows:

# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 6.5.0-rt6-r1208-00003-g999d221864bf
# --------------------------------------------------------------------
# latency: 2552 us, #6/6, CPU#3 | (M:preempt_rt VP:0, KP:0, SP:0 HP:0 #P:4)
#    -----------------
#    | task: cyclictest-196 (uid:0 nice:0 policy:1 rt_prio:99)
#    -----------------
#
#                    _--------=> CPU#
#                   / _-------=> irqs-off/BH-disabled
#                  | / _------=> need-resched
#                  || / _-----=> need-resched-lazy
#                  ||| / _----=> hardirq/softirq
#                  |||| / _---=> preempt-depth
#                  ||||| / _--=> preempt-lazy-depth
#                  |||||| / _-=> migrate-disable
#                  ||||||| /     delay
#  cmd     pid     |||||||| time  |   caller
#     \   /        ||||||||  \    |    /
stress-n-206       3dn.h512    2us :      206:120:R   + [003]     196:  0:R cyclictest
stress-n-206       3dn.h512    7us : <stack trace>
 => __ftrace_trace_stack
 => __trace_stack
 => probe_wakeup
 => ttwu_do_activate
 => try_to_wake_up
 => wake_up_process
 => hrtimer_wakeup
 => __hrtimer_run_queues
 => hrtimer_interrupt
 => riscv_timer_interrupt
 => handle_percpu_devid_irq
 => generic_handle_domain_irq
 => riscv_intc_irq
 => handle_riscv_irq
 => do_irq
stress-n-206       3dn.h512    9us#: 0
stress-n-206       3d...3.. 2544us : __schedule
stress-n-206       3d...3.. 2545us :      206:120:R ==> [003]     196:  0:R cyclictest
stress-n-206       3d...3.. 2551us : <stack trace>
 => __ftrace_trace_stack
 => __trace_stack
 => probe_wakeup_sched_switch
 => __schedule
 => preempt_schedule
 => migrate_enable
 => rt_spin_unlock
 => madvise_cold_or_pageout_pte_range
 => walk_pgd_range
 => __walk_page_range
 => walk_page_range
 => madvise_pageout
 => madvise_vma_behavior
 => do_madvise
 => sys_madvise
 => do_trap_ecall_u
 => ret_from_exception

The tracing results after modification are as follows:

# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 6.5.0-rt6-r1208-00004-gca3876fc69a6-dirty
# --------------------------------------------------------------------
# latency: 1689 us, #6/6, CPU#0 | (M:preempt_rt VP:0, KP:0, SP:0 HP:0 #P:4)
#    -----------------
#    | task: cyclictest-217 (uid:0 nice:0 policy:1 rt_prio:99)
#    -----------------
#
#                    _--------=> CPU#
#                   / _-------=> irqs-off/BH-disabled
#                  | / _------=> need-resched
#                  || / _-----=> need-resched-lazy
#                  ||| / _----=> hardirq/softirq
#                  |||| / _---=> preempt-depth
#                  ||||| / _--=> preempt-lazy-depth
#                  |||||| / _-=> migrate-disable
#                  ||||||| /     delay
#  cmd     pid     |||||||| time  |   caller
#     \   /        ||||||||  \    |    /
stress-n-232       0dn.h413    1us+:      232:120:R   + [000]     217:  0:R cyclictest
stress-n-232       0dn.h413   12us : <stack trace>
 => __ftrace_trace_stack
 => __trace_stack
 => probe_wakeup
 => ttwu_do_activate
 => try_to_wake_up
 => wake_up_process
 => hrtimer_wakeup
 => __hrtimer_run_queues
 => hrtimer_interrupt
 => riscv_timer_interrupt
 => handle_percpu_devid_irq
 => generic_handle_domain_irq
 => riscv_intc_irq
 => handle_riscv_irq
 => do_irq
stress-n-232       0dn.h413   19us#: 0
stress-n-232       0d...3.. 1671us : __schedule
stress-n-232       0d...3.. 1676us+:      232:120:R ==> [000]     217:  0:R cyclictest
stress-n-232       0d...3.. 1687us : <stack trace>
 => __ftrace_trace_stack
 => __trace_stack
 => probe_wakeup_sched_switch
 => __schedule
 => preempt_schedule
 => migrate_enable
 => free_unref_page_list
 => release_pages
 => free_pages_and_swap_cache
 => tlb_batch_pages_flush
 => tlb_flush_mmu
 => unmap_page_range
 => unmap_vmas
 => unmap_region
 => do_vmi_align_munmap.constprop.0
 => do_vmi_munmap
 => __vm_munmap
 => sys_munmap
 => do_trap_ecall_u
 => ret_from_exception

After the modification, the cause of maximum latency is no longer
madvise_cold_or_pageout_pte_range(), so this modification can reduce the
latency caused by madvise_cold_or_pageout_pte_range().


Currently the madvise_cold_or_pageout_pte_range() function exhibits
significant latency under memory pressure, which can be effectively
reduced by adding cond_resched() within the loop.

When the batch_count reaches SWAP_CLUSTER_MAX, we reschedule
the task to ensure fairness and avoid long lock holding times.

Link: https://lkml.kernel.org/r/85363861af65fac66c7a98c251906afc0d9c8098.1695291046.git.wangjiexun@tinylab.org
Signed-off-by: Jiexun Wang <wangjiexun@tinylab.org>
Cc: Zhangjin Wu <falcon@tinylab.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2023-12-06 16:12:50 -08:00
..
damon
mm/damon/sysfs-schemes: add timeout for update_schemes_tried_regions
2023-12-06 16:12:48 -08:00
kasan
Many singleton patches against the MM code. The patch series which are
2023-11-02 19:38:47 -10:00
kfence
LoongArch changes for v6.6
2023-09-08 12:16:52 -07:00
kmsan
mm: kmsan: panic on failure to allocate early boot metadata
2023-10-25 16:47:10 -07:00
backing-dev.c
writeback: remove redundant checks for root memcg
2023-08-21 13:37:48 -07:00
balloon_compaction.c
…
bootmem_info.c
bootmem: use kmemleak_free_part_phys in put_page_bootmem
2023-10-25 16:47:13 -07:00
cma_debug.c
…
cma_sysfs.c
mm: cma: make kobj_type structure constant
2023-03-28 16:20:06 -07:00
cma.c
mm/cma: use nth_page() in place of direct struct page manipulation
2023-10-04 10:32:29 -07:00
cma.h
…
compaction.c
mm/compaction: factor out code to test if we should run compaction for target order
2023-10-04 10:32:19 -07:00
debug_page_alloc.c
mm: page_alloc: split out DEBUG_PAGEALLOC
2023-06-09 16:25:23 -07:00
debug_page_ref.c
…
debug_vm_pgtable.c
mm: fix multiple typos in multiple files
2023-10-25 16:47:14 -07:00
debug.c
mm: update validate_mm() to use vma iterator
2023-06-09 16:25:31 -07:00
dmapool_test.c
dmapool: add alloc/free performance test
2023-04-05 19:42:38 -07:00
dmapool.c
dmapool: create/destroy cleanup
2023-06-09 16:25:17 -07:00
early_ioremap.c
mm/early_ioremap.c: improve the execution efficiency of early_ioremap_setup()
2023-06-09 16:25:56 -07:00
fadvise.c
mm: remove unnecessary pagevec includes
2023-06-23 16:59:31 -07:00
fail_page_alloc.c
mm: page_alloc: split out FAIL_PAGE_ALLOC
2023-06-09 16:25:23 -07:00
failslab.c
mm: fix unexpected changes to {failslab|fail_page_alloc}.attr
2022-11-22 18:50:44 -08:00
filemap.c
mm: fix oops when filemap_map_pmd() without prealloc_pte
2023-12-06 16:12:45 -08:00
folio-compat.c
filemap: Add fgf_t typedef
2023-07-24 18:04:30 -04:00
gup_test.c
Merge mm-hotfixes-stable into mm-stable to pick up depended-upon changes.
2023-06-23 16:58:19 -07:00
gup_test.h
mm/gup_test: start/stop/read functionality for PIN LONGTERM test
2022-11-08 17:37:15 -08:00
gup.c
mm/gup: make failure to pin an error if FOLL_NOWAIT not specified
2023-10-18 14:34:15 -07:00
highmem.c
mm: ptep_get() conversion
2023-06-19 16:19:25 -07:00
hmm.c
mm: enable page walking API to lock vmas during the walk
2023-08-21 13:07:20 -07:00
huge_memory.c
mm: fix for negative counter: nr_file_hugepages
2023-11-15 15:30:09 -08:00
hugetlb_cgroup.c
mm, hugetlb: remove HUGETLB_CGROUP_MIN_ORDER
2023-10-18 14:34:17 -07:00
hugetlb_vmemmap.c
hugetlb_vmemmap: use folio argument for hugetlb_vmemmap_* functions
2023-10-25 16:47:08 -07:00
hugetlb_vmemmap.h
mm: hugetlb_vmemmap: fix reference to nonexistent file
2023-10-25 16:47:14 -07:00
hugetlb.c
hugetlb: fix null-ptr-deref in hugetlb_vma_lock_write
2023-12-06 16:12:43 -08:00
hwpoison-inject.c
mm/hwpoison: add __init/__exit annotations to module init/exit funcs
2022-10-03 14:03:05 -07:00
init-mm.c
mm: move dummy_vm_ops out of a header
2023-08-21 13:37:46 -07:00
internal.h
mm: add page_rmappable_folio() wrapper
2023-10-25 16:47:16 -07:00
interval_tree.c
…
io-mapping.c
…
ioremap.c
mm: ioremap: remove unneeded ioremap_allowed and iounmap_allowed
2023-08-18 10:12:36 -07:00
Kconfig
mm/Kconfig: make userfaultfd a menuconfig
2023-12-06 16:12:47 -08:00
Kconfig.debug
mm: page_table_check: Make it dependent on EXCLUSIVE_SYSTEM_RAM
2023-05-29 16:14:28 +01:00
khugepaged.c
As usual, lots of singleton and doubleton patches all over the tree and
2023-11-02 20:53:31 -10:00
kmemleak.c
mm/kmemleak: move set_track_prepare() outside raw_spinlocks
2023-12-06 16:12:44 -08:00
ksm.c
mm: more ptep_get() conversion
2023-11-15 15:30:09 -08:00
list_lru.c
…
maccess.c
mm: Fix copy_from_user_nofault().
2023-04-12 17:36:23 -07:00
madvise.c
mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()
2023-12-06 16:12:50 -08:00
Makefile
mm: vmscan: move shrinker-related code into a separate file
2023-10-04 10:32:23 -07:00
mapping_dirty_helpers.c
mm: fix clean_record_shared_mapping_range kernel-doc
2023-08-24 16:20:30 -07:00
memblock.c
memblock: report failures when memblock_can_resize is not set
2023-11-08 09:40:13 -08:00
memcontrol.c
mm: kmem: properly initialize local objcg variable in current_obj_cgroup()
2023-12-06 16:12:44 -08:00
memfd.c
memfd: drop warning for missing exec-related flags
2023-10-04 10:32:22 -07:00
memory_hotplug.c
mm/memory_hotplug: fix error handling in add_memory_resource()
2023-12-06 16:12:46 -08:00
memory-failure.c
mm: convert DAX lock/unlock page to lock/unlock folio
2023-10-04 10:32:20 -07:00
memory-tiers.c
dax, kmem: calculate abstract distance with general interface
2023-10-16 15:44:39 -07:00
memory.c
mm/memory.c:zap_pte_range() print bad swap entry
2023-12-06 16:12:43 -08:00
mempolicy.c
Many singleton patches against the MM code. The patch series which are
2023-11-02 19:38:47 -10:00
mempool.c
mempool: do not use ksize() for poisoning
2022-11-30 15:58:41 -08:00
memremap.c
mm/memremap.c: fix outdated comment in devm_memremap_pages
2023-02-09 16:51:46 -08:00
memtest.c
mm: memtest: convert to memtest_report_meminfo()
2023-08-21 13:37:47 -07:00
migrate_device.c
Add x86 shadow stack support
2023-08-31 12:20:12 -07:00
migrate.c
mm: migrate: record the mlocked page status to remove unnecessary lru drain
2023-10-25 16:47:14 -07:00
mincore.c
mm: enable page walking API to lock vmas during the walk
2023-08-21 13:07:20 -07:00
mlock.c
mm: mlock: avoid folio_within_range() on KSM pages
2023-10-25 16:47:14 -07:00
mm_init.c
mm: hugetlb: skip initialization of gigantic tail struct pages if freed by HVO
2023-10-04 10:32:30 -07:00
mm_slot.h
mm: introduce common struct mm_slot
2022-10-03 14:02:43 -07:00
mmap_lock.c
…
mmap.c
Many singleton patches against the MM code. The patch series which are
2023-11-02 19:38:47 -10:00
mmu_gather.c
mm: fix kernel-doc warning from tlb_flush_rmaps()
2023-08-24 16:20:30 -07:00
mmu_notifier.c
mmu_notifiers: rename invalidate_range notifier
2023-08-18 10:12:41 -07:00
mmzone.c
mm: remove page_cpupid_xchg_last()
2023-10-25 16:47:13 -07:00
mprotect.c
mm: mprotect: use a folio in change_pte_range()
2023-10-25 16:47:12 -07:00
mremap.c
mm: abstract VMA merge and extend into vma_merge_extend() helper
2023-10-18 14:34:18 -07:00
msync.c
…
nommu.c
Many singleton patches against the MM code. The patch series which are
2023-11-02 19:38:47 -10:00
oom_kill.c
mm/oom_killer: simplify OOM killer info dump helper
2023-10-25 16:47:10 -07:00
page_alloc.c
mm: add page_rmappable_folio() wrapper
2023-10-25 16:47:16 -07:00
page_counter.c
…
page_ext.c
mm/page_ext: move functions around for minor cleanups to page_ext
2023-08-18 10:12:31 -07:00
page_idle.c
mm: page_idle: convert page idle to use a folio
2023-01-18 17:12:52 -08:00
page_io.c
mm: memcg: add THP swap out info for anonymous reclaim
2023-10-04 10:32:27 -07:00
page_isolation.c
mm/hugetlb: get rid of page_hstate()
2023-08-18 10:12:39 -07:00
page_owner.c
mm/page_owner: remove free_ts from page_owner output
2023-10-18 14:34:19 -07:00
page_poison.c
mm/page_poison: remove unused page_ext.h from page_poison
2023-08-21 13:37:30 -07:00
page_reporting.c
mm, treewide: redefine MAX_ORDER sanely
2023-04-05 19:42:46 -07:00
page_reporting.h
…
page_table_check.c
mm: convert page_table_check_pte_set() to page_table_check_ptes_set()
2023-08-24 16:20:18 -07:00
page_vma_mapped.c
mm: correct stale comment of function check_pte
2023-08-18 10:12:13 -07:00
page-writeback.c
filemap: add a per-mapping stable writes flag
2023-11-20 15:05:18 +01:00
pagewalk.c
mm/pagewalk: fix bootstopping regression from extra pte_unmap()
2023-09-02 08:39:21 -07:00
percpu-internal.h
percpu-internal/pcpu_chunk: re-layout pcpu_chunk structure to reduce false sharing
2023-06-19 16:19:29 -07:00
percpu-km.c
…
percpu-stats.c
…
percpu-vm.c
…
percpu.c
Many singleton patches against the MM code. The patch series which are
2023-11-02 19:38:47 -10:00
pgalloc-track.h
…
pgtable-generic.c
mm/pgtable: notes on pte_offset_map[_lock]()
2023-08-18 10:12:25 -07:00
process_vm_access.c
mm/gup: remove unused vmas parameter from pin_user_pages_remote()
2023-06-09 16:25:25 -07:00
ptdump.c
mm: ptdump should use ptep_get_lockless()
2023-06-19 16:19:24 -07:00
readahead.c
vfs: fix readahead(2) on block devices
2023-10-19 11:02:49 +02:00
rmap.c
mm/rmap: convert page_move_anon_rmap() to folio_move_anon_rmap()
2023-10-18 14:34:14 -07:00
rodata_test.c
mm/rodata_test: use PAGE_ALIGNED() helper
2022-10-03 14:03:05 -07:00
secretmem.c
mm/secretmem: use a folio in secretmem_fault()
2023-08-21 13:38:02 -07:00
shmem_quota.c
shmem: Add default quota limit mount options
2023-08-09 09:15:40 +02:00
shmem.c
As usual, lots of singleton and doubleton patches all over the tree and
2023-11-02 20:53:31 -10:00
show_mem.c
mm: refactor si_mem_available()
2023-10-04 10:32:19 -07:00
shrinker_debug.c
mm: shrinker: convert shrinker_rwsem to mutex
2023-10-04 10:32:26 -07:00
shrinker.c
mm: shrinker: convert shrinker_rwsem to mutex
2023-10-04 10:32:26 -07:00
shuffle.c
mm/shuffle: convert module_param_call to module_param_cb
2022-10-03 14:03:07 -07:00
shuffle.h
mm, treewide: redefine MAX_ORDER sanely
2023-04-05 19:42:46 -07:00
slab_common.c
RCU pull request for v6.7
2023-10-30 18:01:41 -10:00
slab.c
Randomized slab caches for kmalloc()
2023-07-18 10:07:47 +02:00
slab.h
mm: kmem: scoped objcg protection
2023-10-25 16:47:11 -07:00
slub.c
mm/slub: refactor calculate_order() and calc_slab_order()
2023-10-02 11:55:47 +02:00
sparse-vmemmap.c
mm/vmemmap: allow architectures to override how vmemmap optimization works
2023-08-18 10:12:53 -07:00
sparse.c
mm/sparse: remove redundant judgments from macro for_each_present_section_nr
2023-08-18 10:12:14 -07:00
swap_cgroup.c
mm: memcontrol: don't allocate cgroup swap arrays when memcg is disabled
2022-10-03 14:03:36 -07:00
swap_slots.c
mm/swap: convert put_swap_page() to put_swap_folio()
2022-10-03 14:02:46 -07:00
swap_state.c
mempolicy: alloc_pages_mpol() for NUMA policy without vma
2023-10-25 16:47:16 -07:00
swap.c
mm: remove references to pagevec
2023-06-23 16:59:30 -07:00
swap.h
mempolicy: alloc_pages_mpol() for NUMA policy without vma
2023-10-25 16:47:16 -07:00
swapfile.c
mm/swap: Convert to use bdev_open_by_dev()
2023-10-28 13:29:19 +02:00
truncate.c
- Some swap cleanups from Ma Wupeng ("fix WARN_ON in add_to_avail_list")
2023-08-29 14:25:26 -07:00
usercopy.c
mm: Fix copy_from_user_nofault().
2023-04-12 17:36:23 -07:00
userfaultfd.c
mm: more ptep_get() conversion
2023-11-15 15:30:09 -08:00
util.c
parisc: fix mmap_base calculation when stack grows upwards
2023-11-15 15:30:09 -08:00
vmalloc.c
mm/vmalloc: fix the unchecked dereference warning in vread_iter()
2023-11-01 12:38:35 -07:00
vmpressure.c
net-memcg: Fix scope of sockmem pressure indicators
2023-08-16 12:21:32 +01:00
vmscan.c
mm: multi-gen LRU: reuse some legacy trace events
2023-10-18 14:34:14 -07:00
vmstat.c
mm: tune PCP high automatically
2023-10-25 16:47:10 -07:00
workingset.c
mm: workingset: dynamically allocate the mm-shadow shrinker
2023-10-04 10:32:24 -07:00
z3fold.c
mm/z3fold: remove obsolete comment for struct z3fold_pool
2023-08-21 13:37:51 -07:00
zbud.c
mm: zswap: remove shrink from zpool interface
2023-06-19 16:19:27 -07:00
zpool.c
mm: zswap: remove shrink from zpool interface
2023-06-19 16:19:27 -07:00
zsmalloc.c
zsmalloc: use copy_page for full page copy
2023-10-18 14:34:16 -07:00
zswap.c
zswap: export compression failure stats
2023-11-01 12:38:35 -07:00