linux/lib
Linus Torvalds 596ff4a09b cpumask: re-introduce constant-sized cpumask optimizations
Commit aa47a7c215 ("lib/cpumask: deprecate nr_cpumask_bits") resulted
in the cpumask operations potentially becoming hugely less efficient,
because suddenly the cpumask was always considered to be variable-sized.

The optimization was then later added back in a limited form by commit
6f9c07be9d ("lib/cpumask: add FORCE_NR_CPUS config option"), but that
FORCE_NR_CPUS option is not useful in a generic kernel and more of a
special case for embedded situations with fixed hardware.

Instead, just re-introduce the optimization, with some changes.

Instead of depending on CPUMASK_OFFSTACK being false, and then always
using the full constant cpumask width, this introduces three different
cpumask "sizes":

 - the exact size (nr_cpumask_bits) remains identical to nr_cpu_ids.

   This is used for situations where we should use the exact size.

 - the "small" size (small_cpumask_bits) is the NR_CPUS constant if it
   fits in a single word and the bitmap operations thus end up able
   to trigger the "small_const_nbits()" optimizations.

   This is used for the operations that have optimized single-word
   cases that get inlined, notably the bit find and scanning functions.

 - the "large" size (large_cpumask_bits) is the NR_CPUS constant if it
   is an sufficiently small constant that makes simple "copy" and
   "clear" operations more efficient.

   This is arbitrarily set at four words or less.

As a an example of this situation, without this fixed size optimization,
cpumask_clear() will generate code like

        movl    nr_cpu_ids(%rip), %edx
        addq    $63, %rdx
        shrq    $3, %rdx
        andl    $-8, %edx
        callq   memset@PLT

on x86-64, because it would calculate the "exact" number of longwords
that need to be cleared.

In contrast, with this patch, using a MAX_CPU of 64 (which is quite a
reasonable value to use), the above becomes a single

	movq $0,cpumask

instruction instead, because instead of caring to figure out exactly how
many CPU's the system has, it just knows that the cpumask will be a
single word and can just clear it all.

Note that this does end up tightening the rules a bit from the original
version in another way: operations that set bits in the cpumask are now
limited to the actual nr_cpu_ids limit, whereas we used to do the
nr_cpumask_bits thing almost everywhere in the cpumask code.

But if you just clear bits, or scan for bits, we can use the simpler
compile-time constants.

In the process, remove 'cpumask_complement()' and 'for_each_cpu_not()'
which were not useful, and which fundamentally have to be limited to
'nr_cpu_ids'.  Better remove them now than have somebody introduce use
of them later.

Of course, on x86-64 with MAXSMP there is no sane small compile-time
constant for the cpumask sizes, and we end up using the actual CPU bits,
and will generate the above kind of horrors regardless.  Please don't
use MAXSMP unless you really expect to have machines with thousands of
cores.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2023-03-05 14:30:34 -08:00
..
842
crypto crypto: lib/blake2s - Split up test function to halve stack usage 2022-12-30 22:56:27 +08:00
dim dim: initialize all struct fields 2022-05-09 17:20:37 -07:00
fonts lib/fonts: fix undefined behavior in bit shift for get_default_font 2022-11-18 13:55:09 -08:00
kunit kunit: Fix 'hooks.o' build by recursing into kunit 2023-02-27 14:49:03 -08:00
livepatch selftests/livepatch: better synchronize test_klp_callbacks_busy 2022-06-15 10:29:10 +02:00
lz4 lib: make LZ4_decompress_safe_forceExtDict() static 2022-07-17 17:31:39 -07:00
lzo lib/lzo/lzo1x_compress.c: replace ternary operator with min() and min_t() 2022-07-29 18:12:34 -07:00
math math64: favor kernel-doc from header files 2022-11-21 14:30:53 -07:00
mpi lib/mpi: Fix buffer overrun when SG is too long 2023-01-06 17:15:46 +08:00
pldmfw
raid6 for-6.2/block-2022-12-08 2022-12-13 10:43:59 -08:00
reed_solomon treewide: use get_random_u32_below() instead of deprecated function 2022-11-18 02:15:15 +01:00
test_fortify fortify: Detect struct member overflows in memset() at compile-time 2022-02-13 16:50:06 -08:00
vdso lib/vdso: use "grep -E" instead of "egrep" 2022-11-23 19:50:15 +01:00
xz lib/xz, lib/decompress_unxz.c: Fix spelling in comments 2021-10-19 23:44:30 +08:00
zlib_deflate lib/zlib: DFLTCC deflate does not write all available bits for Z_NO_FLUSH 2023-02-27 17:00:14 -08:00
zlib_dfltcc lib/zlib: remove redundation assignement of avail_in dfltcc_gdht() 2023-02-02 22:50:10 -08:00
zlib_inflate lib/zlib: Split deflate and inflate states for DFLTCC 2023-02-02 22:50:09 -08:00
zstd zstd: import usptream v1.5.2 2022-10-24 12:12:32 -07:00
.gitignore bootconfig: Support embedding a bootconfig file in kernel 2022-04-26 17:58:51 -04:00
argv_split.c
ashldi3.c
ashrdi3.c
asn1_decoder.c
asn1_encoder.c lib: remove redundant assignment to variable ret 2022-01-20 08:52:55 +02:00
assoc_array.c assoc_array: Fix BUG_ON during garbage collect 2022-06-01 18:29:06 -07:00
atomic64_test.c
atomic64.c locking/atomic: atomic64: Remove unusable atomic ops 2021-12-13 10:56:09 +01:00
audit.c audit: add support for the openat2 syscall 2021-10-01 16:52:48 -04:00
base64.c lib/base64: RFC4648-compliant base64 encoding 2022-08-02 17:14:47 -06:00
bcd.c
bch.c
bitfield_kunit.c
bitmap.c lib/bitmap: remove bitmap_ord_to_pos 2022-09-26 12:19:12 -07:00
bitrev.c
bootconfig-data.S bootconfig: Support embedding a bootconfig file in kernel 2022-04-26 17:58:51 -04:00
bootconfig.c bootconfig: Support embedding a bootconfig file in kernel 2022-04-26 17:58:51 -04:00
bsearch.c
btree.c lib/btree: simplify btree_{lookup|update} 2022-06-16 19:58:21 -07:00
bucket_locks.c
bug.c cpuidle: lib/bug: Disable rcu_is_watching() during WARN/BUG 2023-01-31 15:01:45 +01:00
build_OID_registry
buildid.c
bust_spinlocks.c kernel/panic: Drop unblank_screen call 2022-09-01 16:55:35 +02:00
check_signature.c
checksum.c
clz_ctz.c
clz_tab.c
cmdline_kunit.c treewide: use get_random_{u8,u16}() when possible, part 1 2022-10-11 17:42:58 -06:00
cmdline.c lib/cmdline: avoid page fault in next_arg 2022-09-11 21:55:06 -07:00
cmpdi2.c
compat_audit.c audit: add support for the openat2 syscall 2021-10-01 16:52:48 -04:00
cpu_rmap.c
cpumask_kunit.c cpumask: re-introduce constant-sized cpumask optimizations 2023-03-05 14:30:34 -08:00
cpumask.c lib/cpumask: update comment for cpumask_local_spread() 2023-02-07 18:20:00 -08:00
crc4.c
crc7.c
crc8.c
crc16.c
crc32.c lib/crc32: Make crc32_be weak for arch override 2022-01-31 11:21:43 +11:00
crc32defs.h
crc32test.c lib/crc32test: correct printed bytes count 2022-01-31 11:21:43 +11:00
crc64-rocksoft.c crypto: add rocksoft 64b crc guard tag framework 2022-03-07 12:48:35 -07:00
crc64.c lib: add rocksoft model crc64 2022-03-07 12:48:35 -07:00
crc-ccitt.c
crc-itu-t.c crc-itu-t: fix typo in CRC ITU-T polynomial comment 2022-06-07 10:27:38 +02:00
crc-t10dif.c
ctype.c
debug_info.c
debug_locks.c
debugobjects.c Non-MM patches for 6.2-rc1. 2022-12-12 17:28:58 -08:00
dec_and_lock.c perf: Fix perf_event_pmu_context serialization 2023-01-31 20:37:18 +01:00
decompress_bunzip2.c
decompress_inflate.c
decompress_unlz4.c
decompress_unlzma.c
decompress_unlzo.c
decompress_unxz.c lib/xz, lib/decompress_unxz.c: Fix spelling in comments 2021-10-19 23:44:30 +08:00
decompress_unzstd.c lib: zstd: Add decompress_sources.h for decompress_unzstd 2021-11-08 16:55:26 -08:00
decompress.c
devmem_is_allowed.c
devres.c devres: remove devm_ioremap_np 2022-09-01 18:04:43 +02:00
dhry_1.c lib: add Dhrystone benchmark test 2023-02-02 22:50:01 -08:00
dhry_2.c lib: add Dhrystone benchmark test 2023-02-02 22:50:01 -08:00
dhry_run.c lib: add Dhrystone benchmark test 2023-02-02 22:50:01 -08:00
dhry.h lib: add Dhrystone benchmark test 2023-02-02 22:50:01 -08:00
digsig.c
dump_stack.c printk: rename cpulock functions 2022-04-22 21:30:57 +02:00
dynamic_debug.c dyndbg: add drm.debug style (drm/parameters/debug) bitmap support 2022-09-07 17:04:49 +02:00
dynamic_queue_limits.c
earlycpio.c lib: move from strlcpy with unused retval to strscpy 2022-09-11 21:55:10 -07:00
errname.c printf: fix errname.c list 2023-02-15 15:44:43 +01:00
error-inject.c error-injection: remove EI_ETYPE_NONE 2023-02-02 22:50:00 -08:00
errseq.c
extable.c
fault-inject-usercopy.c
fault-inject.c fault-injection: make stacktrace filter works as expected 2022-12-15 16:40:44 -08:00
fdt_addresses.c
fdt_empty_tree.c
fdt_ro.c
fdt_rw.c
fdt_strerror.c
fdt_sw.c
fdt_wip.c
fdt.c
find_bit_benchmark.c treewide: use get_random_u32_below() instead of deprecated function 2022-11-18 02:15:15 +01:00
find_bit.c lib/find: introduce find_nth_and_andnot_bit 2023-02-07 18:20:00 -08:00
flex_proportions.c flex_proportions: Disable preemption entering the write section. 2022-09-19 14:35:08 +02:00
fortify_kunit.c kunit/fortify: Validate __alloc_size attribute results 2022-11-22 21:08:28 -08:00
gen_crc32table.c
gen_crc64table.c lib: add rocksoft model crc64 2022-03-07 12:48:35 -07:00
genalloc.c lib/genalloc: use try_cmpxchg in {set,clear}_bits_ll 2023-02-02 22:50:05 -08:00
generic-radix-tree.c
glob.c lib: remove back_str initialization 2022-04-29 14:38:01 -07:00
globtest.c
group_cpus.c genirq/affinity: Only build SMP-only helper functions on SMP kernels 2023-01-18 12:16:47 +01:00
hashtable_test.c lib/hashtable_test.c: add test for the hashtable structure 2023-02-08 14:28:17 -07:00
hexdump.c hex2bin: fix access beyond string end 2022-04-27 10:57:33 -07:00
hweight.c
idr.c ida: don't use BUG_ON() for debugging 2022-07-10 13:55:49 -07:00
inflate.c
interval_tree_test.c
interval_tree.c interval-tree: Add a utility to iterate over spans in an interval tree 2022-11-29 16:34:15 -04:00
iomap_copy.c
iomap.c kmsan: add iomap support 2022-10-03 14:03:21 -07:00
iommu-helper.c
iov_iter.c 46 fs/cifs (smb3 client) changesets, 37 in fs/cifs and 9 for related helper functions and cleanup outside from Dave Howells and Willy 2023-02-22 17:12:44 -08:00
irq_poll.c lib/irq_poll: Prevent softirq pending leak in irq_poll_cpu_dead() 2022-04-13 21:32:21 +02:00
irq_regs.c
is_signed_type_kunit.c lib: assume char is unsigned 2022-11-19 00:56:15 +01:00
is_single_threaded.c
kasprintf.c
Kconfig iommufd for 6.2 2022-12-14 09:15:43 -08:00
Kconfig.debug There is no particular theme here - mainly quick hits all over the tree. 2023-02-23 17:55:40 -08:00
Kconfig.kasan kasan: treat meminstrinsic as builtins in uninstrumented files 2023-03-02 21:54:22 -08:00
Kconfig.kcsan Kernel concurrency sanitizer (KCSAN) updates for v6.3 2023-02-25 13:02:20 -08:00
Kconfig.kfence kfence: allow use of a deferrable timer 2022-03-22 15:57:11 -07:00
Kconfig.kgdb parisc: Convert PDC console to an early console 2022-10-11 12:01:24 +02:00
Kconfig.kmsan kmsan: make sure PREEMPT_RT is off 2022-11-08 15:57:24 -08:00
Kconfig.ubsan ubsan: disable UBSAN_DIV_ZERO for clang 2022-07-14 15:45:26 -07:00
kfifo.c
klist.c
kobject_uevent.c kobject: remove kset from struct kset_uevent_ops callbacks 2021-12-28 11:26:18 +01:00
kobject.c kobject: make dynamic_kobj_ktype and kset_ktype const 2023-02-08 13:34:20 +01:00
kstrtox.c lib/kstrtox.c: add "false"/"true" support to kstrtobool() 2022-05-13 07:20:13 -07:00
kstrtox.h
libcrc32c.c
linear_ranges.c
list_debug.c lib/list_debug.c: Detect uninitialized lists 2022-06-16 19:58:20 -07:00
list_sort.c
list-test.c list: test: Test the hlist structure 2022-04-05 13:32:27 -06:00
llist.c llist: avoid extra memory read in llist_add_batch 2022-11-18 13:55:06 -08:00
locking-selftest-hardirq.h
locking-selftest-mutex.h
locking-selftest-rlock-hardirq.h
locking-selftest-rlock-softirq.h
locking-selftest-rlock.h
locking-selftest-rsem.h
locking-selftest-rtmutex.h
locking-selftest-softirq.h
locking-selftest-spin-hardirq.h
locking-selftest-spin-softirq.h
locking-selftest-spin.h
locking-selftest-wlock-hardirq.h
locking-selftest-wlock-softirq.h
locking-selftest-wlock.h
locking-selftest-wsem.h
locking-selftest.c lockdep/selftests: Adapt ww-tests for PREEMPT_RT 2021-12-04 10:56:24 +01:00
lockref.c lockref: stop doing cpu_relax in the cmpxchg loop 2023-01-13 14:35:38 -06:00
logic_iomem.c lib/logic_iomem: correct fallback config references 2022-03-11 10:42:56 +01:00
logic_pio.c
lru_cache.c lru_cache: remove unused lc_private, lc_set, lc_index_of 2022-11-22 19:38:39 -07:00
lshrdi3.c
Makefile kunit: Fix 'hooks.o' build by recursing into kunit 2023-02-27 14:49:03 -08:00
maple_tree.c maple_tree: reduce stack usage with gcc-9 and earlier 2023-02-16 20:43:55 -08:00
memcat_p.c
memcpy_kunit.c kunit: memcpy: Split slow memcpy tests into MEMCPY_SLOW_KUNIT_TEST 2023-01-25 12:24:40 -08:00
memory-notifier-error-inject.c
memregion.c
memweight.c
muldi3.c
net_utils.c mac_pton: Don't access memory over expected length 2022-11-09 19:28:02 -08:00
netdev-notifier-error-inject.c
nlattr.c netlink: prevent potential spectre v1 gadgets 2023-01-20 17:52:32 -08:00
nmi_backtrace.c x86/nmi: Print reasons why backtrace NMIs are ignored 2023-01-19 15:55:12 -08:00
notifier-error-inject.c lib/notifier-error-inject: fix error when writing -errno to debugfs file 2022-11-30 16:13:16 -08:00
notifier-error-inject.h
objagg.c lib: objagg: Use the bitmap API when applicable 2021-12-24 14:54:29 -08:00
of-reconfig-notifier-error-inject.c
oid_registry.c lib/oid_registry.c: remove redundant assignment to variable num 2022-11-18 13:55:06 -08:00
once.c once: rename _SLOW to _SLEEPABLE 2022-10-03 17:34:32 -07:00
overflow_kunit.c overflow: Introduce overflows_type() and castable_to_type() 2022-11-02 12:39:27 -07:00
packing.c lib: packing: replace bit_reverse() with bitrev8() 2022-12-12 15:06:30 -08:00
parman.c
parser.c lib: parser: update documentation for match_NUMBER functions 2023-03-02 21:54:22 -08:00
pci_iomap.c pci_iounmap'2: Electric Boogaloo: try to make sense of it all 2021-09-19 17:13:35 -07:00
percpu_counter.c lib/percpu_counter: percpu_counter_add_batch() overflow/underflow 2023-02-02 22:50:01 -08:00
percpu_test.c
percpu-refcount.c percpu-refcount: Use call_rcu_hurry() for atomic switch 2022-11-30 13:16:40 -08:00
plist.c
pm-notifier-error-inject.c
polynomial.c lib: add generic polynomial calculation 2022-05-22 11:32:30 -07:00
radix-tree.c lib/radix-tree.c: fix uninitialized variable compilation warning 2022-11-30 16:13:17 -08:00
random32.c treewide: use get_random_bytes() when possible 2022-10-11 17:42:58 -06:00
ratelimit.c ratelimit: Fix data-races in ___ratelimit(). 2022-08-24 13:46:57 +01:00
rbtree_test.c
rbtree.c
ref_tracker.c ref_tracker: remove filter_irq_stacks() call 2022-02-06 11:05:28 +00:00
refcount.c
rhashtable.c rhashtable: Allow rhashtable to be used from irq-safe contexts 2022-12-09 10:42:56 +00:00
sbitmap.c sbitmap: correct wake_batch recalculation to avoid potential IO hung 2023-01-29 20:03:01 -07:00
scatterlist.c lib/scatterlist: Fix to calculate the last_pg properly 2023-01-16 12:08:31 -04:00
seq_buf.c
sg_pool.c lib/sg_pool: change module_init(sg_pool_init) to subsys_initcall 2022-09-23 16:46:19 +02:00
sg_split.c
show_mem.c mm: reduce noise in show_mem for lowmem allocations 2022-09-26 19:46:29 -07:00
siphash_kunit.c siphash: Convert selftest to KUnit 2022-11-01 10:04:52 -07:00
siphash.c SPDX changes for 5.19-rc1 2022-06-03 10:34:34 -07:00
slub_kunit.c linux-kselftest-kunit-next-6.2-rc1 2022-12-12 16:42:57 -08:00
smp_processor_id.c lib/smp_processor_id: fix imbalanced instrumentation_end() call 2022-07-17 17:31:41 -07:00
sort.c lib/sort: Add priv pointer to swap function 2022-03-17 20:17:18 -07:00
stackdepot.c lib/stackdepot: move documentation comments to stackdepot.h 2023-02-16 20:43:52 -08:00
stackinit_kunit.c kernel/range: Uplevel the cxl subsystem's range_contains() helper 2023-02-10 17:32:37 -08:00
stmp_device.c
string_helpers.c lib/string_helpers: Introduce parse_int_array_user() 2022-09-05 14:51:46 +01:00
string.c lib/string: Use strchr() in strpbrk() 2023-01-27 11:42:57 -08:00
strncpy_from_user.c lib/strn*,objtool: Enforce user_access_begin() rules 2022-04-19 21:58:47 +02:00
strnlen_user.c lib/strn*,objtool: Enforce user_access_begin() rules 2022-04-19 21:58:47 +02:00
strscpy_kunit.c fortify: Short-circuit known-safe calls to strscpy() 2022-11-01 10:04:52 -07:00
syscall.c
test_bitmap.c lib/bitmap: add tests for for_each() loops 2022-10-01 10:22:58 -07:00
test_bitops.c
test_bits.c
test_blackhole_dev.c
test_bpf.c net: remove skb->vlan_present 2022-11-11 18:18:05 -08:00
test_debug_virtual.c
test_dynamic_debug.c dyndbg: test DECLARE_DYNDBG_CLASSMAP, sysfs nodes 2022-09-07 17:04:49 +02:00
test_firmware.c test_firmware: Use kstrtobool() instead of strtobool() 2023-01-20 14:09:47 +01:00
test_fprobe.c treewide: use get_random_u32_{above,below}() instead of manual loop 2022-11-18 02:15:22 +01:00
test_fpu.c
test_free_pages.c lib/test_free_pages.c: pass a pointer to virt_to_page() 2022-07-17 17:14:36 -07:00
test_hash.c test_hash.c: refactor into kunit 2022-01-20 08:52:54 +02:00
test_hexdump.c treewide: use get_random_u32_inclusive() when possible 2022-11-18 02:18:02 +01:00
test_hmm_uapi.h hmm-tests: add test for migrate_device_range() 2022-10-12 18:51:50 -07:00
test_hmm.c hmm-tests: add test for migrate_device_range() 2022-10-12 18:51:50 -07:00
test_ida.c
test_kmod.c test_kmod: stop kernel-doc warnings 2023-01-25 14:07:21 -08:00
test_kprobes.c test_kprobes: Add recursed kprobe test case 2023-02-21 08:52:42 +09:00
test_linear_ranges.c lib/test_linear_ranges: Use LINEAR_RANGE() 2022-11-16 13:32:32 +00:00
test_list_sort.c treewide: use get_random_u32_below() instead of deprecated function 2022-11-18 02:15:15 +01:00
test_lockup.c lib/test_lockup: fix kernel pointer check for separate address spaces 2022-02-25 09:36:06 +01:00
test_maple_tree.c test_maple_tree: test modifications while iterating 2023-02-09 16:51:31 -08:00
test_memcat_p.c
test_meminit.c lib/test_meminit: add checks for the allocation functions 2022-10-12 18:51:49 -07:00
test_min_heap.c treewide: use get_random_u32() when possible 2022-10-11 17:42:58 -06:00
test_module.c
test_objagg.c treewide: use get_random_bytes() when possible 2022-10-11 17:42:58 -06:00
test_parman.c
test_printf.c mm: discard __GFP_ATOMIC 2023-02-02 22:33:13 -08:00
test_ref_tracker.c lib: add tests for reference tracker 2021-12-06 16:04:44 -08:00
test_rhashtable.c Networking changes for 6.2. 2022-12-13 15:47:48 -08:00
test_scanf.c lib/test_scanf: split up number parsing test routines 2021-09-06 11:04:03 -07:00
test_sort.c lib/test: convert test_sort.c to use KUnit 2021-09-08 11:50:26 -07:00
test_static_key_base.c
test_static_keys.c
test_string.c lib/test_string.c: add strspn and strcspn tests 2022-04-29 14:38:00 -07:00
test_sysctl.c testing: use the copyleft-next-0.3.1 SPDX tag 2022-11-08 15:44:02 +01:00
test_ubsan.c ubsan: remove CONFIG_UBSAN_OBJECT_SIZE 2022-01-20 08:52:55 +02:00
test_user_copy.c
test_uuid.c
test_vmalloc.c lib/test_vmalloc.c: add parameter use_huge for fix_size_alloc_test 2023-01-18 17:12:41 -08:00
test_xarray.c XArray: Fix xas_create_range() when multi-order entry present 2022-03-28 19:25:11 -04:00
test-kstrtox.c
test-string_helpers.c treewide: use get_random_u32_below() instead of deprecated function 2022-11-18 02:15:15 +01:00
textsearch.c
timerqueue.c
trace_readwrite.c asm-generic/io: Add _RET_IP_ to MMIO trace for more accurate debug info 2022-11-21 22:02:10 +01:00
ts_bm.c lib/ts_bm.c: remove redundant store to variable consumed after addition 2022-07-17 17:31:39 -07:00
ts_fsm.c
ts_kmp.c
ubsan.c hardening updates for v6.3-rc1 2023-02-21 11:07:23 -08:00
ubsan.h arm64: Support Clang UBSAN trap codes for better reporting 2023-02-08 15:26:58 -08:00
ucmpdi2.c
ucs2_string.c
usercopy.c uaccess: Add speculation barrier to copy_from_user() 2023-02-21 14:45:22 -08:00
uuid.c treewide: use get_random_bytes() when possible 2022-10-11 17:42:58 -06:00
vsprintf.c Random number generator updates for Linux 6.2-rc1. 2022-12-12 16:22:22 -08:00
win_minmax.c lib/win_minmax: use /* notation for regular comments 2023-01-11 16:14:21 -08:00
xarray.c mm/huge_memory: Fix xarray node memory leak 2022-06-09 16:24:25 -04:00
xxhash.c