034ff37d34
I didn't really want to do this, but as part of all the other changes to
the user copy loops, I've been looking at this horror.
I tried to clean it up multiple times, but every time I just found more
problems, and the way it's written, it's just too hard to fix them.
For example, the code is written to do quad-word alignment, and will use
regular byte accesses to get to that point. That's fairly simple, but
it means that any initial 8-byte alignment will be done with cached
copies.
However, the code then is very careful to do any 4-byte _tail_ accesses
using an uncached 4-byte write, and that was claimed to be relevant in
commit a82eee742452 ("x86/uaccess/64: Handle the caching of 4-byte
nocache copies properly in __copy_user_nocache()").
So if you do a 4-byte copy using that function, it carefully uses a
4-byte 'movnti' for the destination. But if you were to do a 12-byte
copy that is 4-byte aligned, it would _not_ do a 4-byte 'movnti'
followed by a 8-byte 'movnti' to keep it all uncached.
Instead, it would align the destination to 8 bytes using a
byte-at-a-time loop, and then do a 8-byte 'movnti' for the final 8
bytes.
The main caller that cares is __copy_user_flushcache(), which knows
about this insanity, and has odd cases for it all. But I just can't
deal with looking at this kind of "it does one case right, and another
related case entirely wrong".
And the code really wasn't fixable without hard drugs, which I try to
avoid.
So instead, rewrite it in a form that hopefully not only gets this
right, but is a bit more maintainable. Knock wood.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
77 lines
2.3 KiB
Makefile
77 lines
2.3 KiB
Makefile
# SPDX-License-Identifier: GPL-2.0
|
|
#
|
|
# Makefile for x86 specific library files.
|
|
#
|
|
|
|
# Produces uninteresting flaky coverage.
|
|
KCOV_INSTRUMENT_delay.o := n
|
|
|
|
# KCSAN uses udelay for introducing watchpoint delay; avoid recursion.
|
|
KCSAN_SANITIZE_delay.o := n
|
|
ifdef CONFIG_KCSAN
|
|
# In case KCSAN+lockdep+ftrace are enabled, disable ftrace for delay.o to avoid
|
|
# lockdep -> [other libs] -> KCSAN -> udelay -> ftrace -> lockdep recursion.
|
|
CFLAGS_REMOVE_delay.o = $(CC_FLAGS_FTRACE)
|
|
endif
|
|
|
|
# Early boot use of cmdline; don't instrument it
|
|
ifdef CONFIG_AMD_MEM_ENCRYPT
|
|
KCOV_INSTRUMENT_cmdline.o := n
|
|
KASAN_SANITIZE_cmdline.o := n
|
|
KCSAN_SANITIZE_cmdline.o := n
|
|
|
|
ifdef CONFIG_FUNCTION_TRACER
|
|
CFLAGS_REMOVE_cmdline.o = -pg
|
|
endif
|
|
|
|
CFLAGS_cmdline.o := -fno-stack-protector -fno-jump-tables
|
|
endif
|
|
|
|
inat_tables_script = $(srctree)/arch/x86/tools/gen-insn-attr-x86.awk
|
|
inat_tables_maps = $(srctree)/arch/x86/lib/x86-opcode-map.txt
|
|
quiet_cmd_inat_tables = GEN $@
|
|
cmd_inat_tables = $(AWK) -f $(inat_tables_script) $(inat_tables_maps) > $@
|
|
|
|
$(obj)/inat-tables.c: $(inat_tables_script) $(inat_tables_maps)
|
|
$(call cmd,inat_tables)
|
|
|
|
$(obj)/inat.o: $(obj)/inat-tables.c
|
|
|
|
clean-files := inat-tables.c
|
|
|
|
obj-$(CONFIG_SMP) += msr-smp.o cache-smp.o
|
|
|
|
lib-y := delay.o misc.o cmdline.o cpu.o
|
|
lib-y += usercopy_$(BITS).o usercopy.o getuser.o putuser.o
|
|
lib-y += memcpy_$(BITS).o
|
|
lib-y += pc-conf-reg.o
|
|
lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc.o copy_mc_64.o
|
|
lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o
|
|
lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o
|
|
lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
|
|
lib-$(CONFIG_RETPOLINE) += retpoline.o
|
|
|
|
obj-y += msr.o msr-reg.o msr-reg-export.o hweight.o
|
|
obj-y += iomem.o
|
|
|
|
ifeq ($(CONFIG_X86_32),y)
|
|
obj-y += atomic64_32.o
|
|
lib-y += atomic64_cx8_32.o
|
|
lib-y += checksum_32.o
|
|
lib-y += strstr_32.o
|
|
lib-y += string_32.o
|
|
lib-y += memmove_32.o
|
|
ifneq ($(CONFIG_X86_CMPXCHG64),y)
|
|
lib-y += cmpxchg8b_emu.o atomic64_386_32.o
|
|
endif
|
|
else
|
|
obj-y += iomap_copy_64.o
|
|
ifneq ($(CONFIG_GENERIC_CSUM),y)
|
|
lib-y += csum-partial_64.o csum-copy_64.o csum-wrappers_64.o
|
|
endif
|
|
lib-y += clear_page_64.o copy_page_64.o
|
|
lib-y += memmove_64.o memset_64.o
|
|
lib-y += copy_user_64.o copy_user_uncached_64.o
|
|
lib-y += cmpxchg16b_emu.o
|
|
endif
|