linux/kernel/bpf
Jesper Dangaard Brouer 86d231459d bpf: cpumap memory prefetchw optimizations for struct page
A lot of the performance gain comes from this patch.

While analysing performance overhead it was found that the largest CPU
stalls were caused when touching the struct page area. It is first read with
a READ_ONCE from build_skb_around via page_is_pfmemalloc(), and when freed
written by page_frag_free() call.

Measurements show that the prefetchw (W) variant operation is needed to
achieve the performance gain. We believe this optimization it two fold,
first the W-variant saves one step in the cache-coherency protocol, and
second it helps us to avoid the non-temporal prefetch HW optimizations and
bring this into all cache-levels. It might be worth investigating if
prefetch into L2 will have the same benefit.

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Acked-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Acked-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2019-04-17 19:09:25 -07:00
..
arraymap.c bpf: allow for key-less BTF in array map 2019-04-09 17:05:46 -07:00
bpf_lru_list.c
bpf_lru_list.h bpf: Only set node->ref = 1 if it has not been set 2017-09-01 09:57:39 -07:00
btf.c bpf: allow for key-less BTF in array map 2019-04-09 17:05:46 -07:00
cgroup.c bpf: add map helper functions push, pop, peek in more BPF programs 2019-04-16 10:24:02 +02:00
core.c bpf: implement lookup-free direct value access for maps 2019-04-09 17:05:46 -07:00
cpumap.c bpf: cpumap memory prefetchw optimizations for struct page 2019-04-17 19:09:25 -07:00
devmap.c bpf: devmap: fix wrong interface selection in notifier_call 2018-10-26 00:32:21 +02:00
disasm.c bpf: implement lookup-free direct value access for maps 2019-04-09 17:05:46 -07:00
disasm.h bpf: Remove struct bpf_verifier_env argument from print_bpf_insn 2018-03-23 17:38:57 +01:00
hashtab.c bpf: add program side {rd, wr}only support for maps 2019-04-09 17:05:46 -07:00
helpers.c bpf: Introduce bpf_strtol and bpf_strtoul helpers 2019-04-12 13:54:59 -07:00
inode.c bpf: fix use after free in bpf_evict_inode 2019-03-26 01:38:49 +01:00
local_storage.c bpf: add program side {rd, wr}only support for maps 2019-04-09 17:05:46 -07:00
lpm_trie.c bpf: add program side {rd, wr}only support for maps 2019-04-09 17:05:46 -07:00
Makefile bpf: add queue and stack maps 2018-10-19 13:24:31 -07:00
map_in_map.c bpf: set inner_map_meta->spin_lock_off correctly 2019-02-27 17:03:13 -08:00
map_in_map.h
offload.c bpf: offload: add priv field for drivers 2019-02-12 17:07:09 +01:00
percpu_freelist.c bpf: fix lockdep false positive in percpu_freelist 2019-01-31 23:18:21 +01:00
percpu_freelist.h bpf: fix lockdep false positive in percpu_freelist 2019-01-31 23:18:21 +01:00
queue_stack_maps.c bpf: add program side {rd, wr}only support for maps 2019-04-09 17:05:46 -07:00
reuseport_array.c bpf: Introduce BPF_MAP_TYPE_REUSEPORT_SOCKARRAY 2018-08-11 01:58:46 +02:00
stackmap.c bpf: fix lockdep false positive in stackmap 2019-02-11 16:36:24 +01:00
syscall.c bpf: Sysctl hook 2019-04-12 13:54:58 -07:00
tnum.c bpf/verifier: improve register value range tracking with ARSH 2018-04-29 08:45:53 -07:00
verifier.c bpf: use BPF_CAST_CALL for casting bpf call 2019-04-16 19:28:31 -07:00
xskmap.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net 2018-10-19 11:03:06 -07:00