iv/linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
linux/kernel/bpf
History
Andrey Ignatov 0640c77c46 bpf: Avoid retpoline for bpf_for_each_map_elem 
Similarly to 09772d92cd5a ("bpf: avoid retpoline for
lookup/update/delete calls on maps") and 84430d4232c3 ("bpf, verifier:
avoid retpoline for map push/pop/peek operation") avoid indirect call
while calling bpf_for_each_map_elem.

Before (a program fragment):

  ; if (rules_map) {
   142: (15) if r4 == 0x0 goto pc+8
   143: (bf) r3 = r10
  ; bpf_for_each_map_elem(rules_map, process_each_rule, &ctx, 0);
   144: (07) r3 += -24
   145: (bf) r1 = r4
   146: (18) r2 = subprog[+5]
   148: (b7) r4 = 0
   149: (85) call bpf_for_each_map_elem#143680  <-- indirect call via
                                                    helper

After (same program fragment):

   ; if (rules_map) {
    142: (15) if r4 == 0x0 goto pc+8
    143: (bf) r3 = r10
   ; bpf_for_each_map_elem(rules_map, process_each_rule, &ctx, 0);
    144: (07) r3 += -24
    145: (bf) r1 = r4
    146: (18) r2 = subprog[+5]
    148: (b7) r4 = 0
    149: (85) call bpf_for_each_array_elem#170336  <-- direct call

On a benchmark that calls bpf_for_each_map_elem() once and does many
other things (mostly checking fields in skb) with CONFIG_RETPOLINE=y it
makes program faster.

Before:

  ============================================================================
  Benchmark.cpp                                              time/iter iters/s
  ============================================================================
  IngressMatchByRemoteEndpoint                                80.78ns 12.38M
  IngressMatchByRemoteIP                                      80.66ns 12.40M
  IngressMatchByRemotePort                                    80.87ns 12.37M

After:

  ============================================================================
  Benchmark.cpp                                              time/iter iters/s
  ============================================================================
  IngressMatchByRemoteEndpoint                                73.49ns 13.61M
  IngressMatchByRemoteIP                                      71.48ns 13.99M
  IngressMatchByRemotePort                                    70.39ns 14.21M

Signed-off-by: Andrey Ignatov <rdna@fb.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20211006001838.75607-1-rdna@fb.com
2021-10-05 19:22:33 -07:00
..
preload
libbpf: Make libbpf_version.h non-auto-generated
2021-09-13 15:36:47 -07:00
arraymap.c
bpf: Replace callers of BPF_CAST_CALL with proper function typedef
2021-09-28 16:27:18 -07:00
bpf_inode_storage.c
bpf: Fix spelling mistakes
2021-05-24 21:13:05 -07:00
bpf_iter.c
bpf: Refactor BPF_PROG_RUN into a function
2021-08-17 00:45:07 +02:00
bpf_local_storage.c
bpf: Prevent deadlock from recursive bpf_task_storage_[get|delete]
2021-02-26 11:51:48 -08:00
bpf_lru_list.c
bpf_lru_list: Read double-checked variable once without lock
2021-02-10 15:54:26 -08:00
bpf_lru_list.h
bpf: Fix a typo "inacitve" -> "inactive"
2020-04-06 21:54:10 +02:00
bpf_lsm.c
Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next
2021-06-17 11:54:56 -07:00
bpf_struct_ops_types.h
bpf: tcp: Support tcp_congestion_ops in bpf
2020-01-09 08:46:18 -08:00
bpf_struct_ops.c
bpf: Handle return value of BPF_PROG_TYPE_STRUCT_OPS prog
2021-09-14 11:09:50 -07:00
bpf_task_storage.c
bpf: Consolidate task_struct BTF_ID declarations
2021-08-25 10:37:05 -07:00
btf.c
bpf: selftests: Add selftests for module kfunc support
2021-10-05 17:07:42 -07:00
cgroup.c
bpf: Migrate cgroup_bpf to internal cgroup_bpf_attach_type enum
2021-08-23 17:50:24 -07:00
core.c
bpf: Introduce BPF support for kernel module function calls
2021-10-05 17:07:41 -07:00
cpumap.c
bpf: cpumap: Implement generic cpumap
2021-07-07 20:01:45 -07:00
devmap.c
bpf, devmap: Exclude XDP broadcast to master device
2021-08-09 23:25:14 +02:00
disasm.c
bpf: Relicense disassembler as GPL-2.0-only OR BSD-2-Clause
2021-09-02 14:49:23 +02:00
disasm.h
bpf: Relicense disassembler as GPL-2.0-only OR BSD-2-Clause
2021-09-02 14:49:23 +02:00
dispatcher.c
bpf: Remove bpf_image tree
2020-03-13 12:49:52 -07:00
hashtab.c
bpf: Replace callers of BPF_CAST_CALL with proper function typedef
2021-09-28 16:27:18 -07:00
helpers.c
bpf: Replace callers of BPF_CAST_CALL with proper function typedef
2021-09-28 16:27:18 -07:00
inode.c
bpf: Fix regression on BPF_OBJ_GET with non-O_RDWR flags
2021-06-22 14:57:43 +02:00
Kconfig
sock_map: Relax config dependency to CONFIG_NET
2021-07-15 18:17:49 -07:00
local_storage.c
bpf: Increase supported cgroup storage value size
2021-07-27 15:59:29 -07:00
lpm_trie.c
bpf: Allow RCU-protected lookups to happen from bh context
2021-06-24 19:41:15 +02:00
Makefile
bpf: Enable task local storage for tracing programs
2021-02-26 11:51:47 -08:00
map_in_map.c
bpf: Remember BTF of inner maps.
2021-07-15 22:31:10 +02:00
map_in_map.h
bpf: Add map_meta_equal map ops
2020-08-28 15:41:30 +02:00
map_iter.c
bpf: Implement link_query callbacks in map element iterators
2020-08-21 14:01:39 -07:00
net_namespace.c
bpf: Add support for forced LINK_DETACH command
2020-08-01 20:38:28 -07:00
offload.c
bpf, offload: Replace bitwise AND by logical AND in bpf_prog_offload_info_fill
2020-02-17 16:53:49 +01:00
percpu_freelist.c
bpf: Use raw_spin_trylock() for pcpu_freelist_push/pop in NMI
2020-10-06 00:04:11 +02:00
percpu_freelist.h
bpf: Use raw_spin_trylock() for pcpu_freelist_push/pop in NMI
2020-10-06 00:04:11 +02:00
prog_iter.c
bpf: Refactor bpf_iter_reg to have separate seq_info member
2020-07-25 20:16:32 -07:00
queue_stack_maps.c
bpf: Eliminate rlimit-based memory accounting for queue_stack_maps maps
2020-12-02 18:32:46 -08:00
reuseport_array.c
bpf: Fix spelling mistakes
2021-05-24 21:13:05 -07:00
ringbuf.c
bpf: Fix false positive kmemleak report in bpf_ringbuf_area_alloc()
2021-06-28 15:57:46 +02:00
stackmap.c
bpf, mm: Fix lockdep warning triggered by stack_map_get_build_id_offset()
2021-09-10 22:24:23 +02:00
syscall.c
bpf: Use kvmalloc for map keys in syscalls
2021-08-20 00:09:49 +02:00
sysfs_btf.c
bpf: Load and verify kernel module BTFs
2020-11-10 15:25:53 -08:00
task_iter.c
bpf: Consolidate task_struct BTF_ID declarations
2021-08-25 10:37:05 -07:00
tnum.c
bpf, tnums: Provably sound, faster, and more precise algorithm for tnum_mul
2021-06-01 13:34:15 +02:00
trampoline.c
bpf: Introduce helper bpf_get_branch_snapshot
2021-09-13 10:53:50 -07:00
verifier.c
bpf: Avoid retpoline for bpf_for_each_map_elem
2021-10-05 19:22:33 -07:00