Pavel Begunkov 7cb31c46b9 net: cache for same cpu skb_attempt_defer_free ... Optimise skb_attempt_defer_free() when run by the same CPU the skb was allocated on. Instead of __kfree_skb() -> kmem_cache_free() we can disable softirqs and put the buffer into cpu local caches. CPU bound TCP ping pong style benchmarking (i.e. netbench) showed a 1% throughput increase (392.2 -> 396.4 Krps). Cross checking with profiles, the total CPU share of skb_attempt_defer_free() dropped by 0.6%. Note, I'd expect the win doubled with rx only benchmarks, as the optimisation is for the receive path, but the test spends >55% of CPU doing writes. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Reviewed-by: Eric Dumazet <edumazet@google.com> Link: https://lore.kernel.org/r/a887463fb219d973ec5ad275e31194812571f1f5.1712711977.git.asml.silence@gmail.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>		2024-04-10 19:27:32 -07:00
..
bpf_sk_storage.c	netlink: introduce type-checking attribute iteration	2024-03-29 15:06:02 -07:00
datagram.c	net: remove skb_free_datagram_locked()	2024-03-26 15:37:24 +01:00
dev_addr_lists_test.c	net: fill in MODULE_DESCRIPTION()s under net/core	2023-10-28 11:29:27 +01:00
dev_addr_lists.c
dev_ioctl.c	net: partial revert of the "Make timestamping selectable: series	2023-11-18 18:42:37 -08:00
dev.c	net: phy: Introduce ethernet link topology representation	2024-04-06 18:25:14 +01:00
dev.h	net: move dev_xmit_recursion() helpers to net/core/dev.h	2024-04-01 11:28:31 +01:00
drop_monitor.c	genetlink: Use internal flags for multicast groups	2023-12-29 08:43:59 +00:00
dst_cache.c
dst.c	net: dst: Make dst_destroy() static and return void.	2024-02-06 11:45:53 +01:00
failover.c	net: failover: use IFF_NO_ADDRCONF flag to prevent ipv6 addrconf	2022-12-12 15:18:25 -08:00
fib_notifier.c
fib_rules.c	fib: rules: remove repeated assignment in fib_nl2rule	2024-01-07 15:16:19 +00:00
filter.c	ip_tunnel: convert __be16 tunnel flags to bitmaps	2024-04-01 10:49:28 +01:00
flow_dissector.c	ip_tunnel: convert __be16 tunnel flags to bitmaps	2024-04-01 10:49:28 +01:00
flow_offload.c	tc: flower: Enable offload support IPSEC SPI field.	2023-08-02 10:09:32 +01:00
gen_estimator.c	treewide: Convert del_timer*() to timer_shutdown*()	2022-12-25 13:38:09 -08:00
gen_stats.c
gro_cells.c	net: move netdev_max_backlog to net_hotdata	2024-03-07 21:12:42 -08:00
gro.c	gro: fix ownership transfer	2024-03-29 11:30:44 +00:00
gso.c	net: introduce struct net_hotdata	2024-03-07 21:12:41 -08:00
hotdata.c	net: move dev_rx_weight to net_hotdata	2024-03-07 21:12:42 -08:00
hwbm.c
link_watch.c	net: add netdev_set_operstate() helper	2024-02-14 11:20:13 +00:00
lwt_bpf.c	lwt: Fix return values of BPF xmit ops	2023-08-18 16:05:26 +02:00
lwtunnel.c
Makefile	net: net_test: add tests for IP tunnel flags conversion helpers	2024-04-01 10:49:28 +01:00
neighbour.c	neighbour: Don't let neigh_forced_gc() disable preemption for long	2023-12-08 10:37:43 +00:00
net_namespace.c	net: use synchronize_rcu_expedited in cleanup_net()	2024-02-12 12:17:03 +00:00
net_test.c	pfcp: always set pfcp metadata	2024-04-01 10:49:28 +01:00
net-procfs.c	net: make softnet_data.dropped an atomic_t	2024-04-01 11:28:32 +01:00
net-sysfs.c	net: dqs: use sysfs_emit() in favor of sprintf()	2024-04-08 11:04:08 +01:00
net-sysfs.h
net-traces.c	udp6: add a missing call into udp_fail_queue_rcv_skb tracepoint	2023-07-07 09:16:52 +01:00
netclassid_cgroup.c	cgroup, netclassid: on modifying netclassid in cgroup, only consider the main process.	2023-10-16 16:36:53 -07:00
netdev-genl-gen.c	netdev: add per-queue statistics	2024-03-07 21:13:25 -08:00
netdev-genl-gen.h	netdev: add per-queue statistics	2024-03-07 21:13:25 -08:00
netdev-genl.c	netdev: add queue stat for alloc failures	2024-03-07 21:13:26 -08:00
netevent.c
netpoll.c	netpoll: allocate netdev tracker right away	2023-06-15 08:21:11 +01:00
netprio_cgroup.c
of_net.c	net: Explicitly include correct DT includes	2023-07-27 20:33:16 -07:00
page_pool_priv.h	net: page_pool: report when page pool was destroyed	2023-11-28 15:48:39 +01:00
page_pool_user.c	Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net	2024-03-07 10:29:36 -08:00
page_pool.c	page_pool: try direct bulk recycling	2024-04-02 18:13:49 -07:00
pktgen.c	net: pktgen: Use wait_event_freezable_timeout() for freezable kthread	2023-12-27 14:34:52 +00:00
ptp_classifier.c
request_sock.c	tcp: make sure init the accept_queue's spinlocks once	2024-01-19 21:13:25 -08:00
rtnetlink.c	netlink: introduce type-checking attribute iteration	2024-03-29 15:06:02 -07:00
scm.c	af_unix: Link struct unix_edge when queuing skb.	2024-03-29 08:27:50 -07:00
secure_seq.c
selftests.c	net: fill in MODULE_DESCRIPTION()s under net/core	2023-10-28 11:29:27 +01:00
skbuff.c	net: cache for same cpu skb_attempt_defer_free	2024-04-10 19:27:32 -07:00
skmsg.c	bpf, sockmap: Fix NULL pointer dereference in sk_psock_verdict_data_ready()	2024-02-21 17:15:23 +01:00
sock_destructor.h
sock_diag.c	sock_diag: remove sock_diag_mutex	2024-01-23 15:13:55 +01:00
sock_map.c	bpf, sockmap: Prevent lock inversion deadlock in map delete elem	2024-04-02 16:31:05 +02:00
sock_reuseport.c
sock.c	net: skbuff: generalize the skb->decrypted bit	2024-04-06 17:34:31 +01:00
stream.c	net: Return error from sk_stream_wait_connect() if sk_wait_event() fails	2023-12-15 10:48:51 +00:00
sysctl_net_core.c	net: move rps_sock_flow_table to net_hotdata	2024-03-07 21:12:43 -08:00
timestamping.c	net: partial revert of the "Make timestamping selectable: series	2023-11-18 18:42:37 -08:00
tso.c	net: tso: inline tso_count_descs()	2022-12-12 15:04:39 -08:00
utils.c
xdp.c	net: move skbuff_cache(s) to net_hotdata	2024-03-07 21:12:42 -08:00