iv/linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

net: optimize ____napi_schedule() to avoid extra NET_RX_SOFTIRQ

Browse Source 
____napi_schedule() adds a napi into current cpu softnet_data poll_list,
then raises NET_RX_SOFTIRQ to make sure net_rx_action() will process it.

Idea of this patch is to not raise NET_RX_SOFTIRQ when being called indirectly
from net_rx_action(), because we can process poll_list from this point,
without going to full softirq loop.

This needs a change in net_rx_action() to make sure we restart
its main loop if sd->poll_list was updated without NET_RX_SOFTIRQ
being raised.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Jason Xing <kernelxing@tencent.com>
Reviewed-by: Jason Xing <kerneljasonxing@gmail.com>
Tested-by: Jason Xing <kerneljasonxing@gmail.com>
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
This commit is contained in:
Eric Dumazet 2023-03-28 23:50:21 +00:00 committed by Paolo Abeni
parent 821eba962d
commit 8b43fd3d1d
1 changed files with 18 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
net/core/dev.c
View File

@ -4360,7 +4360,11 @@ static inline void ____napi_schedule(struct softnet_data *sd,
} }
list_add_tail(&napi->poll_list, &sd->poll_list); list_add_tail(&napi->poll_list, &sd->poll_list);
__raise_softirq_irqoff(NET_RX_SOFTIRQ); /* If not called from net_rx_action()
* we have to raise NET_RX_SOFTIRQ.
*/
if (!sd->in_net_rx_action)
__raise_softirq_irqoff(NET_RX_SOFTIRQ);
} }
#ifdef CONFIG_RPS #ifdef CONFIG_RPS
@ -6648,6 +6652,7 @@ static __latent_entropy void net_rx_action(struct softirq_action *h)
LIST_HEAD(list); LIST_HEAD(list);
LIST_HEAD(repoll); LIST_HEAD(repoll);
start:
sd->in_net_rx_action = true; sd->in_net_rx_action = true;
local_irq_disable(); local_irq_disable();
list_splice_init(&sd->poll_list, &list); list_splice_init(&sd->poll_list, &list);
@ -6659,9 +6664,18 @@ static __latent_entropy void net_rx_action(struct softirq_action *h)
skb_defer_free_flush(sd); skb_defer_free_flush(sd);
if (list_empty(&list)) { if (list_empty(&list)) {
sd->in_net_rx_action = false; if (list_empty(&repoll)) {
if (!sd_has_rps_ipi_waiting(sd) && list_empty(&repoll)) sd->in_net_rx_action = false;
goto end; barrier();
/* We need to check if ____napi_schedule()
* had refilled poll_list while
* sd->in_net_rx_action was true.
*/
if (!list_empty(&sd->poll_list))
goto start;
if (!sd_has_rps_ipi_waiting(sd))
goto end;
}
break; break;
} }