linux/io_uring
Jens Axboe de36a15f9a io_uring/io-wq: reduce frequency of acct->lock acquisitions
When we check if we have work to run, we grab the acct lock, check,
drop it, and then return the result. If we do have work to run, then
running the work will again grab acct->lock and get the work item.

This causes us to grab acct->lock more frequently than we need to.
If we have work to do, have io_acct_run_queue() return with the acct
lock still acquired. io_worker_handle_work() is then always invoked
with the acct lock already held.

In a simple test cases that stats files (IORING_OP_STATX always hits
io-wq), we see a nice reduction in locking overhead with this change:

19.32%   -12.55%  [kernel.kallsyms]      [k] __cmpwait_case_32
20.90%   -12.07%  [kernel.kallsyms]      [k] queued_spin_lock_slowpath

Reviewed-by: Hao Xu <howeyxu@tencent.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-08-11 10:36:17 -06:00
..
advise.c io_uring: always go async for unsupported fadvise flags 2023-01-29 15:18:26 -07:00
advise.h
alloc_cache.h io_uring/rsrc: consolidate node caching 2023-04-12 12:09:41 -06:00
cancel.c io_uring/cancel: wire up IORING_ASYNC_CANCEL_OP for sync cancel 2023-07-17 10:05:48 -06:00
cancel.h io_uring/cancel: support opcode based lookup and cancelation 2023-07-17 10:05:48 -06:00
epoll.c io_uring: undeprecate epoll_ctl support 2023-05-26 20:22:41 -06:00
epoll.h
fdinfo.c io_uring/fdinfo: get rid of ref tryget 2023-08-10 10:24:19 -06:00
fdinfo.h
filetable.c io_uring: add helpers to decode the fixed file file_ptr 2023-06-20 09:36:22 -06:00
filetable.h io_uring: add helpers to decode the fixed file file_ptr 2023-06-20 09:36:22 -06:00
fs.c io_uring: for requests that require async, force it 2023-01-29 15:18:26 -07:00
fs.h
io_uring.c io_uring: remove unnecessary forward declaration 2023-08-10 15:01:58 -06:00
io_uring.h io_uring: have io_file_put() take an io_kiocb rather than the file 2023-08-10 10:27:46 -06:00
io-wq.c io_uring/io-wq: reduce frequency of acct->lock acquisitions 2023-08-11 10:36:17 -06:00
io-wq.h
kbuf.c for-6.4/io_uring-2023-04-21 2023-04-26 12:40:31 -07:00
kbuf.h io_uring: add support for user mapped provided buffer ring 2023-04-03 07:14:21 -06:00
Makefile
msg_ring.c io_uring: use io_file_from_index in io_msg_grab_file 2023-06-20 09:36:22 -06:00
msg_ring.h io_uring: get rid of double locking 2022-12-07 06:47:13 -07:00
net.c io_uring-6.5-2023-07-03 2023-07-03 18:43:10 -07:00
net.h io_uring: Add KASAN support for alloc_caches 2023-04-03 07:16:14 -06:00
nop.c
nop.h
notif.c io_uring/notif: add constant for ubuf_info flags 2023-04-15 14:21:04 -06:00
notif.h io_uring/notif: add constant for ubuf_info flags 2023-04-15 14:21:04 -06:00
opdef.c io_uring: Pass whole sqe to commands 2023-05-04 08:19:05 -06:00
opdef.h io_uring: Split io_issue_def struct 2023-01-29 15:17:41 -07:00
openclose.c fsnotify: move fsnotify_open() hook into do_dentry_open() 2023-06-12 10:43:45 +02:00
openclose.h
poll.c io_uring/cancel: support opcode based lookup and cancelation 2023-07-17 10:05:48 -06:00
poll.h io_uring: avoid indirect function calls for the hottest task_work 2023-06-02 08:55:37 -06:00
refs.h
rsrc.c - Yosry Ahmed brought back some cgroup v1 stats in OOM logs. 2023-06-28 10:28:11 -07:00
rsrc.h io_uring/rsrc: Remove unused declaration io_rsrc_put_tw() 2023-08-09 10:46:46 -06:00
rw.c io_uring: return REQ_F_ flags from io_file_get_flags 2023-06-20 09:36:22 -06:00
rw.h io_uring: avoid indirect function calls for the hottest task_work 2023-06-02 08:55:37 -06:00
slist.h io_uring: silence variable ‘prev’ set but not used warning 2023-03-09 10:10:58 -07:00
splice.c io_uring/splice: use fput() directly 2023-08-10 10:24:25 -06:00
splice.h
sqpoll.c io_uring: unlock sqd->lock before sq thread release CPU 2023-05-25 09:30:13 -06:00
sqpoll.h io_uring: make io_sqpoll_wait_sq return void 2023-01-29 15:17:40 -07:00
statx.c io_uring: for requests that require async, force it 2023-01-29 15:18:26 -07:00
statx.h
sync.c io_uring: for requests that require async, force it 2023-01-29 15:18:26 -07:00
sync.h
tctx.c io_uring: Add io_uring_setup flag to pre-register ring fd and never install it 2023-05-16 08:06:00 -06:00
tctx.h io_uring: simplify __io_uring_add_tctx_node 2022-10-07 12:25:30 -06:00
timeout.c io_uring: use cancelation match helper for poll and timeout requests 2023-07-17 10:05:48 -06:00
timeout.h io_uring: remove unused return from io_disarm_next 2022-09-21 13:15:01 -06:00
uring_cmd.c io_uring: Add io_uring command support for sockets 2023-08-09 10:46:15 -06:00
uring_cmd.h io_uring: Remove unnecessary BUILD_BUG_ON 2023-05-04 08:19:05 -06:00
xattr.c io_uring: for requests that require async, force it 2023-01-29 15:18:26 -07:00
xattr.h