linux/io_uring
Pavel Begunkov 2f3893437a io_uring: cmpxchg for poll arm refs release
Replace atomically substracting the ownership reference at the end of
arming a poll with a cmpxchg. We try to release ownership by setting 0
assuming that poll_refs didn't change while we were arming. If it did
change, we keep the ownership and use it to queue a tw, which is fully
capable to process all events and (even tolerates spurious wake ups).

It's a bit more elegant as we reduce races b/w setting the cancellation
flag and getting refs with this release, and with that we don't have to
worry about any kinds of underflows. It's not the fastest path for
polling. The performance difference b/w cmpxchg and atomic dec is
usually negligible and it's not the fastest path.

Cc: stable@vger.kernel.org
Fixes: aa43477b04 ("io_uring: poll rework")
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/0c95251624397ea6def568ff040cad2d7926fd51.1668963050.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-11-25 06:54:16 -07:00
..
advise.c io_uring: make io_kiocb_to_cmd() typesafe 2022-08-12 17:01:00 -06:00
advise.h
alloc_cache.h
cancel.c io_uring: add IORING_SETUP_DEFER_TASKRUN 2022-09-21 10:30:42 -06:00
cancel.h
epoll.c io_uring: make io_kiocb_to_cmd() typesafe 2022-08-12 17:01:00 -06:00
epoll.h
fdinfo.c io_uring: fix fdinfo sqe offsets calculation 2022-10-12 16:30:56 -06:00
fdinfo.h
filetable.c
filetable.h io_uring: kill hot path fixed file bitmap debug checks 2022-10-16 17:07:53 -06:00
fs.c io_uring: make io_kiocb_to_cmd() typesafe 2022-08-12 17:01:00 -06:00
fs.h
io_uring.c io_uring: fix multishot accept request leaks 2022-11-17 12:33:33 -07:00
io_uring.h io_uring: fix multishot accept request leaks 2022-11-17 12:33:33 -07:00
io-wq.c io-wq: Fix memory leak in worker creation 2022-10-20 05:48:59 -07:00
io-wq.h
kbuf.c io_uring: check for rollover of buffer ID when providing buffers 2022-11-10 11:07:41 -07:00
kbuf.h io_uring: allow buffer recycling in READV 2022-09-21 10:30:43 -06:00
Makefile
msg_ring.c io_uring/msg_ring: Fix NULL pointer dereference in io_msg_send_fd() 2022-10-19 12:33:33 -07:00
msg_ring.h
net.c io_uring: fix multishot recv request leaks 2022-11-17 12:33:33 -07:00
net.h io_uring/net: zerocopy sendmsg 2022-09-21 13:15:02 -06:00
nop.c
nop.h
notif.c io_uring/notif: Remove the unused function io_notif_complete() 2022-09-05 11:42:39 -06:00
notif.h io_uring/net: simplify zerocopy send user API 2022-09-01 09:13:33 -06:00
opdef.c io_uring/opdef: remove 'audit_skip' from SENDMSG_ZC 2022-10-12 16:30:56 -06:00
opdef.h io_uring: add custom opcode hooks on fail 2022-09-21 13:15:02 -06:00
openclose.c io_uring: make io_kiocb_to_cmd() typesafe 2022-08-12 17:01:00 -06:00
openclose.h
poll.c io_uring: cmpxchg for poll arm refs release 2022-11-25 06:54:16 -07:00
poll.h
refs.h
rsrc.c io_uring: remove FFS_SCM 2022-10-16 17:07:12 -06:00
rsrc.h io_uring: remove FFS_SCM 2022-10-16 17:07:12 -06:00
rw.c io_uring/rw: remove leftover debug statement 2022-10-16 17:24:10 -06:00
rw.h io_uring/rw: don't lose partial IO result on fail 2022-09-21 13:15:02 -06:00
slist.h
splice.c io_uring: make io_kiocb_to_cmd() typesafe 2022-08-12 17:01:00 -06:00
splice.h
sqpoll.c
sqpoll.h
statx.c io_uring: make io_kiocb_to_cmd() typesafe 2022-08-12 17:01:00 -06:00
statx.h
sync.c io_uring: make io_kiocb_to_cmd() typesafe 2022-08-12 17:01:00 -06:00
sync.h
tctx.c io_uring: remove io_register_submitter 2022-10-07 12:25:30 -06:00
tctx.h io_uring: simplify __io_uring_add_tctx_node 2022-10-07 12:25:30 -06:00
timeout.c io_uring: remove unused return from io_disarm_next 2022-09-21 13:15:01 -06:00
timeout.h io_uring: remove unused return from io_disarm_next 2022-09-21 13:15:01 -06:00
uring_cmd.c io_uring: introduce fixed buffer support for io_uring_cmd 2022-09-30 07:50:59 -06:00
uring_cmd.h
xattr.c __io_setxattr(): constify path 2022-09-01 17:39:05 -04:00
xattr.h