linux/fs/afs
David Howells 52bf9f6c09 afs: Fix refcount underflow from error handling race
If an AFS cell that has an unreachable (eg. ENETUNREACH) server listed (VL
server or fileserver), an asynchronous probe to one of its addresses may
fail immediately because sendmsg() returns an error.  When this happens, a
refcount underflow can happen if certain events hit a very small window.

The way this occurs is:

 (1) There are two levels of "call" object, the afs_call and the
     rxrpc_call.  Each of them can be transitioned to a "completed" state
     in the event of success or failure.

 (2) Asynchronous afs_calls are self-referential whilst they are active to
     prevent them from evaporating when they're not being processed.  This
     reference is disposed of when the afs_call is completed.

     Note that an afs_call may only be completed once; once completed
     completing it again will do nothing.

 (3) When a call transmission is made, the app-side rxrpc code queues a Tx
     buffer for the rxrpc I/O thread to transmit.  The I/O thread invokes
     sendmsg() to transmit it - and in the case of failure, it transitions
     the rxrpc_call to the completed state.

 (4) When an rxrpc_call is completed, the app layer is notified.  In this
     case, the app is kafs and it schedules a work item to process events
     pertaining to an afs_call.

 (5) When the afs_call event processor is run, it goes down through the
     RPC-specific handler to afs_extract_data() to retrieve data from rxrpc
     - and, in this case, it picks up the error from the rxrpc_call and
     returns it.

     The error is then propagated to the afs_call and that is completed
     too.  At this point the self-reference is released.

 (6) If the rxrpc I/O thread manages to complete the rxrpc_call within the
     window between rxrpc_send_data() queuing the request packet and
     checking for call completion on the way out, then
     rxrpc_kernel_send_data() will return the error from sendmsg() to the
     app.

 (7) Then afs_make_call() will see an error and will jump to the error
     handling path which will attempt to clean up the afs_call.

 (8) The problem comes when the error handling path in afs_make_call()
     tries to unconditionally drop an async afs_call's self-reference.
     This self-reference, however, may already have been dropped by
     afs_extract_data() completing the afs_call

 (9) The refcount underflows when we return to afs_do_probe_vlserver() and
     that tries to drop its reference on the afs_call.

Fix this by making afs_make_call() attempt to complete the afs_call rather
than unconditionally putting it.  That way, if afs_extract_data() manages
to complete the call first, afs_make_call() won't do anything.

The bug can be forced by making do_udp_sendmsg() return -ENETUNREACH and
sticking an msleep() in rxrpc_send_data() after the 'success:' label to
widen the race window.

The error message looks something like:

    refcount_t: underflow; use-after-free.
    WARNING: CPU: 3 PID: 720 at lib/refcount.c:28 refcount_warn_saturate+0xba/0x110
    ...
    RIP: 0010:refcount_warn_saturate+0xba/0x110
    ...
    afs_put_call+0x1dc/0x1f0 [kafs]
    afs_fs_get_capabilities+0x8b/0xe0 [kafs]
    afs_fs_probe_fileserver+0x188/0x1e0 [kafs]
    afs_lookup_server+0x3bf/0x3f0 [kafs]
    afs_alloc_server_list+0x130/0x2e0 [kafs]
    afs_create_volume+0x162/0x400 [kafs]
    afs_get_tree+0x266/0x410 [kafs]
    vfs_get_tree+0x25/0xc0
    fc_mount+0xe/0x40
    afs_d_automount+0x1b3/0x390 [kafs]
    __traverse_mounts+0x8f/0x210
    step_into+0x340/0x760
    path_openat+0x13a/0x1260
    do_filp_open+0xaf/0x160
    do_sys_openat2+0xaf/0x170

or something like:

    refcount_t: underflow; use-after-free.
    ...
    RIP: 0010:refcount_warn_saturate+0x99/0xda
    ...
    afs_put_call+0x4a/0x175
    afs_send_vl_probes+0x108/0x172
    afs_select_vlserver+0xd6/0x311
    afs_do_cell_detect_alias+0x5e/0x1e9
    afs_cell_detect_alias+0x44/0x92
    afs_validate_fc+0x9d/0x134
    afs_get_tree+0x20/0x2e6
    vfs_get_tree+0x1d/0xc9
    fc_mount+0xe/0x33
    afs_d_automount+0x48/0x9d
    __traverse_mounts+0xe0/0x166
    step_into+0x140/0x274
    open_last_lookups+0x1c1/0x1df
    path_openat+0x138/0x1c3
    do_filp_open+0x55/0xb4
    do_sys_openat2+0x6c/0xb6

Fixes: 34fa47612b ("afs: Fix race in async call refcounting")
Reported-by: Bill MacAllister <bill@ca-zephyr.org>
Closes: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1052304
Suggested-by: Jeffrey E Altman <jaltman@auristor.com>
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Jeffrey Altman <jaltman@auristor.com>
cc: Marc Dionne <marc.dionne@auristor.com>
cc: linux-afs@lists.infradead.org
Link: https://lore.kernel.org/r/2633992.1702073229@warthog.procyon.org.uk/ # v1
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2023-12-11 15:40:41 -08:00
..
addr_list.c
afs_cm.h
afs_fs.h
afs_vl.h
afs.h rxrpc: Fix timeout of a call that hasn't yet been granted a channel 2023-05-01 07:43:19 +01:00
callback.c netfs: Fix gcc-12 warning by embedding vfs inode in netfs_i_context 2022-06-09 13:55:00 -07:00
cell.c afs: Use refcount_t rather than atomic_t 2022-08-02 18:10:11 +01:00
cmservice.c rxrpc: Tidy up abort generation infrastructure 2023-01-06 09:43:32 +00:00
dir_edit.c afs: fix the afs_dir_get_folio return value 2023-05-06 10:10:08 -07:00
dir_silly.c netfs: Fix gcc-12 warning by embedding vfs inode in netfs_i_context 2022-06-09 13:55:00 -07:00
dir.c afs: Fix setting of mtime when creating a file/dir/symlink 2023-06-07 09:03:12 -07:00
dynroot.c afs: Make error on cell lookup failure consistent with OpenAFS 2023-11-17 07:55:28 +00:00
file.c splice: Use filemap_splice_read() instead of generic_file_splice_read() 2023-05-24 08:42:17 -06:00
flock.c fs: remove locks_inode 2023-01-11 06:52:43 -05:00
fs_operation.c netfs: Fix gcc-12 warning by embedding vfs inode in netfs_i_context 2022-06-09 13:55:00 -07:00
fs_probe.c afs: Fix lost servers_outstanding count 2022-12-22 11:40:35 +00:00
fsclient.c afs: Use the operation issue time instead of the reply time for callbacks 2022-09-01 11:44:13 +01:00
inode.c afs: convert to new timestamp accessors 2023-10-18 13:26:18 +02:00
internal.h afs: Fix afs_server_list to be cleaned up with RCU 2023-11-17 07:55:27 +00:00
Kconfig afs: Convert afs to use the new fscache API 2022-01-07 13:44:47 +00:00
main.c arch: Remove Itanium (IA-64) architecture 2023-09-11 08:13:17 +00:00
Makefile afs: Convert afs to use the new fscache API 2022-01-07 13:44:47 +00:00
misc.c afs: Return -EAGAIN, not -EREMOTEIO, when a file already locked 2022-09-06 21:33:01 -04:00
mntpt.c afs: Remove check of PageError 2022-06-29 08:51:06 -04:00
proc.c afs: Use refcount_t rather than atomic_t 2022-08-02 18:10:11 +01:00
protocol_afs.h afs: Fix corruption in reads at fpos 2G-4G from an OpenAFS server 2021-09-13 09:14:21 +01:00
protocol_uae.h
protocol_yfs.h afs: Fix corruption in reads at fpos 2G-4G from an OpenAFS server 2021-09-13 09:14:21 +01:00
rotate.c afs: Adjust ACK interpretation to try and cope with NAT 2022-05-22 21:03:02 +01:00
rxrpc.c afs: Fix refcount underflow from error handling race 2023-12-11 15:40:41 -08:00
security.c fs: port ->permission() to pass mnt_idmap 2023-01-19 09:24:28 +01:00
server_list.c afs: Fix afs_server_list to be cleaned up with RCU 2023-11-17 07:55:27 +00:00
server.c afs: Fix server->active leak in afs_put_server 2022-11-30 10:02:37 -08:00
super.c afs: Mark a superblock for an R/O or Backup volume as SB_RDONLY 2023-11-24 14:52:24 +00:00
vl_alias.c
vl_list.c afs: Use refcount_t rather than atomic_t 2022-08-02 18:10:11 +01:00
vl_probe.c afs: Fix vlserver probe RTT handling 2023-06-16 14:43:41 -07:00
vl_rotate.c afs: Return ENOENT if no cell DNS record can be found 2023-11-24 14:51:18 +00:00
vlclient.c afs: Fix fall-through warnings for Clang 2021-05-25 07:30:34 -10:00
volume.c afs: remove variable nr_servers 2022-12-22 11:40:35 +00:00
write.c afs: convert to new timestamp accessors 2023-10-18 13:26:18 +02:00
xattr.c afs: move afs_xattr_handlers to .rodata 2023-10-09 16:24:16 +02:00
xdr_fs.h
yfsclient.c afs: Use the operation issue time instead of the reply time for callbacks 2022-09-01 11:44:13 +01:00