Filipe Manana
e110f8911d
btrfs: fix lockdep splat and potential deadlock after failure running delayed items
...
When running delayed items we are holding a delayed node's mutex and then
we will attempt to modify a subvolume btree to insert/update/delete the
delayed items. However if have an error during the insertions for example,
btrfs_insert_delayed_items() may return with a path that has locked extent
buffers (a leaf at the very least), and then we attempt to release the
delayed node at __btrfs_run_delayed_items(), which requires taking the
delayed node's mutex, causing an ABBA type of deadlock. This was reported
by syzbot and the lockdep splat is the following:
WARNING: possible circular locking dependency detected
6.5.0-rc7-syzkaller-00024-g93f5de5f648d #0 Not tainted
------------------------------------------------------
syz-executor.2/13257 is trying to acquire lock:
ffff88801835c0c0 (&delayed_node->mutex){+.+.}-{3:3}, at: __btrfs_release_delayed_node+0x9a/0xaa0 fs/btrfs/delayed-inode.c:256
but task is already holding lock:
ffff88802a5ab8e8 (btrfs-tree-00){++++}-{3:3}, at: __btrfs_tree_lock+0x3c/0x2a0 fs/btrfs/locking.c:198
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (btrfs-tree-00){++++}-{3:3}:
__lock_release kernel/locking/lockdep.c:5475 [inline]
lock_release+0x36f/0x9d0 kernel/locking/lockdep.c:5781
up_write+0x79/0x580 kernel/locking/rwsem.c:1625
btrfs_tree_unlock_rw fs/btrfs/locking.h:189 [inline]
btrfs_unlock_up_safe+0x179/0x3b0 fs/btrfs/locking.c:239
search_leaf fs/btrfs/ctree.c:1986 [inline]
btrfs_search_slot+0x2511/0x2f80 fs/btrfs/ctree.c:2230
btrfs_insert_empty_items+0x9c/0x180 fs/btrfs/ctree.c:4376
btrfs_insert_delayed_item fs/btrfs/delayed-inode.c:746 [inline]
btrfs_insert_delayed_items fs/btrfs/delayed-inode.c:824 [inline]
__btrfs_commit_inode_delayed_items+0xd24/0x2410 fs/btrfs/delayed-inode.c:1111
__btrfs_run_delayed_items+0x1db/0x430 fs/btrfs/delayed-inode.c:1153
flush_space+0x269/0xe70 fs/btrfs/space-info.c:723
btrfs_async_reclaim_metadata_space+0x106/0x350 fs/btrfs/space-info.c:1078
process_one_work+0x92c/0x12c0 kernel/workqueue.c:2600
worker_thread+0xa63/0x1210 kernel/workqueue.c:2751
kthread+0x2b8/0x350 kernel/kthread.c:389
ret_from_fork+0x2e/0x60 arch/x86/kernel/process.c:145
ret_from_fork_asm+0x11/0x20 arch/x86/entry/entry_64.S:304
-> #0 (&delayed_node->mutex){+.+.}-{3:3}:
check_prev_add kernel/locking/lockdep.c:3142 [inline]
check_prevs_add kernel/locking/lockdep.c:3261 [inline]
validate_chain kernel/locking/lockdep.c:3876 [inline]
__lock_acquire+0x39ff/0x7f70 kernel/locking/lockdep.c:5144
lock_acquire+0x1e3/0x520 kernel/locking/lockdep.c:5761
__mutex_lock_common+0x1d8/0x2530 kernel/locking/mutex.c:603
__mutex_lock kernel/locking/mutex.c:747 [inline]
mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:799
__btrfs_release_delayed_node+0x9a/0xaa0 fs/btrfs/delayed-inode.c:256
btrfs_release_delayed_node fs/btrfs/delayed-inode.c:281 [inline]
__btrfs_run_delayed_items+0x2b5/0x430 fs/btrfs/delayed-inode.c:1156
btrfs_commit_transaction+0x859/0x2ff0 fs/btrfs/transaction.c:2276
btrfs_sync_file+0xf56/0x1330 fs/btrfs/file.c:1988
vfs_fsync_range fs/sync.c:188 [inline]
vfs_fsync fs/sync.c:202 [inline]
do_fsync fs/sync.c:212 [inline]
__do_sys_fsync fs/sync.c:220 [inline]
__se_sys_fsync fs/sync.c:218 [inline]
__x64_sys_fsync+0x196/0x1e0 fs/sync.c:218
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x63/0xcd
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(btrfs-tree-00);
lock(&delayed_node->mutex);
lock(btrfs-tree-00);
lock(&delayed_node->mutex);
*** DEADLOCK ***
3 locks held by syz-executor.2/13257:
#0 : ffff88802c1ee370 (btrfs_trans_num_writers){++++}-{0:0}, at: spin_unlock include/linux/spinlock.h:391 [inline]
#0 : ffff88802c1ee370 (btrfs_trans_num_writers){++++}-{0:0}, at: join_transaction+0xb87/0xe00 fs/btrfs/transaction.c:287
#1 : ffff88802c1ee398 (btrfs_trans_num_extwriters){++++}-{0:0}, at: join_transaction+0xbb2/0xe00 fs/btrfs/transaction.c:288
#2 : ffff88802a5ab8e8 (btrfs-tree-00){++++}-{3:3}, at: __btrfs_tree_lock+0x3c/0x2a0 fs/btrfs/locking.c:198
stack backtrace:
CPU: 0 PID: 13257 Comm: syz-executor.2 Not tainted 6.5.0-rc7-syzkaller-00024-g93f5de5f648d #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e7/0x2d0 lib/dump_stack.c:106
check_noncircular+0x375/0x4a0 kernel/locking/lockdep.c:2195
check_prev_add kernel/locking/lockdep.c:3142 [inline]
check_prevs_add kernel/locking/lockdep.c:3261 [inline]
validate_chain kernel/locking/lockdep.c:3876 [inline]
__lock_acquire+0x39ff/0x7f70 kernel/locking/lockdep.c:5144
lock_acquire+0x1e3/0x520 kernel/locking/lockdep.c:5761
__mutex_lock_common+0x1d8/0x2530 kernel/locking/mutex.c:603
__mutex_lock kernel/locking/mutex.c:747 [inline]
mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:799
__btrfs_release_delayed_node+0x9a/0xaa0 fs/btrfs/delayed-inode.c:256
btrfs_release_delayed_node fs/btrfs/delayed-inode.c:281 [inline]
__btrfs_run_delayed_items+0x2b5/0x430 fs/btrfs/delayed-inode.c:1156
btrfs_commit_transaction+0x859/0x2ff0 fs/btrfs/transaction.c:2276
btrfs_sync_file+0xf56/0x1330 fs/btrfs/file.c:1988
vfs_fsync_range fs/sync.c:188 [inline]
vfs_fsync fs/sync.c:202 [inline]
do_fsync fs/sync.c:212 [inline]
__do_sys_fsync fs/sync.c:220 [inline]
__se_sys_fsync fs/sync.c:218 [inline]
__x64_sys_fsync+0x196/0x1e0 fs/sync.c:218
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f3ad047cae9
Code: 28 00 00 00 75 (...)
RSP: 002b:00007f3ad12510c8 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
RAX: ffffffffffffffda RBX: 00007f3ad059bf80 RCX: 00007f3ad047cae9
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000005
RBP: 00007f3ad04c847a R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007f3ad059bf80 R15: 00007ffe56af92f8
</TASK>
------------[ cut here ]------------
Fix this by releasing the path before releasing the delayed node in the
error path at __btrfs_run_delayed_items().
Reported-by: syzbot+a379155f07c134ea9879@syzkaller.appspotmail.com
Link: https://lore.kernel.org/linux-btrfs/000000000000abba27060403b5bd@google.com/
CC: stable@vger.kernel.org # 4.14+
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-09-08 14:10:53 +02:00
..
2023-07-20 19:21:48 +00:00
2023-05-24 08:42:17 -06:00
2023-05-24 08:42:17 -06:00
2023-07-04 12:24:32 -07:00
2023-07-29 14:08:22 -07:00
2023-05-30 16:42:00 -07:00
2023-05-24 08:42:17 -06:00
2023-09-08 14:10:53 +02:00
2023-06-26 10:14:36 -07:00
2023-08-06 15:08:35 +02:00
2023-08-06 15:08:35 +02:00
2023-04-03 09:23:54 +02:00
2023-05-24 08:42:17 -06:00
2023-05-23 19:46:09 -07:00
2023-05-31 19:02:14 +01:00
2023-03-13 12:36:34 +01:00
2023-06-29 13:27:50 -07:00
2023-05-24 08:42:17 -06:00
2023-05-17 18:21:34 +02:00
2023-08-01 16:12:24 +08:00
2023-08-06 15:08:35 +02:00
2023-08-06 15:08:35 +02:00
2023-06-29 13:39:51 -07:00
2023-07-23 08:21:14 -04:00
2023-07-05 14:14:37 -07:00
2023-05-24 08:42:17 -06:00
2023-02-23 17:55:40 -08:00
2023-01-30 12:51:54 +00:00
2023-07-19 11:00:27 -07:00
2023-08-07 18:42:04 +02:00
2023-05-24 08:42:17 -06:00
2023-05-24 08:42:17 -06:00
2023-06-27 17:10:27 -07:00
2023-05-24 08:42:17 -06:00
2023-06-23 16:59:32 -07:00
2023-07-17 08:49:57 -07:00
2023-07-10 23:09:21 -04:00
2023-06-26 11:52:12 -07:00
2023-08-06 15:08:35 +02:00
2023-07-03 12:56:23 -07:00
2023-07-01 14:38:25 -07:00
2023-05-24 08:42:17 -06:00
2023-06-08 13:42:33 +02:00
2023-07-01 14:38:25 -07:00
2023-04-13 13:13:52 -07:00
2023-08-03 09:26:34 -07:00
2023-08-04 13:03:43 -07:00
2023-07-25 00:30:02 -05:00
2023-07-04 13:29:29 +02:00
2023-08-06 15:08:35 +02:00
2023-07-07 14:59:38 -07:00
2023-08-06 15:08:35 +02:00
2023-05-24 08:42:17 -06:00
2023-05-24 08:42:16 -06:00
2023-08-06 15:08:35 +02:00
2023-08-11 14:19:20 -07:00
2023-06-27 21:21:32 -07:00
2023-03-14 12:56:30 -06:00
2023-03-14 12:56:30 -06:00
2023-06-05 16:50:30 +02:00
2023-06-28 10:28:11 -07:00
2023-06-28 10:28:11 -07:00
2023-05-24 08:42:17 -06:00
2023-08-16 00:26:07 -05:00
2023-07-08 09:29:30 -07:00
2023-06-15 13:37:53 +02:00
2023-06-26 11:52:12 -07:00
2023-01-19 09:24:26 +01:00
2023-05-24 08:42:17 -06:00
2023-06-29 13:39:51 -07:00
2023-05-24 08:42:17 -06:00
2023-04-13 13:13:54 -07:00
2023-08-08 14:59:49 -07:00
2023-06-20 22:47:55 -07:00
2023-07-17 08:48:56 -07:00
2023-08-10 12:59:47 +09:00
2023-06-15 09:22:23 +02:00
2023-03-30 08:51:48 +02:00
2023-01-19 09:24:28 +01:00
2023-05-30 15:49:46 -07:00
2023-06-28 20:35:21 -07:00
2022-12-02 13:57:04 -08:00
2023-06-29 13:39:51 -07:00
2023-05-15 09:42:01 +02:00
2023-06-26 09:50:21 -07:00
2023-05-17 09:16:59 +02:00
2023-06-26 07:54:23 -06:00
2023-06-28 10:28:11 -07:00
2023-06-15 09:22:23 +02:00
2023-06-26 09:50:21 -07:00
2023-06-29 13:31:44 -07:00
2023-02-20 11:53:11 -08:00
2023-06-12 10:43:45 +02:00
2023-07-02 13:15:49 +02:00
2023-08-06 15:08:36 +02:00
2023-06-07 21:49:55 +02:00
2022-12-01 10:46:54 -05:00
2023-06-09 16:25:14 -07:00
2023-01-19 09:24:28 +01:00
2023-08-12 09:18:47 -07:00
2023-06-26 10:14:36 -07:00
2023-01-19 09:24:29 +01:00
2023-05-24 16:29:21 -05:00
2023-06-09 16:25:53 -07:00
2023-03-09 22:36:12 +01:00
2023-06-26 12:47:20 -07:00
2022-12-08 21:49:25 -05:00
2023-01-19 09:24:30 +01:00
2023-04-18 16:30:02 -07:00
2023-07-04 10:20:29 +02:00
2023-06-26 10:27:04 -07:00
2023-04-20 22:55:35 -04:00
2023-08-06 15:08:35 +02:00
2023-05-12 17:17:27 +02:00
2023-05-19 04:30:22 +02:00
2023-05-19 04:30:22 +02:00
2023-03-06 09:59:20 +01:00
2023-05-24 08:42:16 -06:00
2023-05-24 08:42:17 -06:00
2023-08-06 15:08:35 +02:00
2023-05-24 11:03:59 +02:00
2022-11-25 13:01:55 -05:00
2023-07-26 21:56:06 -07:00
2023-02-20 11:53:11 -08:00
2023-05-17 15:20:17 +02:00
2023-06-29 13:39:51 -07:00
2023-05-23 21:43:26 -07:00
2023-06-23 16:58:19 -07:00
2023-02-20 11:53:11 -08:00
2023-05-17 15:25:20 +02:00