iv/linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,187,044 Commits 104 Branches 1,843 Tags 5.7 GiB
C 97.6%
Assembly 1%
Shell 0.5%
Python 0.3%
Makefile 0.3%
afd384f0db
Go to file
Michael S. Tsirkin afd384f0db Revert "virtio-blk: support completion batching for the IRQ path" 
This reverts commit 07b679f70d.

This change appears to have broken things...
We now see applications hanging during disk accesses.
e.g.
multi-port virtio-blk device running in h/w (FPGA)
Host running a simple 'fio' test.
[global]
thread=1
direct=1
ioengine=libaio
norandommap=1
group_reporting=1
bs=4K
rw=read
iodepth=128
runtime=1
numjobs=4
time_based
[job0]
filename=/dev/vda
[job1]
filename=/dev/vdb
[job2]
filename=/dev/vdc
...
[job15]
filename=/dev/vdp

i.e. 16 disks; 4 queues per disk; simple burst of 4KB reads
This is repeatedly run in a loop.

After a few, normally <10 seconds, fio hangs.
With 64 queues (16 disks), failure occurs within a few seconds; with 8 queues (2 disks) it may take ~hour before hanging.
Last message:
fio-3.19
Starting 8 threads
Jobs: 1 (f=1): [_(7),R(1)][68.3%][eta 03h:11m:06s]
I think this means at the end of the run 1 queue was left incomplete.

'diskstats' (run while fio is hung) shows no outstanding transactions.
e.g.
$ cat /proc/diskstats
...
252       0 vda 1843140071 0 14745120568 712568645 0 0 0 0 0 3117947 712568645 0 0 0 0 0 0
252      16 vdb 1816291511 0 14530332088 704905623 0 0 0 0 0 3117711 704905623 0 0 0 0 0 0
...

Other stats (in the h/w, and added to the virtio-blk driver ([a]virtio_queue_rq(), [b]virtblk_handle_req(), [c]virtblk_request_done()) all agree, and show every request had a completion, and that virtblk_request_done() never gets called.
e.g.
PF= 0                         vq=0           1           2           3
[a]request_count     -   839416590   813148916   105586179    84988123
[b]completion1_count -   839416590   813148916   105586179    84988123
[c]completion2_count -           0           0           0           0

PF= 1                         vq=0           1           2           3
[a]request_count     -   823335887   812516140   104582672    75856549
[b]completion1_count -   823335887   812516140   104582672    75856549
[c]completion2_count -           0           0           0           0

i.e. the issue is after the virtio-blk driver.

This change was introduced in kernel 6.3.0.
I am seeing this using 6.3.3.
If I run with an earlier kernel (5.15), it does not occur.
If I make a simple patch to the 6.3.3 virtio-blk driver, to skip the blk_mq_add_to_batch()call, it does not fail.
e.g.
kernel 5.15 - this is OK
virtio_blk.c,virtblk_done() [irq handler]
                 if (likely(!blk_should_fake_timeout(req->q))) {
                          blk_mq_complete_request(req);
                 }

kernel 6.3.3 - this fails
virtio_blk.c,virtblk_handle_req() [irq handler]
                 if (likely(!blk_should_fake_timeout(req->q))) {
                          if (!blk_mq_complete_request_remote(req)) {
                                  if (!blk_mq_add_to_batch(req, iob, virtblk_vbr_status(vbr), virtblk_complete_batch)) {
                                           virtblk_request_done(req);    //this never gets called... so blk_mq_add_to_batch() must always succeed
                                   }
                          }
                 }

If I do, kernel 6.3.3 - this is OK
virtio_blk.c,virtblk_handle_req() [irq handler]
                 if (likely(!blk_should_fake_timeout(req->q))) {
                          if (!blk_mq_complete_request_remote(req)) {
                                   virtblk_request_done(req); //force this here...
                                  if (!blk_mq_add_to_batch(req, iob, virtblk_vbr_status(vbr), virtblk_complete_batch)) {
                                           virtblk_request_done(req);    //this never gets called... so blk_mq_add_to_batch() must always succeed
                                   }
                          }
                 }

Perhaps you might like to fix/test/revert this change...
Martin

Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202306090826.C1fZmdMe-lkp@intel.com/
Cc: Suwan Kim <suwan.kim027@gmail.com>
Tested-by: edliaw@google.com
Reported-by: "Roberts, Martin" <martin.roberts@intel.com>
Message-Id: <336455b4f630f329380a8f53ee8cad3868764d5c.1686295549.git.mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2023-06-21 04:14:28 -04:00
arch parisc architecture fixes for kernel v6.4-rc7: 2023-06-18 09:41:39 -07:00
block blk-cgroup: Flush stats before releasing blkcg_gq 2023-06-11 19:49:29 -06:00
certs KEYS: Add missing function documentation 2023-04-24 16:15:52 +03:00
crypto KEYS: asymmetric: Copy sig and digest in public_key_verify_signature() 2023-06-02 15:36:23 +02:00
Documentation RISC-V Fix for 6.4-rc7 2023-06-16 11:27:34 -07:00
drivers Revert "virtio-blk: support completion batching for the IRQ path" 2023-06-21 04:14:28 -04:00
fs afs: Fix vlserver probe RTT handling 2023-06-16 14:43:41 -07:00
include ata fixes for 6.4-rc7 2023-06-18 09:48:39 -07:00
init Objtool changes for v6.4: 2023-04-28 14:02:54 -07:00
io_uring io_uring/io-wq: clear current->worker_private on exit 2023-06-14 12:54:55 -06:00
ipc Merge branch 'work.namespace' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs 2023-02-24 19:20:07 -08:00
kernel 19 hotfixes. 14 are cc:stable and the remainder address issues which were 2023-06-12 16:14:34 -07:00
lib 19 hotfixes. 14 are cc:stable and the remainder address issues which were 2023-06-12 16:14:34 -07:00
LICENSES LICENSES: Add the copyleft-next-0.3.1 license 2022-11-08 15:44:01 +01:00
mm mm/mmap: Fix error path in do_vmi_align_munmap() 2023-06-18 09:32:17 -07:00
net dccp: Print deprecation notice. 2023-06-15 15:08:59 -07:00
rust Rust changes for v6.4 2023-04-30 11:20:22 -07:00
samples samples/bpf: Drop unnecessary fallthrough 2023-05-16 19:44:05 +02:00
scripts Locking changes in v6.4: 2023-05-05 12:56:55 -07:00
security selinux: don't use make's grouped targets feature yet 2023-06-01 13:56:13 -04:00
sound ASoC: Fixes for v6.4 2023-06-16 09:28:27 +02:00
tools Including fixes from wireless, and netfilter. 2023-06-15 21:11:17 -07:00
usr initramfs: Check negative timestamp to prevent broken cpio archive 2023-04-16 17:37:01 +09:00
virt KVM: Fix vcpu_array[0] races 2023-05-19 13:56:26 -04:00
.clang-format cxl for v6.4 2023-04-30 11:51:51 -07:00
.cocciconfig
.get_maintainer.ignore
.gitattributes .gitattributes: use 'dts' diff driver for *.dtso files 2023-02-26 15:28:23 +09:00
.gitignore linux-kselftest-kunit-6.4-rc1 2023-04-24 12:31:32 -07:00
.mailmap mailmap: add entry for John Keeping 2023-06-12 11:31:52 -07:00
.rustfmt.toml rust: add .rustfmt.toml 2022-09-28 09:02:20 +02:00
COPYING
CREDITS MAINTAINERS: sctp: move Neil to CREDITS 2023-05-12 08:51:32 +01:00
Kbuild Kbuild updates for v6.1 2022-10-10 12:00:45 -07:00
Kconfig
MAINTAINERS Including fixes from wireless, and netfilter. 2023-06-15 21:11:17 -07:00
Makefile Linux 6.4-rc7 2023-06-18 14:06:27 -07:00
README

README 

Linux kernel
============

There are several guides for kernel developers and users. These guides can
be rendered in a number of formats, like HTML and PDF. Please read
Documentation/admin-guide/README.rst first.

In order to build the documentation, use ``make htmldocs`` or
``make pdfdocs``.  The formatted documentation can also be read online at:

    https://www.kernel.org/doc/html/latest/

There are various text files in the Documentation/ subdirectory,
several of them using the Restructured Text markup notation.

Please read the Documentation/process/changes.rst file, as it contains the
requirements for building and running the kernel, and information about
the problems which may result by upgrading your kernel.