iv/linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

RDMA/rw: Support threshold for registration vs scattering to local pages

Browse Source 
If there are more scatter entries than the recommended limit provided by
the ib device, UMR registration is used. This will provide optimal
performance when performing large RDMA READs over devices that advertise
the threshold capability.

With ConnectX-5 running NVMeoF RDMA with FIO single QP 128KB writes:
Without use of cap: 70Gb/sec
With use of cap: 84Gb/sec

Link: https://lore.kernel.org/r/20191007135933.12483-3-leon@kernel.org
Signed-off-by: Yamin Friedman <yaminf@mellanox.com>
Reviewed-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
This commit is contained in:
Yamin Friedman 2019-10-07 16:59:32 +03:00 committed by Jason Gunthorpe
parent 7d47433cf7
commit 00bd1439f4
2 changed files with 17 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

25
drivers/infiniband/core/rw.c
View File

@ -20,14 +20,17 @@ module_param_named(force_mr, rdma_rw_force_mr, bool, 0);
MODULE_PARM_DESC(force_mr, "Force usage of MRs for RDMA READ/WRITE operations");
/*
* Check if the device might use memory registration. This is currently only
* true for iWarp devices. In the future we can hopefully fine tune this based
* on HCA driver input.
* Report whether memory registration should be used. Memory registration must
* be used for iWarp devices because of iWARP-specific limitations. Memory
* registration is also enabled if registering memory might yield better
* performance than using multiple SGE entries, see rdma_rw_io_needs_mr()
*/
static inline bool rdma_rw_can_use_mr(struct ib_device *dev, u8 port_num)
{
if (rdma_protocol_iwarp(dev, port_num))
return true;
if (dev->attrs.max_sgl_rd)
return true;
if (unlikely(rdma_rw_force_mr))
return true;
return false;
@ -35,17 +38,19 @@ static inline bool rdma_rw_can_use_mr(struct ib_device *dev, u8 port_num)
/*
* Check if the device will use memory registration for this RW operation.
* We currently always use memory registrations for iWarp RDMA READs, and
* have a debug option to force usage of MRs.
*
* XXX: In the future we can hopefully fine tune this based on HCA driver
* input.
* For RDMA READs we must use MRs on iWarp and can optionally use them as an
* optimization otherwise. Additionally we have a debug option to force usage
* of MRs to help testing this code path.
*/
static inline bool rdma_rw_io_needs_mr(struct ib_device *dev, u8 port_num,
enum dma_data_direction dir, int dma_nents)
{
if (rdma_protocol_iwarp(dev, port_num) && dir == DMA_FROM_DEVICE)
return true;
if (dir == DMA_FROM_DEVICE) {
if (rdma_protocol_iwarp(dev, port_num))
return true;
if (dev->attrs.max_sgl_rd && dma_nents > dev->attrs.max_sgl_rd)
return true;
}
if (unlikely(rdma_rw_force_mr))
return true;
return false;

2
include/rdma/ib_verbs.h
View File

@ -445,6 +445,8 @@ struct ib_device_attr {
struct ib_tm_caps tm_caps;
struct ib_cq_caps cq_caps;
u64 max_dm_size;
/* Max entries for sgl for optimized performance per READ */
u32 max_sgl_rd;
};
enum ib_mtu {