Have a small cluster with Rocky Linux 9.6 with a working Infiniband network. I want to export one folder on machineA to machineB via NFS over RDMA. Have followed various guides from RedHat and Gemini.
Where I am stuck is telling the server to use port 20049 for rdma:
[root@gpu001 scratch]# echo "rdma 20049" > /proc/fs/nfsd/portlist
-bash: echo: write error: Protocol not supported
Some googling suggest Mellanox no longer supports NFS over RDMA, per various posts on Nvidia forum. Seems they dropped support after RedHat 8.2.
Does anyone have this working now? Or is there some better way to do what I want ? Some googling said to try install Mellanox drivers by hand and passing it option for rdma support( seems “hacky” though )…
Here is some more output from. my server if it helps
[root@gpu001 scratch]# lsmod | grep rdma
svcrdma 12288 0
rpcrdma 12288 0
xprtrdma 12288 0
rdma_ucm 36864 0
rdma_cm 163840 2 beegfs,rdma_ucm
iw_cm 69632 1 rdma_cm
ib_cm 155648 2 rdma_cm,ib_ipoib
ib_uverbs 225280 2 rdma_ucm,mlx5_ib
ib_core 585728 9 beegfs,rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
mlx_compat 20480 16 beegfs,rdma_cm,ib_ipoib,mlxdevm,rpcrdma,mlxfw,xprtrdma,iw_cm,svcrdma,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core
[root@gpu001 scratch]# dmesg | grep rdma
[1257122.629424] xprtrdma: xprtrdma is obsoleted, loading rpcrdma instead
[1257208.479330] svcrdma: svcrdma is obsoleted, loading rpcrdma instead
