Rocky 8/9 and Mellanox Infiniband MT25208 problem

Rocky Linux Help & Support
1

Hi,

I have IBM Blade HS21 with ConnectX-3(?)
0c:00.0 InfiniBand: Mellanox Technologies MT25208 InfiniHost III Ex (Tavor compatibility mode) (rev a0).

Everything works perfectly out of the box on CentOS 7.7-7.9. I simply checked the Server with “Infiniband Support” box during installation from DVD image and that’s it.

When trying to install Rocky 9.6, 8.10, 8.8, or CentOS 8.2, it doesn’t work. The required interface doesn’t appear after installation.

I saw this topic How to install network driver, but I didn’t really understand what to do.

My adapter is also on this list kmod-ib_mthca deviceids [ELRepo Wiki], but I don’t know what that means.

As far as I understand, there are two types of drivers, OFED and from “Infiniband Support”. I think any one will suit me.

I found information on Reddit that the latest version of the OFED drivers from Nvidia for ConnectX3 was 4.9 LTS with support for Rocky 8.8, but I didn’t get anything working. The drivers installed, but nothing happened. The ibstat command output is empty. Old firmware or something?

Starting MLNX_OFED_LINUX-4.9-6.0.6.0 installation ...
Device (0c:00.0):
        0c:00.0 InfiniBand: Mellanox Technologies MT25208 InfiniHost III Ex (Tavor compatibility mode) (rev a0)
        Link Width: x8
        PCI Link Speed: 2.5GT/s
Installation finished successfully.
Updating / installing...
   1:mlnx-fw-updater-4.9-6.0.6.0      ################################# [100%]
Added 'RUN_FW_UPDATER_ONBOOT=no to /etc/infiniband/openib.conf
Initializing...
Attempting to perform Firmware update...
Querying Mellanox devices firmware ...
Device #1:
----------
  Device Type:      N/A
  Part Number:      --
  Description:
  PSID:
  PCI Device Name:  0c:00.0
  Port1 MAC:        N/A
  Port1 GUID:       N/A
  Port2 MAC:        N/A
  Port2 GUID:       N/A
  Versions:         Current        Available
     FW             --
  Status:           Failed to open device
---------
-E- Failed to query 0c:00.0 device, error : No such file or directory. MFE_CR_ERROR
Log File: /tmp/xWQLg1XROA
Real log file: /tmp/MLNX_OFED_LINUX.40695.logs/fw_update.log

So the question is, what is the latest version of Rocky (for this hardware(?)) that will allow me to use Infiniband with my adapter and how to make it work?

Now I have two machine with Centos 8.2 and Rocky 8.8. And I’m ready to start from scratch with any version of the distribution.

Thanks a lot :folded_hands:.

2

To install Rocky 8, you’d need the dd-ib_mthca-1.0.20080404-1.1.el8_10.elrepo.iso image which is found in:

If you need the instructions on how to use the iso image, there is a video that may help you:

The procedure for Rocky 9 is basically the same.

2 Likes
3

It’s so simple :man_facepalming: :grin:. I’m familiar with DD because I use it for my SAS controller. And it turns out I even have the dd-ib_mthca-1.0.20080404-1.1.el8_10.elrepo.iso file in my downloads…

Thanks again. Everything worked with Rocky 8.10.

Do you think this is the latest major OS version for my hardware?
On 9.6, the installer wouldn’t even load.

изображение
изображение691×447 23.5 KB

4

You have Rocky 8. Therefore, you can run:

ld.so --help

This output is from machine that can run Rocky 9 (requires x86-64-v2) but not Rocky 10 (requires x86-64-v3):

Subdirectories of glibc-hwcaps directories, in priority order:
  x86-64-v4
  x86-64-v3
  x86-64-v2 (supported, searched)
1 Like
5

from Blade hs21 with Rocky8.10

This program interpreter self-identifies as: /lib64/ld-linux-x86-64.so.2

Shared library search path:
  (libraries located via /etc/ld.so.cache)
  /lib64 (system search path)
  /usr/lib64 (system search path)

Subdirectories of glibc-hwcaps directories, in priority order:
  x86-64-v4
  x86-64-v3
  x86-64-v2

Legacy HWCAP subdirectories under library search path directories:
  x86_64 (AT_PLATFORM; supported, searched)
  tls (supported, searched)
  avx512_1
  x86_64 (supported, searched)

from Blade hs23 with Rocky8.10

This program interpreter self-identifies as: /lib64/ld-linux-x86-64.so.2

Shared library search path:
  (libraries located via /etc/ld.so.cache)
  /lib64 (system search path)
  /usr/lib64 (system search path)

Subdirectories of glibc-hwcaps directories, in priority order:
  x86-64-v4
  x86-64-v3
  x86-64-v2 (supported, searched)

Legacy HWCAP subdirectories under library search path directories:
  x86_64 (AT_PLATFORM; supported, searched)
  tls (supported, searched)
  avx512_1
  x86_64 (supported, searched)

So, Rocky 9 will run on hs23, but not on hs21, only Rocky 8?

2 Likes
6

Thank a lot to this forum and everyone who helped :slightly_smiling_face:

1 Like