9.1 -> 9.2 , unable to load dual SAS2008 raid 6 volume

Rocky Linux General
1

Several nights ago I did a dnf update which took me from 9.1 (5.14.0-162.23.1.el9_1.x86_64) to 9.2 (5.15.0-284.11.1.el9_2.x86_64) and since then my dual LSI SAS2008 card raid 6 volume fails to load and I’m dumped to maintenance mode. This is a large’ish media server which has been happily running for about a year without issue. There are several other disk configurations that appear to be loading just fine: 1x hardware Raid 5 (os), 1x m2 disk & 1x RAM drive.

image
image1287×522 97.2 KB

Booting from any one of the previous kernal versions results in the same issue.
dnf history rollback failed with ‘Cannot find rpm nevra…’ on what looks like every package

I have successfully downgraded kmod-mpt3sas from 43.100.00.00-1.el9_2.elrepo.x86_64 back to kmod-mpt3sas-40.100.00.00-2.el9_1.elrepo.x86_64 without any effect.

Looking for a little guidance on how to fix this, I’m now at a loss…

2

Greetings cd084,
Are you running LVM on the device? (there’s a known issue with a workaround if so)

It doesn’t quite look like from the screenshot but I wanted to ask as there’s an easy workaround while we wait on upstream.

If not, can you see the raid card with lspci?

~Stack~

3

I’m trying to understand the status here.

You had kernel-5.14.0-162.23.1.el9_1.x86_64 and kmod-mpt3sas-40.100.00.00-2.el9_1.elrepo.x86_64. This was working fine.

You updated to kernel-5.15.0-284.11.1.el9_2.x86_64 and kmod-mpt3sas-43.100.00.00-1.el9_2.elrepo.x86_64. This did not work.

If you have downgraded kmod-mpt3sas to the version for el9_1, then you should be able to boot kernel-5.14.0-162.23.1.el9_1 without a problem.

We do not have hardware to test but on a system running kernel-5.14.0-284.11.1.el9_2.x86_64 and kmod-mptsas-3.04.20-4.el9_2.elrepo.x86_64, we were able to load the mpt3sas driver.

$ sudo modprobe mpt3sas
$ lsmod | head -4
Module Size Used by
mpt3sas 520192 0
raid_class 16384 1 mpt3sas
scsi_transport_sas 49152 1 mpt3sas
$ dmesg | tail -3
[37488.281579] mpt3sas: loading out-of-tree module taints kernel.
[37488.281770] mpt3sas: module verification failed: signature and/or required key missing - tainting kernel
[37488.289119] mpt3sas version 43.100.00.00 loaded

Ignore the “module verification failed” because this is due to secure boot not enabled on this system.

4

Hi Stack,

I am running LVM, just don’t remember if I was on that particular drive: I usually always use LVM so assume I did there as well. While I do see my other volumes using lvdisplay I do not see anything relating the missing volume. Not sure if lvdisplay only shows active volumes…

I do see both cards with lspci

image
image1067×237 115 KB

5

Hi toracat

Correct

Correct

Unfortunately not, same result as when it was at kernel-5.14.0-162.23.1.el9_1.x86_64 and kmod-mpt3sas-40.100.00.00-2.el9_1.elrepo.x86_64

image
image1198×224 48.4 KB

…no longer enabled in this release :cry:

6

Ah-ha, that indicates you are using the in-kernel module, not the one provided by the kmod package. “modinfo” will show where the driver is.

$ modinfo mpt3sas | grep filename
filename:       /lib/modules/5.14.0-284.11.1.el9_2.x86_64/extra/mpt3sas/mpt3sas.ko

That is from kmod.

7

Please show us the output from:

$ find /lib/modules -name mpt3sas.ko
8

image
image1193×104 42.3 KB

9

image
image1124×96 52.1 KB

10

Yes, the one with “weak-updates” came from the kmod package. In this case it was correctly installed for the 9.1 kernel.

11

Just another try,

$ ls -l `find /lib/modules -name mpt3sas.ko`

Note the back tick `.

12

image
image1157×208 98.2 KB

13

That is just as expected. Try loading the module by,

sudo modprobe mpt3sas
14

No output, no error.
Is there something I should check for a result?

15

Yes, run lsmod and check that mpt3sas is loaded.

16

image
Loaded, or does the 0 mean not loaded?

17

It is loaded.So, I don’t know why it is not working.

18

Thank you toracat for your time and effort, greatly appreciated.

19

Sit tight. I’m asking the maintainer of the kmod-mpt3sas package now.

1 Like
20

Here’s what we talked about.

Because the mpt3sas module is loaded for the 9.1 kernel, I think it is working there. The reason you had issues downgrading back to 9.1 is the system would not boot because of nvidia which was now built for 9.2 having upgraded and of course is not backward compatible.

We have no clue why mpt3sas didn’t work for 9.2. Can you show the output from:

lsinitrd -k $(uname -r) | grep -i mpt3sas

And run the same command for the 9.2 kernel.