May 19, 2023, 2:25am
Several nights ago I did a dnf update which took me from 9.1 (5.14.0-162.23.1.el9_1.x86_64) to 9.2 (5.15.0-284.11.1.el9_2.x86_64) and since then my dual LSI SAS2008 card raid 6 volume fails to load and I’m dumped to maintenance mode. This is a large’ish media server which has been happily running for about a year without issue. There are several other disk configurations that appear to be loading just fine: 1x hardware Raid 5 (os), 1x m2 disk & 1x RAM drive.
Booting from any one of the previous kernal versions results in the same issue.
dnf history rollback failed with ‘Cannot find rpm nevra…’ on what looks like every package
I have successfully downgraded kmod-mpt3sas from 43.100.00.00-1.el9_2.elrepo.x86_64 back to kmod-mpt3sas-40.100.00.00-2.el9_1.elrepo.x86_64 without any effect.
Looking for a little guidance on how to fix this, I’m now at a loss…
May 19, 2023, 2:38am
Are you running LVM on the device? (there’s a known issue with a workaround if so)
It doesn’t quite look like from the screenshot but I wanted to ask as there’s an easy workaround while we wait on upstream.
If not, can you see the raid card with lspci?
I’m trying to understand the status here.
You had kernel-5.14.0-162.23.1.el9_1.x86_64 and kmod-mpt3sas-40.100.00.00-2.el9_1.elrepo.x86_64. This was working fine.
You updated to kernel-5.15.0-284.11.1.el9_2.x86_64 and kmod-mpt3sas-43.100.00.00-1.el9_2.elrepo.x86_64. This did not work.
If you have downgraded kmod-mpt3sas to the version for el9_1, then you should be able to boot kernel-5.14.0-162.23.1.el9_1 without a problem.
We do not have hardware to test but on a system running kernel-5.14.0-284.11.1.el9_2.x86_64 and kmod-mptsas-3.04.20-4.el9_2.elrepo.x86_64, we were able to load the mpt3sas driver.
$ sudo modprobe mpt3sas
$ lsmod | head -4
Module Size Used by
mpt3sas 520192 0
raid_class 16384 1 mpt3sas
scsi_transport_sas 49152 1 mpt3sas
$ dmesg | tail -3
[37488.281579] mpt3sas: loading out-of-tree module taints kernel.
[37488.281770] mpt3sas: module verification failed: signature and/or required key missing - tainting kernel
[37488.289119] mpt3sas version 43.100.00.00 loaded
Ignore the “module verification failed” because this is due to secure boot not enabled on this system.
May 19, 2023, 3:08am
I am running LVM, just don’t remember if I was on that particular drive: I usually always use LVM so assume I did there as well. While I do see my other volumes using lvdisplay I do not see anything relating the missing volume. Not sure if lvdisplay only shows active volumes…
I do see both cards with lspci
May 19, 2023, 3:28am
Unfortunately not, same result as when it was at kernel-5.14.0-162.23.1.el9_1.x86_64 and kmod-mpt3sas-40.100.00.00-2.el9_1.elrepo.x86_64
…no longer enabled in this release
Ah-ha, that indicates you are using the in-kernel module, not the one provided by the kmod package. “modinfo” will show where the driver is.
$ modinfo mpt3sas | grep filename
That is from kmod.
Please show us the output from:
$ find /lib/modules -name mpt3sas.ko
Yes, the one with “weak-updates” came from the kmod package. In this case it was correctly installed for the 9.1 kernel.
Just another try,
$ ls -l `find /lib/modules -name mpt3sas.ko`
Note the back tick `.
That is just as expected. Try loading the module by,
sudo modprobe mpt3sas
May 19, 2023, 4:30am
No output, no error.
Is there something I should check for a result?
Yes, run lsmod and check that mpt3sas is loaded.
May 19, 2023, 4:51am
Loaded, or does the 0 mean not loaded?
It is loaded.So, I don’t know why it is not working.
May 19, 2023, 5:10am
Thank you toracat for your time and effort, greatly appreciated.
Sit tight. I’m asking the maintainer of the kmod-mpt3sas package now.
Here’s what we talked about.
Because the mpt3sas module is loaded for the 9.1 kernel, I think it is working there. The reason you had issues downgrading back to 9.1 is the system would not boot because of nvidia which was now built for 9.2 having upgraded and of course is not backward compatible.
We have no clue why mpt3sas didn’t work for 9.2. Can you show the output from:
lsinitrd -k $(uname -r) | grep -i mpt3sas
And run the same command for the 9.2 kernel.