After converting from CentOS Stream 8 to Rocky Linux 8, ext4 not found, megaraid_sas hardware issues

aram535 · March 14, 2025, 10:42am

Hi,

I migrated over from CentOS 8 to Rocky Linux 8 using the shell script and I have run into two issues: On reboot, I’m in emergency mode so I have very limited access.

a) megaraid_sas is tagged as deprecated (it’s just a warning),
b) ext4 is not available (all of my filesystems are ext4 at the moment).

lsblk does list all of my drives, so looks like the megaraid card is being recognized now (used: modprobe megaraid_sas).for a temporary module load.

Any advice on how to recover and boot would be appreciated.

In case anyone cares, this is a Dell R720 with the built-in raid.

iwalker · March 14, 2025, 11:13am

Make sure you have e2fsprogs package installed. Although the kernel should support ext2/3/4 by default.

aram535 · March 14, 2025, 11:37am

I did verify that GRUB is booting into Rocky Linux grub entry.

Thanks @iwalker, I’m still in emergency mode so can’t see any package tools. Is my next step to load into a ISO image and mount the root to see what packages are installed?

Is this some sort of strange journal issue?

EDIT: I did a fsck /dev/sda1 (which is root) and it came back clean.

aram535 · March 15, 2025, 12:51am

I’m still struggling with this system.

I can boot of a rescue ISO and mount everything, all the drives have been fscked. I removed non-eseential /etc/fstab entries. Reinstalled the kernel-* packages, regenerated a new initfsram file. e2fs packages are installed.

FrankCox · March 15, 2025, 3:01am

unknown filesystem type ‘ext4’

Somehow you don’t have ext4 support loaded and/or enabled. I think all the rest of those errors stem from that.

jlehtone · March 15, 2025, 9:30am

Is ext4 support in the kernel or a kernel module? If the latter, then it must be in the initramfs or else kernel cannot mount the /. One would boot the Rocky installer image in troubleshoot mode to gain access to filesystems for regenerating initramfs, etc.

ext4: Unknown symbol jbd2...

Something is missing or a mismatch. Is it possible that there is somehow a mix of Stream and Rocky kernel files?

aram535 · March 15, 2025, 2:23pm

@jlehtone I didn’t do the migration myself I used the migration script from rocky linux. Is it possible that it missed something? certainly but I figured the kernel and grub would be on the top of the things it would convert. I was able to use a 9.5 troulbeshooting ISO to mount everything, fsck and reinstalling the kernel and ram files but that was just via dnf reinstall kernel\* which I believe reinstall whatever is in the cache directory.

I’ve downloaded the 8.5 ISO and burning it to a USB stick at the moment and planning on rpm installing the kernel-core,modules,etc from the ISO rather than the cache, hoping that’s the issue.

aram535 · March 16, 2025, 2:35pm

Still no luck. Booted from a 8.5 ISO, troubleshooting.

chroot,
enabled NIC
downloaded the original 8.5 images from the repo
reinstalled kernel, kernel-{core,modules,tools,tools-libs,headers, etc} all that was installed.
regenerated the initram using dracut
checked out the efibootmgr and deleted the old CentOS and some off RedHat entries. Rocky is there.

Reboot goes back into the emergency and missing ext4 … I’m out of ideas here.

iwalker · March 16, 2025, 4:38pm

Current Rocky 8 version is 8.10, also current kernel is:

[root@rocky8 ~]# uname -r
4.18.0-553.44.1.el8_10.x86_64

older versions than 8.10 are unsupported, your system seems to show extremely old packages. Either way, ext4 should work fine in the kernel.

Make a backup of your data and clean install your system - obviously something is wrong most likely from CentOS Stream 8.

aram535 · March 16, 2025, 6:48pm

@iwalker I didn’t pick the version, that’s the version the migration script moved me to.

iwalker · March 16, 2025, 6:59pm

Yep, understand that, but obviously the upgrade hasn’t gone well and even if you do get the ext filesystem situation fixed, there’s no telling what else might be problematic in the future. Upgrades don’t always work, even with using leapp in RHEL.

I still stick by what I said though, backup your data and make a clean installation. That is what I would do. I use Rocky systems with ext4 without problems, so in yours something has seriously broken.

aram535 · March 16, 2025, 7:27pm

@iwalker I appreciate the advice, unfortunately it’s a little too late to back it up and it’s just too big. If I get it to boot at least I can remotely try to fix the rest or move the data off, right now it’s just a bad setup. This is the plex server with TB of data and dozens and dozens of containers.
I should never have attempted the migration - just got frustrated with the remotes.

iwalker · March 16, 2025, 8:04pm

Perhaps you can try booting it in rescue mode with network configured, chrooting into it and then doing a dnf update to update all packages to the latest, so that the machine is Rocky 8.10.

Not great, doing things when there isn’t a backup, but too late for that now. You could even boot it, configure network and mount disks to copy data off using rsync or whatever.

KernelKurtz · March 17, 2025, 1:26am

If you can read /var/log/migrate2rocky.log it might have some clues in the form of any errors that may have occurred during your migration (assuming that was the tool you used).

VAXman65 · March 17, 2025, 12:35pm

My setup is based on a MegaRAID 9260i on a consumer motherboard. My path through fail and success was a little different than yours, but the MegaRAID reference reminded me of my troubles.

In my setup, I boot from a SATA SSD, so i am not dependant upon the presence of my RAID card to boot. When I last upgraded/rebuilt my system, I started by unplugging the RAID card and installing the OS to a fresh SSD. However, when I plugged the RAID card back in, I would see the card with lspci, but none of the /dev/sdx devices appeared. I had to get through the MegaRAID kit install before this would work.

I keep a copy of the install kit I downloaded directly from the Broadcom hosted MegaRAID SAS 9260-4i / 9260-8i / 9260DE-8i / 9261-8i downloads page which ncludes a link to the official downloads page which in turn has a link to MegaRAID Storage Manager - Linux x64 (current version 17.05.06.00)

I used the install.csh and, after a few false starts, had a clean install. Note that it is critical to have a clean install run. Missing prereq. kits will mess it up and you have to force things a little with dnf/yum to get out the partially installed kit. for me, after that clean install and the follwoing reboot, the devices showed up; I even had the web GUI back and Zabbix monitoring. I’ve had no problem running this MegaRAID version with Rocky Linux 8.x or 9.x.

Are you booting from MegaRAID? If so, problems would compound if the system could not see and mount the root partition.

jlehtone · March 17, 2025, 1:01pm

In principle, a fresh install (of Rocky) to such media would be nice – just mount the existing data volumes afterwards. Alas, the “Dell R720” will probably limit the chance of “add SSD for OS”.

One could boot the Rocky 8.10 installer to see whether it can read the controller/filesystems.
It is possible to supply the installer a “Driver Update Disk” (“dud”, with inst.dd=URL) kernel cmdline option, and ELRepo does have duds for (some) RAID controllers.

If installer can see the RAID volumes, then question is where to install?

I leave some of storage unallocated, so I can always add partitions – the “system volumes” do not need much space. If there is no unallocated space, then the next option is to shrink existing volumes (if safe and possible to do) to create unallocated space. If even that is not possible, then backup the system volumes (to retain copy of configs within) and overwrite. If the data is on the / filesystem, then even that is not feasible.

aram535 · March 22, 2025, 7:04am

To give an update, I was about to give up and do a careful reinstall when I accidently booted to the hard disk and choose the installed Rocky Rescue image… and the system booted?!?
So the problem is 100% that the kernel for some reason can’t it’s /lib/modules/$(uname -r)/.

/sdb is the boot drive, /sdb3 is the root partition with the lib directory, it isn’t a separate mount.

$ lsblk 
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0  18.2T  0 disk 
└─sda1   8:1    0  18.2T  0 part /fs1
sdb      8:16   0   223G  0 disk 
├─sdb1   8:17   0   600M  0 part /boot/efi
├─sdb2   8:18   0     1G  0 part /boot
├─sdb3   8:19   0    50G  0 part /
├─sdb4   8:20   0  15.7G  0 part [SWAP]
└─sdb5   8:21   0 155.7G  0 part /home
sdc      8:32   1  58.6G  0 disk 
├─sdc1   8:33   1  58.6G  0 part 
└─sdc2   8:34   1    32M  0 part 
sr0     11:0    1  1024M  0 rom  
$ df -h
Filesystem            Type      Size  Used Avail Use% Mounted on
devtmpfs              devtmpfs   16G     0   16G   0% /dev
/dev/sdb3             ext4       49G   20G   27G  42% /
/dev/sdb5             ext4      153G  7.3G  138G   6% /home
/dev/sdb2             ext4      974M  198M  709M  22% /boot
/dev/sdb1             vfat      599M  5.9M  594M   1% /boot/efi
//192.168.1.94/Public cifs       16T  8.6T  7.5T  54% /nas
/dev/sda1             ext4       19T   15T  2.3T  87% /fs1
$ ls -al /lib/modules/$(uname -r)/
total 19488
drwxr-xr-x.  6 root root     4096 Mar 17 15:22 ./
drwxr-xr-x.  9 root root     4096 Mar 17 15:21 ../
drwxr-xr-x. 14 root root     4096 Mar 17 15:21 kernel/
drwxr-xr-x.  2 root root     4096 Mar 10 07:42 updates/
drwxr-xr-x.  2 root root     4096 Mar 17 15:21 vdso/
drwxr-xr-x.  3 root root     4096 Mar 17 15:22 weak-updates/
-rw-r--r--.  1 root root      174 Mar 10 07:42 .vmlinuz.hmac
-rw-------.  1 root root  4510017 Mar 10 07:43 System.map
-rw-r--r--.  1 root root      346 Mar 10 07:43 bls.conf
lrwxrwxrwx.  1 root root       46 Mar 10 07:43 build -> /usr/src/kernels/4.18.0-553.44.1.el8_10.x86_64/
-rw-r--r--.  1 root root   202338 Mar 10 07:43 config
-rw-r--r--.  1 root root   942216 Mar 17 15:22 modules.alias
-rw-r--r--.  1 root root   896333 Mar 17 15:22 modules.alias.bin
-rw-r--r--.  1 root root      478 Mar 10 07:43 modules.block
-rw-r--r--.  1 root root     7930 Mar 10 07:43 modules.builtin
-rw-r--r--.  1 root root    10102 Mar 17 15:22 modules.builtin.bin
-rw-r--r--.  1 root root   344561 Mar 17 15:22 modules.dep
-rw-r--r--.  1 root root   462555 Mar 17 15:22 modules.dep.bin
-rw-r--r--.  1 root root      365 Mar 17 15:22 modules.devname
-rw-r--r--.  1 root root      149 Mar 10 07:43 modules.drm
-rw-r--r--.  1 root root       34 Mar 10 07:43 modules.modesetting
-rw-r--r--.  1 root root     1693 Mar 10 07:43 modules.networking
-rw-r--r--.  1 root root   112117 Mar 10 07:43 modules.order
-rw-r--r--.  1 root root      695 Mar 17 15:22 modules.softdep
-rw-r--r--.  1 root root   495662 Mar 17 15:22 modules.symbols
-rw-r--r--.  1 root root   599539 Mar 17 15:22 modules.symbols.bin
lrwxrwxrwx.  1 root root        5 Mar 10 07:43 source -> build/
-rw-r--r--.  1 root root   404349 Mar 10 07:43 symvers.gz
-rwxr-xr-x.  1 root root 10881352 Mar 10 07:43 vmlinuz*
$ sudo dracut --regenerate-all 
dracut: *** Creating initramfs image file '/boot/initramfs-4.18.0-348.el8.0.2.x86_64.img' done ***
dracut: Executing: /usr/bin/dracut --kver=4.18.0-553.44.1.el8_10.x86_64 -N -v --force
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'btrfs' will not be installed, because command 'btrfs' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut: dracut module 'nbd' will not be installed, because command 'nbd-client' could not be found!
dracut: dracut module 'nvmf' will not be installed, because command 'nvme' could not be found!
dracut: memstrack is available
dracut: *** Including module: bash ***
dracut: *** Including module: systemd ***
dracut: *** Including module: systemd-initrd ***
dracut: *** Including module: modsign ***
dracut: *** Including module: nss-softokn ***
dracut: *** Including module: rdma ***
dracut: *** Including module: rngd ***
dracut: *** Including module: i18n ***
dracut: *** Including module: network-legacy ***
dracut: *** Including module: network ***
dracut: *** Including module: ifcfg ***
dracut: *** Including module: drm ***
dracut: *** Including module: plymouth ***
dracut: *** Including module: prefixdevname ***
dracut: *** Including module: crypt ***
dracut: *** Including module: dm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 60-persistent-storage-dm.rules
dracut: Skipping udev rule: 55-dm.rules
dracut: *** Including module: kernel-modules ***
dracut: *** Including module: kernel-modules-extra ***
dracut: *** Including module: kernel-network-modules ***
dracut: *** Including module: lvm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 56-lvm.rules
dracut: Skipping udev rule: 60-persistent-storage-lvm.rules
dracut: *** Including module: mdraid ***
dracut: Skipping udev rule: 64-md-raid.rules
dracut: *** Including module: multipath ***
dracut: Skipping udev rule: 40-multipath.rules
dracut: Skipping udev rule: 56-multipath.rules
dracut: *** Including module: qemu ***
dracut: *** Including module: qemu-net ***
dracut: *** Including module: cifs ***
dracut: *** Including module: fcoe ***
dracut: *** Including module: fcoe-uefi ***
dracut: *** Including module: iscsi ***
dracut: *** Including module: lunmask ***
dracut: *** Including module: nfs ***
dracut: *** Including module: resume ***
dracut: *** Including module: rootfs-block ***
dracut: *** Including module: terminfo ***
dracut: *** Including module: udev-rules ***
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut: *** Including module: biosdevname ***
dracut: *** Including module: dracut-systemd ***
dracut: *** Including module: usrmount ***
dracut: *** Including module: base ***
dracut: *** Including module: fs-lib ***
dracut: *** Including module: memstrack ***
dracut: *** Including module: microcode_ctl-fw_dir_override ***
dracut:   microcode_ctl module: mangling fw_dir
dracut:     microcode_ctl: reset fw_dir to "/lib/firmware/updates /lib/firmware"
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel"...
intel: model '', path ' intel-ucode/*', kvers ''
dracut:       microcode_ctl: intel: caveats check for kernel version "4.18.0-553.44.1.el8_10.x86_64" passed, adding "/usr/share/microcode_ctl/ucode_with_caveats/intel" to fw_dir variable
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-2d-07"...
intel-06-2d-07: model 'GenuineIntel 06-2d-07', path ' intel-ucode/06-2d-07', kvers ''
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
dracut:       microcode_ctl: intel-06-2d-07: caveats check for kernel version "4.18.0-553.44.1.el8_10.x86_64" passed, adding "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-2d-07" to fw_dir variable
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-4e-03"...
intel-06-4e-03: model 'GenuineIntel 06-4e-03', path ' intel-ucode/06-4e-03', kvers ''
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
intel-06-4e-03: caveat is disabled in configuration
dracut:     microcode_ctl: kernel version "4.18.0-553.44.1.el8_10.x86_64" failed early load check for "intel-06-4e-03", skipping
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-4f-01"...
intel-06-4f-01: model 'GenuineIntel 06-4f-01', path ' intel-ucode/06-4f-01', kvers ' 4.17.0 3.10.0-894 3.10.0-862.6.1 3.10.0-693.35.1 3.10.0-514.52.1 3.10.0-327.70.1 2.6.32-754.1.1 2.6.32-573.58.1 2.6.32-504.71.1 2.6.32-431.90.1 2.6.32-358.90.1'
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
intel-06-4f-01: caveat is disabled in configuration
dracut:     microcode_ctl: kernel version "4.18.0-553.44.1.el8_10.x86_64" failed early load check for "intel-06-4f-01", skipping
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-55-04"...
intel-06-55-04: model 'GenuineIntel 06-55-04', path ' intel-ucode/06-55-04', kvers ''
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
dracut:       microcode_ctl: intel-06-55-04: caveats check for kernel version "4.18.0-553.44.1.el8_10.x86_64" passed, adding "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-55-04" to fw_dir variable
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-5e-03"...
intel-06-5e-03: model 'GenuineIntel 06-5e-03', path ' intel-ucode/06-5e-03', kvers ''
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
dracut:       microcode_ctl: intel-06-5e-03: caveats check for kernel version "4.18.0-553.44.1.el8_10.x86_64" passed, adding "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-5e-03" to fw_dir variable
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-8c-01"...
intel-06-8c-01: model 'GenuineIntel 06-8c-01', path ' intel-ucode/06-8c-01', kvers ''
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
dracut:       microcode_ctl: intel-06-8c-01: caveats check for kernel version "4.18.0-553.44.1.el8_10.x86_64" passed, adding "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-8c-01" to fw_dir variable
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-8e-9e-0x-0xca"...
intel-06-8e-9e-0x-0xca: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
intel-06-8e-9e-0x-0xca: caveat is disabled in configuration
dracut:     microcode_ctl: kernel version "4.18.0-553.44.1.el8_10.x86_64" failed early load check for "intel-06-8e-9e-0x-0xca", skipping
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-8e-9e-0x-dell"...
intel-06-8e-9e-0x-dell: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
dracut:       microcode_ctl: intel-06-8e-9e-0x-dell: caveats check for kernel version "4.18.0-553.44.1.el8_10.x86_64" passed, adding "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-8e-9e-0x-dell" to fw_dir variable
dracut:     microcode_ctl: final fw_dir: "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-8e-9e-0x-dell /usr/share/microcode_ctl/ucode_with_caveats/intel-06-8c-01 /usr/share/microcode_ctl/ucode_with_caveats/intel-06-5e-03 /usr/share/microcode_ctl/ucode_with_caveats/intel-06-55-04 /usr/share/microcode_ctl/ucode_with_caveats/intel-06-2d-07 /usr/share/microcode_ctl/ucode_with_caveats/intel /lib/firmware/updates /lib/firmware"
dracut: *** Including module: shutdown ***
dracut: *** Including module: uefi-lib ***
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies ***
dracut: *** Installing kernel module dependencies done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done***
dracut: *** Hardlinking files ***
dracut: *** Hardlinking files done ***
dracut: *** Generating early-microcode cpio image ***
dracut: *** Constructing AuthenticAMD.bin ***
dracut: *** Constructing GenuineIntel.bin ***
dracut: *** Constructing GenuineIntel.bin ***
dracut: *** Constructing GenuineIntel.bin ***
dracut: *** Constructing GenuineIntel.bin ***
dracut: *** Constructing GenuineIntel.bin ***
dracut: *** Constructing GenuineIntel.bin ***
dracut: *** Constructing GenuineIntel.bin ***
dracut: *** Store current command line parameters ***
dracut: *** Stripping files ***
dracut: *** Stripping files done ***
dracut: *** Creating image file '/boot/initramfs-4.18.0-553.44.1.el8_10.x86_64.img' ***
dracut: *** Creating initramfs image file '/boot/initramfs-4.18.0-553.44.1.el8_10.x86_64.img' done ***

iwalker · March 22, 2025, 9:04am

In rescue mode, assuming there is also network access, you can try and reinstall the kernel and kernel-modules packages for the kernel that is problematic. Perhaps that might help? There are kernel, kernel-core, kernel-modules and kernel-modules-core for each kernel version.

aram535 · March 22, 2025, 12:09pm

@iwalker I was able to do that, no change. The system is fully functional at the moment, just running on that barebone kernel.

I have re-installed the kernels-* packages, re-install grub2, reconfigured/rebuilt the grub2 configuration, regenerated dracut (as I posted in the output above) with no mention os ext4 missing, in fact the module is there.

I even went as far as upgrading the kernel (it’s the same version, newer build)
4.18.0-348.el8.0.2.x86_64 → 4.18.0-553.44.1.el8_10.x86_64

I’m tempted to try to upgrade to 8.9 or 8.10 to see what would happen. The good news at least for the moment I can backup carefully and take my time.

iwalker · March 22, 2025, 12:10pm

I would do the upgrade, since 8.10 is only supported version, all previous versions are not. If you are upgraded and on 8.0 then that is way way old now.