ran updates over the weekend, everything appears to have worked, but got messages that it was out of diskspace when building the initial ramdisk. So, of course it now won’t boot. the recovery kernel won’t boot either, giving the same error message as trying to boot the latest kernel, claiming it can’t find the initial ramdisk. but it’s there, I can see it.
complications are that /, /boot, swap and /home are each separate RAID1 partitions, and worse, it is an XFS filesystem on /boot, /root and/home. there’s lots of spare diskspace on /home/ but there is no unused space on the drive. Since you can’t shrink anXFS filesystem I can’t (easily) use gparted to move things around to get some extra space for /boot.
However / has multiple gigs unused, too, so I was thinking of just copying /boot to a directory (named boot, of course) on the / partition. I think I can make that work.
So, assuming all that goes right, I will stillhave the problem with broken initial ramdisk, and since I can’t boot it up how can I run a dnf update on it that will populate the new /boot rather than the one on the CD I use to boot up the system? Do I need to do something like chroot?
/boot is normally a 1G xfs partition, are there any files from older kernels that you can delete? What exact space is being used (not used)? It’s a bit strange none of the older kernels are working, as those files should be “as they were” even if the partition got full.
/boot is 600 MB, probably my fault since I was manually configuring the partitions. to the best of my memory (which isn’t great at the best of times, and since it was 2+ years ago) I spent some skull sweat figuring out what sizes to use based on my old C7 system, trying to leave extra room. clearly I was wrong.
and, no, all that’s left of the kernels are the failed install and the rescue kernel (with its initial ram disk). I have no clue why it claims it can’t find the rescue ram disk.
someone else with a similar problem has a posting here in which they reported that if they defragged the rescue ram disk that the system then booted, but this one doesn’t.
I also used to configure /boot at 512MB but now it’s not enough, especially with multiple kernels installed. About the only thing you can do right now, is remove the old kernels and only leave the last one or two instead of the three that it defaults to. That will at least solve when you update, that it will be able to bring in the new kernel. Once the old kernels have been removed, reinstall the last kernel so that it generates it’s initramfs. Since you can’t boot, you’ll need to boot from Rocky ISO in rescue mode, chroot into your install and do it that way.
Isn’t that a “Live” image? Those have OS installed on the stick/CD so one can run Rocky (in this case with Mate) without touching any storage inside the PC. One can start Rocky installer from such session.
The above-mentioned “ISO” does not offer “desktop session”. Instead, it starts the Rocky installer (or alternatively a shell for rescue).
Since you run “full Rocky” from the CD, you can there poke the “dead beef” inside the machine. All the tools ought to be there for messing with non-mounted filesystems, etc …
Should be available from the grub menu before booting the ISO. I usually use the mininal or DVD installation ISO’s rather than the LiveCD’s, so if the LiveCD doesn’t have the rescue option available during boot, then one of the others should have it.
Thanks to all of you I’ve gotten recovered to the point where the vmlinuz-5.14.0-570.26.1.el9_6.x86_64 kernel boots and appears fine. The 570.28.1 will not boot because there isn’t enough disk space to build the initramfs, just like before.
Experimenting I find that the 570.23.1 kernel also fails to boot (whyever, I don’t know) and also the rescue kernel also fails to boot. I’ve been trying to find out where the rescue kernel comes from so I can try reinstalling it, but I can’t find any package that claims to own it. Anybody got a clue on this?
It looks as if the nvidia drivers are messed up too but I don’t want to try to fix that until I can get both the proper latest kernel AND the rescue kernel working.
Though you still need a 1GIG “/boot” filesystem now (for just about all Linux distributions), you can reduce the amount of space used by doing the following…
In /etc/yum.conf, set the number of retained ‘previous’ kernels and initramfs files you want…
(for ‘home’ use, I only keep ‘1’, and have never had to fallback to it)
Package dracut-config-rescue configures the ‘dracut’ to create rescue kernel when new kernel is installed (if there is no rescue kernel yet).
It creates three files that ls /boot/*rescue* /boot/loader/entries/*rescue* should show. You can simply delete those files.
The rescue kernel is simply a copy of an installed kernel but the initramfs for it is much larger, because it has more drivers/whatever in it.
For each installed kernel (but not the rescue) there is probably also a second, larger initramfs: /boot/initramfs-${kernelversian}kdump.img
The kdump service (if enabled), supposedly can reboot the machine with kdump version of initramfs in case of kernel error to facilitate debug of the error.
To prevent creation of the kdump.img for each installed kernel I had to do two things:
Disable kdump.service
Set auto_reset_crashkernel no in /etc/kdump.conf (as it is ‘yes’ by default)
That is a good question. I do recall trying to use the rescue kernel once (and failed), so don’t even know what it can do. Besides, if something breaks the bootloader, then one cannot even load the rescue and has to use the ISO anyway.
The package dracut-config-rescue adds the file /etc/dracut.conf.d/02-rescue.conf
with one line in it: dracut_rescue_image="yes"
If you remove that package, then you remove the file /etc/dracut.conf.d/02-rescue.conf and no more rescue kernels will be created.
The package dracut-config-rescue adds the file /etc/dracut.conf.d/02-rescue.conf
with one line in it: dracut_rescue_image="yes"
It is better to leave this file and change the value to “no”. Reason being that if there is an update to dracut this file might get added back unbeknownst to you. With the file changed and existing an update should add a same name file with the suffix “.rpmnew” without affecting your changes to the original conf file.
Well that implies that some other updated packages would create a requirement to re-install package (dracut-config-rescue) that we have removed. That is possible. Currently no package requires the dracut-config-rescue. It gets installed, because it is a default package for group Core (and Core is mandatory even in Minimal Install).
Yes, existing modified config file does not take any chances on the matter.
Another option for squeezing initramfs image files into /boot is to configure a different compression method. Create a config in /etc/dracut.conf.d/ or add a line to /etc/dracut.confcontaining the line, e.g. compress=xz, and forcefully regenerate the image files for your kernels.
See dracut.conf(5) for details.
Compression options may include bzip2, lzma, xz, gzip, lzop, lz4, and zstd. Try each of them to determine which makes the smallest image files. You will need the userland tool installed in order to perform the compression.
Illustrative partial output of dracut above is from Fedora.
I set it to use xz then ran “dnf update”, which did its thing, removing the .28 kernel and installing the .30. Or so it said.
when it was done and I rebooted, the grub menu still showed the .28 as present, and not surprisingly it still won’t boot. No sign of the .30 kernel there.
wonder what I messed up?
Fred
PS: the .26 kernel is the only one that is now bootable, here’s hoping that any further attempts won’t remove it before I get another one working…
It sounds like your underlying problem remains and with you only telling, not showing, us what’s happening leaves us to speculate.
If I was cleaning up this mess, I’d start by removing all kernel packages and related files in /boot except those for the current bootable kernel, then verify the installed (bootable) kernel packages, remove any straggler packages and clean all package things and reboot with an SELinux automatic relabel just for fun [so be patient].
