New kernel doesn't show up on GRUB startup list (9.5 to 9.6)

Hello everyone,

I’m running Rocky Linux 9.5 (5.14.0-503.38.1.el9_5.x86_64) on a VM (vSphere). Today, I ran a yum upgrade to install the new kernel 9.6 (5.14.0-570.55.1.el9_6.x86_64).

After the command ran successfully, I rebooted the VM, but the new kernel didn’t appear in the GRUB menu—only the old one. When I listed the kernel, I saw that I already have one on 9.6 (5.14.0-570.49.1.el9_6.x86_64) that didn’t show up either.

When I run a grubby –default-kernelThe default version is 5.14.0-570.55.1.el9_6.x86_64. Running uname -r: 5.14.0-503.38.1.el9_5.x86_64.

Any suggestions? I tried grub2-mkconfig on both /boot/grub2 and /boot/efi/EFI/rocky, didn’t help at all.

Best regards,

Wrong target if this is an UEFI system. In a terminal while booted into the subject VM issue this command, can be as normal user.

lsblk -f

to confirm your drive structure and get the UUID of /boot. You are going to have to restore the grub.cfg stub file in /boot/efi/EFI/rocky/grub.cfg to it’s original format so the boot loader can find the config files in /boot/loader/entries/
Which should look like this:

search --no-floppy --root-dev-only --fs-uuid --set=dev b2ad8362-352a-440e-904c-c35dca855f3b
set prefix=($dev)/boot/grub2
export $prefix
configfile $prefix/grub.cfg

where the UUID after --set=dev is that of your /boot partition if you did the default rocky install.

So run the lsblk command as stated above and paste the output here in a code block, the </> symbol above. You might create a test stub grub.cfg and post its contents here in a code block also.

Hello @jbkt23, the lsblk -f as you requested

NAME            FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1          xfs                        a1e22952-62c9-4b02-beb1-bfbada14db20
└─sda2          LVM2_member LVM2 001       7GZn5v-9R8i-Pz1m-kjmi-YEv0-CqqH-5DZ0pj
  ├─rk9_vg-root xfs                        5dd96f2f-60dc-4d5b-a1d8-e2f99dfe8852    190.8G     8% /
  ├─rk9_vg-swap swap        1              3fe3308f-ff63-4d3c-9415-4a0ea1de08ab                  [SWAP]
  ├─rk9_vg-tmp  xfs                        35d9605a-178e-426a-b1f0-d14f5162d002     50.5G     1% /tmp
  ├─rk9_vg-var  xfs                        f9b33b81-bb42-442f-866b-cabc1fb4ae87    149.9G     4% /var
  ├─rk9_vg-home xfs                        5559a4ce-25ee-4b5d-8ad4-351a9225cc3a     10.8G     1% /home
  └─rk9_vg-boot xfs                        e8753b20-30a0-4264-ae6b-0e98a74906f4    665.2M    31% /boot

Best regards,

So this does not appear to be a default rocky install partition layout for a UEFI system. No vfat esp partition and no boot partition outside the vg group. I’m stumped. It must be bios boot but I can’t figure out how grub finds the kernel inside a volume group.

It probably does not.

The (bios boot) assumption is that vSphere loads first part of bootloader from sector 0 of the “sda” volume. Then the first part loads rest of bootloader from somewhere, and bootloader loads menu and kernel from that same location.

We do usually see the somewhere location as /boot. However, the rk9_vg-boot is a LVM LV and first part of bootloader cannot load from LV. It is more likely that GRUB, menu, and kernel (that loads) are in “sda1”.

One can mount the sda1 to some path and look what it has. For example:

sudo mkdir /mnt/test
sudo mount -o ro -v /dev/sda1 /mnt/test
ls /mnt/test

So jumping ahead for my own edification, if sda1 does turn out to be a boot partition then it should have been mounted to the vg-boot partition in the volume group. So since it is not it only contains the original install kernel while the subsequent kernel updates are all on vg-boot and thus can’t be found.
That’s the speculation anyway.

I would rephrase that:

If sda1 contains the filesystem that the boot process actually reads the bootloader, kernel and initramfs from, then it should be mounted as /boot in order for package updates to write new kernels and their boot menu entries into correct filesystem. The vg-boot filesystem is mounted as /boot now. That is why the new kernels are there, not in the filesystem that the boot process does use.

The “sda1” and “vg-boot” are block devices that contain filesystems.
The /, /boot, /home are directories in the directory tree.
Filesystem can be mounted to directory, which converts that directory into mountpoint.


Note on rk9_vg-var /var:
Red Hat writes that /var (and /usr) on distinct filesystem(s) can make early stages of boot process more complex, because the process may access some bits in them. Subdirs of /var ought to be safer to be mountpoints.

@jlehtone You are correct. The boot entries are on /dev/sda1. I tested the mount you suggested, and voila. Now, what I see is exactly what I see in the GRUB menu at startup.

Any suggestions on how to bring the new kernels here?

Maybe point /dev/sda1 to /boot in fstab and reinstall the kernel with dnf?

Thanks!

Yes, one can run dnf reinstall name_of_package

One can also list packages that are “per kernel version”:

dnf rq --installonly

and limit that to N latest versions of each package (here N=2):

dnf rq --installonly --latest=2

So one could uninstall those kernel packages before one remounts /boot:

dnf rm $(dnf rq --installonly --latest=2)

However, kernel packages are not the only ones with something in /boot.
One could list all packages that have something in there:

find /boot -exec rpm -qf '{}' \; | grep -v "is not owned by any package" | sort -u

That would be a list of packages to reinstall.

There are also files that are not from any package but have been created by something.


As for update of /etc/fstab, I’d recommend:

  1. umount /boot
  2. Replace the ‘/boot’ line with:
UUID=a1e22952-62c9-4b02-beb1-bfbada14db20  /boot  xfs  defaults  1 2
  1. mount /boot

You can still mount the rk9_vg-boot to somewhere else to compare/copy files (if necessary).