Kernelopts ignored after recent update

Hello guys,

my lovely old inconsistent network device names (eg. ethX) STOPPED working after recent update (427.26->37):

old (working):

old:~# cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-427.26.1.el9_4.x86_64 root=/dev/mapper/rl-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.lvm.lv=rl/root rd.lvm.lv=rl/swap noresume net.ifnames=0 biosdevname=0
old:~# grep kernelopts= /boot/grub2/grub.cfg
  set kernelopts="root=/dev/mapper/rl-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M  rd.lvm.lv=rl/root rd.lvm.lv=rl/swap noresume net.ifnames=0 biosdevname=0 "

new (not working):

[root@sandbox4 ~]# cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-427.37.1.el9_4.x86_64 root=/dev/mapper/rl-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap
[root@sandbox4 ~]# grep kernelopts= /boot/grub2/grub.cfg
  set kernelopts="root=/dev/mapper/rl-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M  rd.lvm.lv=rl/root rd.lvm.lv=rl/swap noresume net.ifnames=0 biosdevname=0 "
[root@sandbox4 ~]# dmesg | grep eth
[    3.117735] virtio_net virtio2 enp2s0: renamed from eth1
[    3.132288] virtio_net virtio1 enp1s0: renamed from eth0

dracut -f didnt help… any help for this please?

Thanks.

Look at directory /boot/loader/entries/. Is there something? If yes, what is in those files?

Thats it… the noresume + biosdevname is missing on the new one:

old (working):

old:~# grep bios  /boot/loader/entries/*
/boot/loader/entries/1a89fb2e83194d3aa3213fc4a60e21e0-0-rescue.conf:options root=/dev/mapper/rl-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M  rd.lvm.lv=rl/root rd.lvm.lv=rl/swap noresume net.ifnames=0 biosdevname=0
/boot/loader/entries/1a89fb2e83194d3aa3213fc4a60e21e0-5.14.0-162.12.1.el9_1.0.2.x86_64.conf:options root=/dev/mapper/rl-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M  rd.lvm.lv=rl/root rd.lvm.lv=rl/swap noresume net.ifnames=0 biosdevname=0
/boot/loader/entries/1a89fb2e83194d3aa3213fc4a60e21e0-5.14.0-284.30.1.el9_2.x86_64.conf:options root=/dev/mapper/rl-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M  rd.lvm.lv=rl/root rd.lvm.lv=rl/swap noresume net.ifnames=0 biosdevname=0
/boot/loader/entries/1a89fb2e83194d3aa3213fc4a60e21e0-5.14.0-427.26.1.el9_4.x86_64.conf:options root=/dev/mapper/rl-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.lvm.lv=rl/root rd.lvm.lv=rl/swap noresume net.ifnames=0 biosdevname=0

new (not working)

[root@sandbox4 entries]# grep bios  /boot/loader/entries/*
[root@sandbox4 entries]# echo $?
1

both are identical systems, both Rocky Linux release 9.4 (Blue Onyx). The only difference is that the new system is updated after 30/7/2024.

Is this a bug or expected behaviour?

Both systems are installed and tuned by same Ansible scripts.

Thank you for response.

EDIT: interesting… 1a89fb2e83194d3aa3213fc4a60e21e0-5.14.0-284.30.1.el9_2.x86_64.conf … so 284.30 are present on both systems, but the new one has the kernelopts missing.

So it is another component who does this.

The problem is still there… I thought it could be a double space in /etc/default/grub… it wasnt.

/boot/grub2/grub.cfg:

looks like this code gets ignored:

if [ -z "${kernelopts}" ]; then
  set kernelopts="root=/dev/mapper/rl-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.lvm.lv=rl/root rd.lvm.lv=rl/swap noresume net.ifnames=0 biosdevname=0 "
fi

When I add it manually to:

/boot/loader/entries/1a89fb2e83194d3aa3213fc4a60e21e0-5.14.0-427.37.1.el9_4.x86_64.conf it works. But it doesnt get added with grub2-mkconfig -o /boot/grub2/grub.cfg.

[root@localhost ~]# cat /boot/loader/entries/1a89fb2e83194d3aa3213fc4a60e21e0-5.14.0-427.37.1.el9_4.x86_64.conf
title Rocky Linux (5.14.0-427.37.1.el9_4.x86_64) 9.4 (Blue Onyx)
version 5.14.0-427.37.1.el9_4.x86_64
linux /vmlinuz-5.14.0-427.37.1.el9_4.x86_64
initrd /initramfs-5.14.0-427.37.1.el9_4.x86_64.img
options root=/dev/mapper/rl-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap net.ifnames=0 biosdevname=0
grub_users $grub_users
grub_arg --unrestricted
grub_class rocky

As you notice, there is no dereference $kernelopts in these entries – the options are there verbatim. Setting the value of kernelopts in grub.cfg or in grubenv has thus no effect.

When you install a new kernel, it effectively runs grubby. The grubby reads /etc/kernel/cmdline and constructs GRUB entry file for the new kernel.

One can modify options in entries with grubby. IIRC, with option --update-kernel=ALL it does update /etc/kernel/cmdline too. Chapter 4. Configuring kernel command-line parameters | Red Hat Product Documentation

One can also modify /etc/default/grub and then run grub-mkconfig, but one has to add --update-bls-cmdline for it to update the GRUB entries (and /etc/kernel/cmdline). 9.3 Release Notes | Red Hat Product Documentation


The interesting part is what made the latest installed kernel get different options than the previous kernels in the same system,if /etc/kernel/cmdline has not changed?

2 Likes

thats it. Love ya! <3

So much thanks! Didnt notice it in 9.3 release notes.

I think it is GRUB_ENABLE_BLSCFG, which became effective. It broke the things during update to new kernel. Ansible snippet to fix it below.

- name: disable GRUB_ENABLE_BLSCFG
  ini_file:
    path: /etc/default/grub
    section: ""
    option: "GRUB_ENABLE_BLSCFG"
    value: "false"
    no_extra_spaces: yes
    state: present
  notify: reboot

I did fix my Ansible play by adding the --update-bls-cmdline.

Not sure if it is enough.

If I understand that docs properly, without GRUB_ENABLE_BLSCFG=false there is a risk of losing your kernelopts with each subsequent kernel update…?

Shouldn’t be. The grubby (called by post-install script of the kernel package) merely reads the /etc/kernel/cmdline to generate one new BLS entry file.


With GRUB_ENABLE_BLSCFG=false all the menu entries are in the grub.cfg and grubby has to modify that file for each new (and removed) kernel. That sounds more risky than “add file”.

Looks like you’re right. I’ve just tested updating older installation (but still 9.4 from the beginning) and the our kernelopts are present in the /boot/loader/entries/ for the new kernel.

So running grub2-mkconfig --update-bls-cmdline -o /boot/grub2/grub.cfg looks like the only requirement in the brand new system.

So I’m discarding that switching of GRUB_ENABLE_BLSCFG to false.

The only issue left is whether I should switch the code to grubby completely, I mean it the editing of /etc/default/grub is still a modern and optimal way how to handle the kernelopts for ALL present and future kernels.

The question is about maintaining consistency and what is customized. You probably want the /etc/default/grub to match what is actually used.


man grubby writes:

--update-kernel=kernel-path
The entries for kernels matching kernel-path are updated. Currently the only items that can be updated is the kernel argument list, which is modified via the --args and --remove-args options. If the ALL argument is used the variable GRUB_CMDLINE_LINUX in /etc/default/grub is updated with the latest kernel argument list, unless the --no-etc-grub-update option is used or the file does not exist (e.g., on s390x).

man grub2-mkconfig writes:

--update-bls-cmdline
overwrite BLS cmdline args with default args


  • The grub2-mkconfig recreates entire grub.cfg (and optionally BLS entries and /etc/kernel/cmdline) based on /etc/default/grub
  • The grubby updates BLS entries and GRUB_CMDLINE_LINUX in /etc/default/grub – with --update-kernel=ALL

If you want to update/set the cmdline for all installed kernels, then both grubby and grub2-mkconfig can do it, but the latter rewrites the grub.cfg unnecessarily (when BLS is used). Both can keep the /etc/default/grub and /etc/kernel/cmdline consistent.

If you want to update other features of GRUB via /etc/default/grub, then you have to use grub2-mkconfig.

Since GRUB_CMDLINE_LINUX and other variables in /etc/default/grub are probably managed by separate Ansible tasks, you can use the most appropriate tool in each task.


PS. When el9.3 was released, I had the dilemma that Ansible task that did call grub2-mkconfig on 9.2 systems could not use --update-bls-cmdline, but on 9.3 systems the option was necessary. I still don’t use grubby, although – for reasons mentioned above – it would be “better” for some tasks.