How do I remove crashkernel from cmdline?

I don’t like crashkernel reserving 256MB of memory on my cloud vm. There’s no point! VMs rarely crash. It’s just a waste of precious memory.

I couldn’t find a way to prevent crashkernel args from getting to BLS entries. It seems that the devs didn’t think this through. The BLS entries are generated by installkernel. It pulls the param using an undocumented feature of kdumpctl(/usr/lib/kernel/install.d/92-crashkernel.install).

All the materials on the internet are outdated as /etc/defaults/grub has nothing to do with BLS entries anymore. I can always remove/hide 92-crashkernel.install, but there’s no guarantee that the package responsible for the file(kexec-tools) won’t get updated and dnf won’t install it again.

What am I missing here? Just posting before filing a bug report.

Have you tried using grubby?

## This is on Rocky Linux 8, but applies to 9 as well.

[root@cm01 ~]# grubby --info /boot/vmlinuz-4.18.0-513.18.1.el8_9.x86_64
index=0
kernel="/boot/vmlinuz-4.18.0-513.18.1.el8_9.x86_64"
args="ro crashkernel=auto resume=/dev/mapper/rockyvg-swap rd.lvm.lv=rockyvg/root rd.lvm.lv=rockyvg/swap rhgb quiet audit=1 $tuned_params"
root="/dev/mapper/rockyvg-root"
initrd="/boot/initramfs-4.18.0-513.18.1.el8_9.x86_64.img $tuned_initrd"
title="Rocky Linux (4.18.0-513.18.1.el8_9.x86_64) 8.9 (Green Obsidian)"
id="e8c646ccc5f644c5957eea81d8201810-4.18.0-513.18.1.el8_9.x86_64"
[root@cm01 ~]# grubby --remove-args="crashkernel" --update=ALL
[root@cm01 ~]# grubby --info /boot/vmlinuz-4.18.0-513.18.1.el8_9.x86_64
index=0
kernel="/boot/vmlinuz-4.18.0-513.18.1.el8_9.x86_64"
args="ro resume=/dev/mapper/rockyvg-swap rd.lvm.lv=rockyvg/root rd.lvm.lv=rockyvg/swap rhgb quiet audit=1 $tuned_params"
root="/dev/mapper/rockyvg-root"
initrd="/boot/initramfs-4.18.0-513.18.1.el8_9.x86_64.img $tuned_initrd"
title="Rocky Linux (4.18.0-513.18.1.el8_9.x86_64) 8.9 (Green Obsidian)"
id="e8c646ccc5f644c5957eea81d8201810-4.18.0-513.18.1.el8_9.x86_64"

See this article for more information.

You didn’t understand the problem. I know how to remove args from individual BLS loaders. Future kernels installed by the package managers will still have crashkernel args.

I just disable the kdump service and then it’s no longer using it.

systemctl stop kdump
systemctl disable kdump

also the entries in grub are removed and no longer will there be files relating to it under /boot.

Disabling kdump.service does not magically turn off the crashkernel cmdline or reclaim the memory reserved for crashkernel during boot time. The only way to reclaim it is reloading the kernel with crashkernel option turned off.

[    0.162689] Reserving 256MB of memory at 512MB for crashkernel (System RAM: 16382MB)

My AlmaLinux 8 does still have entries like:

title AlmaLinux (4.18.0-513.18.1.el8_9.x86_64) 8.9 (Midnight Oncilla)
version 4.18.0-513.18.1.el8_9.x86_64
linux /vmlinuz-4.18.0-513.18.1.el8_9.x86_64
initrd /initramfs-4.18.0-513.18.1.el8_9.x86_64.img $tuned_initrd
options $kernelopts $tuned_params
id almalinux-20240222081209-4.18.0-513.18.1.el8_9.x86_64
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel

Note the $kernelopts that is from grubenv:

# grub2-editenv list
saved_entry=9c..ce-4.18.0-513.18.1.el8_9.x86_64
kernelopts=root=UUID=ce..f2 ro resume=UUID=1d..ad nomodeset intel_iommu=on iommu=pt
boot_success=0

if your new entries do use $kernelopts, then update the grubenv. (And/or update /etc/default/grub and use grub2-mkconfig.)


PS. EL9 does things differently.

PS. EL9 does things differently.

Yes, Rocky 9 does it different by including the cmdline in each bls loader conf. I think AlmaLinux does it elegantly, too. I like the idea of BLS. The problem is that RHEL devs thought too much about customizability.

OK, so I checked like you, and yes it seems I have:

[root@rocky9 ~]# dmesg | grep -i crash
[    0.000000] Command line: BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-5.14.0-362.18.1.el9_3.x86_64 root=UUID=b5e3355a-1ec0-44c0-bec7-fba810bf7b3e ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
[    0.027658] Reserving 256MB of memory at 1776MB for crashkernel (System RAM: 4095MB)
[    0.062017] Kernel command line: BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-5.14.0-362.18.1.el9_3.x86_64 root=UUID=b5e3355a-1ec0-44c0-bec7-fba810bf7b3e ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M

So using some of the posts above with a slight change of setting crashkernel=no:

[root@rocky9 ~]# grubby --update-kernel=ALL --args="crashkernel=no"

[root@rocky9 ~]# grubby --info /boot/vmlinuz-5.14.0-362.18.1.el9_3.0.1.x86_64 
index=1
kernel="/boot/vmlinuz-5.14.0-362.18.1.el9_3.0.1.x86_64"
args="ro crashkernel=no"
root="UUID=b5e3355a-1ec0-44c0-bec7-fba810bf7b3e"
initrd="/boot/initramfs-5.14.0-362.18.1.el9_3.0.1.x86_64.img"
title="Rocky Linux (5.14.0-362.18.1.el9_3.0.1.x86_64) 9.3 (Blue Onyx)"
id="269bb085de0744ab8477fd952e23fbc4-5.14.0-362.18.1.el9_3.0.1.x86_64"

we can now see crashkernel=no, and so reboot:

[root@rocky9 ~]# dmesg | grep -i crash
[    0.000000] Command line: BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-5.14.0-362.24.1.el9_3.x86_64 root=UUID=b5e3355a-1ec0-44c0-bec7-fba810bf7b3e ro crashkernel=no
[    0.026246] crashkernel: memory value expected
[    0.052301] Kernel command line: BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-5.14.0-362.24.1.el9_3.x86_64 root=UUID=b5e3355a-1ec0-44c0-bec7-fba810bf7b3e ro crashkernel=no

no crashkernel or reserved memory.

EDIT: Although now, after doing the command @nazunalika gave:

grubby --remove-args="crashkernel=no" --update=ALL

no output in dmesg for crashkernel whatsoever.

Cool!

Now, wait for a kernel update and see that the crashkernel=... option appear in the new loader entry for the new version of kernel. The only way to prevent it from happening is deleting/disabling in Rocky 9 /usr/lib/kernel/install.d/92-crashkernel.install which is a cheap hack as mentioned in the original post. My question was what’s the “right way” that devs as intended and designed?

Yeah just noticed that when installing kernel-ml. Give me a sec, gonna try and figure out the right way.

Actually this is weird, since, the section here: Chapter 7. Making persistent changes to the GRUB boot loader Red Hat Enterprise Linux 8 | Red Hat Customer Portal

mentions using grub2-editenv, as well as the commands we already listed above which are supposed to set the defaults for existing and new kernels, and it doesn’t work.

That would seem to be the correct way, but for some bizarre reason it’s not working as intended.

IIRC, there were some “fluctuation” on whether (BLS) entry has $kernelopts on the options line or the “hardcoded” options verbatim. Was it on EL8, were I confused by EL9? I seem to recall reading bugzilla about it. (I was confused by EL9’s grub initially.)

The former does use that variable from grubenv, has thus same set of options for all entries, and install of kernel does not need to do anything to “copy” the options.
The latter allows each entry to have different options, but the install of new kernel has to copy those options from somewhere.

The EL9 does use the latter method and apparently copies options from /etc/kernel/cmdline
grub2-mkconfig --update-bls-cmdline seems to update the /etc/kernel/cmdline (and BLS entries).
The --update-bls-cmdline was added in el9_3.

2 Likes

Now we’re getting somewhere! Sorry, I think I’ve found it myself…

https://git.centos.org/rpms/kexec-tools/c/4335b50c0d914ba8c72a42cbdc170bad444b11ed
https://src.fedoraproject.org/rpms/kexec-tools/blob/rawhide/f/kdumpctl

_should_reset_crashkernel() {
	
	[[ $(kdump_get_conf_val auto_reset_crashkernel) != no ]] && systemctl is-enabled kdump &> /dev/null
	
}
 -     reset-crashkernel-for-installed_kernel)
+     _reset-crashkernel-for-installed_kernel)
         if [[ $(kdump_get_conf_val auto_reset_crashkernel) != no ]]; then
             reset_crashkernel_for_installed_kernel "$2"
         fi

Use auto_reset_crashkernel or just disable kdump.service like you said. AND then regenerate the BLS entries. That should do it. If not, this is clearly a bug.

This whole thing has to be documented somewhere!

What a nightmare. One hell of a rubberduck debugging we did here.

1 Like

9.3?! That explains why there’s little to none docs. Not a single blog post talking about it.

Indeed. The Release Notes (that we all read so carefully) is a the only place I know: Chapter 4. New features Red Hat Enterprise Linux 9 | Red Hat Customer Portal

1 Like

Using grubby with the --updata=ALL option should create this file:
/etc/kernel/cmdline
The “kernel-install” script reads this file to populate the kernel parameters on new kernel installs. The most important parameter that must be in this file is the “root=<path to / and subvolume or UUID and subvolume>” .
If it doesn’t then new kernels won’t boot.

So, to summarize it - for EL9:

## Must be disabled
# systemctl is-enabled kdump.service 
disabled
# grub2-mkconfig --update-bls-cmdline > /etc/grub2.cfg

Cross check via:

# grep -f /etc/kernel/cmdline /boot/loader/entries/*

No, you have to use grubby to make changes to the BLS entries.

Nothing worked. crashkernel option still makes it to the BLS. You can test it by reinstalling the kernel.

dnf reinstall kernel kernel-core

It’s a dead end. If your distro allows, you can always remove kexec-tools. Or put exit 0 in the kernel install hook. Couldn’t find a better way.

Let’s wait a bit more to see if someone with the secret comes along.

Following needs to be done …

sed -i 's/^auto_reset_crashkernel yes/auto_reset_crashkernel no/' /etc/kdump.conf

Mate, you haven’t tested it yourself. Do you even see what the problem is? Have you reinstalled/upgraded the kernel to see if that worked? Well, last time I checked, it didn’t.