How do I remove crashkernel from cmdline?

Nope, doesn’t work. I just did it, editing the kdump.conf and set it to no. Then installed kernel-ml to test it, still crashkernel enabled. Just confirming. None of what we tested so far, including this doesn’t work.

I also realised that /etc/kernel/install.d is analogous to /usr/lib/kernel/install.d - so similar to like putting your own systemd services in /etc/systemd/ instead of under /usr/lib/systemd. I even tried doing the 92-crashkernel thing there by creating a script just to exit 0 and then making it executable, and that also didn’t work. That would have at least been better than editing the one under /usr/lib/kernel/install.d anyway to try and make it a bit more of an official way of doing it.

Even using a combo of grub-mkconfig --update-bls-cmdline and grubby doesn’t work either. It’s like every single option is completely crippled.

I have kdump service disabled anyway which removed the additional files that were usually generated under /boot each time a kernel was installed. Whether that actually helps any, even when dmesg still shows crashkernel memory as being allocated I’ve no idea. But I’m OK with my VM memory usage anyway at present. Was just attempting to figure a way of fixing it to help you guys out.

1 Like

I think we are missing some additional preconditions here. My tests shows that a reinstall of the current running kernel does not get the bls+crashkernel entries added. Enabling auto_reset_crashkernel introduces the crashkernel entry again when doing a reinstall. My context does not have any crashkernel entry in /etc/default/grub, /etc/grub2.cfg, /proc/cmdline, /etc/kernel/cmdline. GRUB_ENABLE_BLSCFG=true is set in /etc/default/grub.

Now try this installing a completely new kernel that isn’t on the system. I did this pulling a kernel from elrepo, the parameters appear. So, if you try this and they don’t for you, then that means something else is different on your system. Existing kernels aren’t the problem here. It’s installing new ones.

Sure, here we go:


[root@rh9 ~]# grep ^PR /etc/os-release 
PRETTY_NAME="Red Hat Enterprise Linux 9.3 (Plow)"


[root@rh9 ~]# uname -a
Linux rh9.localdomain 5.14.0-362.24.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Feb 15 07:18:13 EST 2024 x86_64 x86_64 x86_64 GNU/Linux


[root@rh9 ~]# rpm -q kernel
kernel-5.14.0-362.13.1.el9_3.x86_64
kernel-5.14.0-362.18.1.el9_3.x86_64
kernel-5.14.0-362.24.1.el9_3.x86_64


[root@rh9 ~]# grep -f /etc/kernel/cmdline /boot/loader/entries/*
/boot/loader/entries/70cd3f15f6a74555bae071621a01e74b-5.14.0-362.13.1.el9_3.x86_64.conf:options root=UUID=00843ae4-939d-4d41-946e-45ef756d57d3 ro rd.luks.uuid=luks-f302d431-2e80-43be-9790-6d5a5fcd9cd9 rd.lvm.lv=storage/root rd.shell=0 rhgb quiet  $tuned_params
/boot/loader/entries/70cd3f15f6a74555bae071621a01e74b-5.14.0-362.18.1.el9_3.x86_64.conf:options root=UUID=00843ae4-939d-4d41-946e-45ef756d57d3 ro rd.luks.uuid=luks-f302d431-2e80-43be-9790-6d5a5fcd9cd9 rd.lvm.lv=storage/root rd.shell=0 rhgb quiet  $tuned_params
/boot/loader/entries/70cd3f15f6a74555bae071621a01e74b-5.14.0-362.24.1.el9_3.x86_64.conf:options root=UUID=00843ae4-939d-4d41-946e-45ef756d57d3 ro rd.luks.uuid=luks-f302d431-2e80-43be-9790-6d5a5fcd9cd9 rd.lvm.lv=storage/root rd.shell=0 rhgb quiet $tuned_params


[root@rh9 ~]# yum update  --enablerepo=*-stream kernel
Updating Subscription Management repositories.
Last metadata expiration check: 0:01:28 ago on Thu Mar 28 16:05:03 2024.
Dependencies resolved.
===================================================================================================================================================================================================================
 Package                                            Architecture                          Version                                              Repository                                                     Size
===================================================================================================================================================================================================================
Installing:
 kernel                                             x86_64                                5.14.0-432.el9                                       baseos-stream                                                  66 k
 kernel-core                                        x86_64                                5.14.0-432.el9                                       baseos-stream                                                  15 M
 kernel-modules                                     x86_64                                5.14.0-432.el9                                       baseos-stream                                                  33 M
 kernel-modules-core                                x86_64                                5.14.0-432.el9                                       baseos-stream                                                  27 M
Removing:
 kernel                                             x86_64                                5.14.0-362.13.1.el9_3                                @rhel-9-for-x86_64-baseos-rpms                                  0  
 kernel-core                                        x86_64                                5.14.0-362.13.1.el9_3                                @rhel-9-for-x86_64-baseos-rpms                                 63 M
 kernel-modules                                     x86_64                                5.14.0-362.13.1.el9_3                                @rhel-9-for-x86_64-baseos-rpms                                 31 M
 kernel-modules-core                                x86_64                                5.14.0-362.13.1.el9_3                                @rhel-9-for-x86_64-baseos-rpms                                 26 M

Transaction Summary
===================================================================================================================================================================================================================
Install  4 Packages
Remove   4 Packages

Total download size: 76 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): kernel-5.14.0-432.el9.x86_64.rpm                                                                                                                                            304 kB/s |  66 kB     00:00    
(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm                                                                                                                                       3.9 MB/s |  15 MB     00:03    
(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm                                                                                                                                    2.7 MB/s |  33 MB     00:12    
(4/4): kernel-modules-core-5.14.0-432.el9.x86_64.rpm                                                                                                                               2.2 MB/s |  27 MB     00:12    
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                              5.7 MB/s |  76 MB     00:13     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                           1/1 
  Installing       : kernel-modules-core-5.14.0-432.el9.x86_64                                                                                                                                                 1/8 
  Installing       : kernel-core-5.14.0-432.el9.x86_64                                                                                                                                                         2/8 
  Running scriptlet: kernel-core-5.14.0-432.el9.x86_64                                                                                                                                                         2/8 
  Installing       : kernel-modules-5.14.0-432.el9.x86_64                                                                                                                                                      3/8 
  Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64                                                                                                                                                      3/8 
  Installing       : kernel-5.14.0-432.el9.x86_64                                                                                                                                                              4/8 
  Erasing          : kernel-5.14.0-362.13.1.el9_3.x86_64                                                                                                                                                       5/8 
  Running scriptlet: kernel-5.14.0-362.13.1.el9_3.x86_64                                                                                                                                                       5/8 
  Erasing          : kernel-modules-5.14.0-362.13.1.el9_3.x86_64                                                                                                                                               6/8 
  Running scriptlet: kernel-modules-5.14.0-362.13.1.el9_3.x86_64                                                                                                                                               6/8 
  Erasing          : kernel-modules-core-5.14.0-362.13.1.el9_3.x86_64                                                                                                                                          7/8 
  Running scriptlet: kernel-modules-core-5.14.0-362.13.1.el9_3.x86_64                                                                                                                                          7/8 
  Running scriptlet: kernel-core-5.14.0-362.13.1.el9_3.x86_64                                                                                                                                                  8/8 
/usr/sbin/weak-modules: line 1086: cd: /lib/modules/5.14.0-362.13.1.el9_3.x86_64/weak-updates: No such file or directory

  Erasing          : kernel-core-5.14.0-362.13.1.el9_3.x86_64                                                                                                                                                  8/8 
warning: file /lib/modules/5.14.0-362.13.1.el9_3.x86_64/modules.builtin.modinfo: remove failed: No such file or directory
warning: file /lib/modules/5.14.0-362.13.1.el9_3.x86_64/modules.builtin: remove failed: No such file or directory

  Running scriptlet: kernel-core-5.14.0-362.13.1.el9_3.x86_64                                                                                                                                                  8/8 
  Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64                                                                                                                                                 8/8 
  Running scriptlet: kernel-core-5.14.0-432.el9.x86_64                                                                                                                                                         8/8 
dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y

  Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64                                                                                                                                                      8/8 
  Running scriptlet: kernel-core-5.14.0-362.13.1.el9_3.x86_64                                                                                                                                                  8/8 
  Verifying        : kernel-5.14.0-432.el9.x86_64                                                                                                                                                              1/8 
  Verifying        : kernel-core-5.14.0-432.el9.x86_64                                                                                                                                                         2/8 
  Verifying        : kernel-modules-5.14.0-432.el9.x86_64                                                                                                                                                      3/8 
  Verifying        : kernel-modules-core-5.14.0-432.el9.x86_64                                                                                                                                                 4/8 
  Verifying        : kernel-5.14.0-362.13.1.el9_3.x86_64                                                                                                                                                       5/8 
  Verifying        : kernel-core-5.14.0-362.13.1.el9_3.x86_64                                                                                                                                                  6/8 
  Verifying        : kernel-modules-5.14.0-362.13.1.el9_3.x86_64                                                                                                                                               7/8 
  Verifying        : kernel-modules-core-5.14.0-362.13.1.el9_3.x86_64                                                                                                                                          8/8 
Installed products updated.

Installed:
  kernel-5.14.0-432.el9.x86_64                  kernel-core-5.14.0-432.el9.x86_64                  kernel-modules-5.14.0-432.el9.x86_64                  kernel-modules-core-5.14.0-432.el9.x86_64                 
Removed:
  kernel-5.14.0-362.13.1.el9_3.x86_64           kernel-core-5.14.0-362.13.1.el9_3.x86_64           kernel-modules-5.14.0-362.13.1.el9_3.x86_64           kernel-modules-core-5.14.0-362.13.1.el9_3.x86_64          

Complete!




[root@rh9 ~]# grep -f /etc/kernel/cmdline /boot/loader/entries/*
/boot/loader/entries/70cd3f15f6a74555bae071621a01e74b-5.14.0-362.18.1.el9_3.x86_64.conf:options root=UUID=00843ae4-939d-4d41-946e-45ef756d57d3 ro rd.luks.uuid=luks-f302d431-2e80-43be-9790-6d5a5fcd9cd9 rd.lvm.lv=storage/root rd.shell=0 rhgb quiet  $tuned_params
/boot/loader/entries/70cd3f15f6a74555bae071621a01e74b-5.14.0-362.24.1.el9_3.x86_64.conf:options root=UUID=00843ae4-939d-4d41-946e-45ef756d57d3 ro rd.luks.uuid=luks-f302d431-2e80-43be-9790-6d5a5fcd9cd9 rd.lvm.lv=storage/root rd.shell=0 rhgb quiet $tuned_params
/boot/loader/entries/70cd3f15f6a74555bae071621a01e74b-5.14.0-432.el9.x86_64.conf:options root=UUID=00843ae4-939d-4d41-946e-45ef756d57d3 ro rd.luks.uuid=luks-f302d431-2e80-43be-9790-6d5a5fcd9cd9 rd.lvm.lv=storage/root rd.shell=0 rhgb quiet $tuned_params
[root@rh9 ~]# 



No crashkernel appended!

I have a Alma 9.3 that (unintentionally) has kdump enabled.
The /etc/default/grub:GRUB_CMDLINE_LINUX does not contain crashkernel.
The /etc/kernel/cmdline does not contain crashkernel.
Every bootloader-entry does have crashkernel and hence the kernel runs with it too.

I did systemctl disable kdump.service ; systemctl stop kdump.service and reboot.
The kernel still boots with crashkernel option because BLS-entry is unchanged.

Then: grub2-mkconfig --update-bls-cmdline -o /boot/grub2/grub.cfg
Now the bootloader-entries do not contain crashkernel any more.

Conclusion: the generation of bootloader-entry does not get the crashkernel option from /etc/default/grub nor from /etc/kernel/cmdline; some other source does provide that. (Similarly, the root=XXX ro is not from /etc/default/grub – but exists in /etc/kernel/cmdline.)

I have a stand alone RL9 server and also RL9 installed in a vm on a Fedora machine. Both have crashkernel on the command line but the vm does not reserve the 256 MB memory space. On the vm the kdump status is enabled and the output is that a condition is not met thus it is dead.
On the server kdump is disabled but as has been shown that has no effect on memory reservation.

Definitely a weird one.

When I have a clean Rocky 9 install I don’t have an /etc/kernel/cmdline file. I only get this when running the grubby commands to configure removal of crashkernel. Even without it, the grubby info command reports crashkernel as being there. I have disabled kdump service, and my /boot/grub2/grub.cfg or /etc/grub2.cfg do not contain crashkernel entries. dmesg does claim that it reserves 256mb, although I cannot verify if it does actually reserve it, or just outputs the message anyway.

Not been able to get crashkernel disabled using any of the methods here, usually up until this thread appeared I just disabled kdump and forgot about it. We could probably spend so much time on this probably far more than it’s worth tbh :slight_smile:

1 Like

That option (crashkernel) is generated by the install script, that are executed when a kernel is installed (all scripts in /usr/lib/kernel/install.d/ are executed). So, you need the additional key
auto_reset_crashkernel set to no in /etc/kdump.conf to dictate to not add the mention option to new kernels.

1 Like

Looks like this was fixed in the RH 9.4 beta.

# makedumpfile --mem-usage /proc/kcore
show_mem_usage: No memory is reserved for crashkernel!

# free -m
               total        used        free      shared  buff/cache   available
Mem:            3915         455        3323           8         359        3460
Swap:           3071           0        3071

1 Like

Because of the way the option is crafted, crashkernel reserves memory up to 512MB depending on the total system memory available to the kernel. That’s what the option means. Not a good assumption if you ask me. Very confusing.

crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
1 Like

I thought this is Rocky forums. Why are people talking about RHEL? I’m so confused.

1 Like

It’s a bug guys! I’m filing it. Everyone go home and have a nice break!

No idea why people are talking about other distroes here. This is def a distro-specific issue. I came to Rocky forums because it’s a Rocky problem!

I’ve had known this issue far back since CentOS 7. Devs still didn’t get rid of that stupid kernel option. So annoying. Rocky(or RHEL) nowadays run on VMs anyways. The kernel crash on CSP VMs are often due to underlying host issue, not the kernel itself. Even if it is, ain’t nobody got time for carefully pulling the dump off the vm and inspecting/investigating it. More of than not, the dump alone cannot solve all the problems. You have to use kernel module options to troubleshoot.

1 Like

It’s only a coincidence that I tested using the 9.4 beta. Disabling dump on install, the beta removed the whole crashkernel option.

Would it work on RL 9.3? Removal of crashkernel did return the 256mb. But the utilities make dump --mem-usage /proc/kcore and kdumpctl showmem do return errors. Unsure what would happen on a system crash.

1 Like

Well, I think it’s worth investigating it. Consider that other distroes don’t even ship with kdump enabled. Think how many PHP connections you can handle with 512MB. It’s at least 20. You can run a whole mariadb with that amount of memory. It’s ridiculous. All that much memory sitting on its arse doing nothing. Memory is expensive on cloud, you know. Every megabyte counts. Think how much work put into that memory thus saving energy.

Or maybe I’m just too paranoid because I’m from the embedded world.

1 Like

My premise was that you don’t need kdump for most use cases. It’s not too late to turn it on when you start experiencing some issues. Most distroes don’t ship with it enabled anyways for that very reason.

It’s only a coincidence that I tested using the 9.4 beta. Disabling dump on install, the beta removed the whole crashkernel option.

On that note, I might play around with 9.4.

Rocky is based on RHEL. Therefore, if there is a bug in RHEL, then that bug is in Rocky as well. This is why people mention RHEL.

So when RHEL 9.4 is released, Rocky 9.4 will appear sometime after it.

1 Like

Rocky is based on RHEL, so most bugs in Rocky are same as the RHEL has. Those have to be reported to Red Hat as “bug in RHEL (or CentOS Stream)” in order for RH to fix them. Rocky then rebuilds from fixed versions of code. (CentOS was the same way – bug for bug what RHEL had.)

It is possible for Rocky to have some bugs of its own. Errors in the build process, like some package initially missing. Those bugs Red Hat cannot fix.

1 Like

This seems to work in RL 9.3. Now we can wait until RL 9.4 is ready.

[    0.000000] Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-362.24.1.el9_3.x86_64 root=/dev/mapper/rl-root ro resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet audit=1 crashkernel=no
[    0.005467] crashkernel: memory value expected
[    0.049198] Kernel command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-362.24.1.el9_3.x86_64 root=/dev/mapper/rl-root ro resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet audit=1 crashkernel=no

grubby --update-kernel=ALL --args="crashkernel=no"
(1004 / 5) [~]$: free -m
               total        used        free      shared  buff/cache   available
Mem:            3913         999        2527          15         636        2914
Swap:           3071           0        3071
(1003 / 4) [~]$: makedumpfile --mem-usage /proc/kcore
show_mem_usage: No memory is reserved for crashkernel!

makedumpfile Failed.
(1001 / 2) [~]$: kdumpctl estimate
Reserved crashkernel:    0M
Recommended crashkernel: 192M

Kernel image size:   53M
Kernel modules size: 9M
Initramfs size:      32M
Runtime reservation: 64M
Large modules:
    xfs: 2473984
    ext4: 1167360
WARNING: Current crashkernel size is lower than recommended size 192M.
(1002 / 3) [~]$: kdumpctl showmem
kdump: Reserved 0MB memory for crash kernel
1 Like

No it doesn’t for new installs of kernel. Many people have already suggested using grubby. Please review the threads.

1 Like

Yes. That’s what I’m saying. I’m sure there are differences from RHEL.

1 Like