372.13 update problem (virtual machine)

There is a problem when update to 4.18.0-372.13.1.el8_6.x86_64 in several virtual machines.
After dnf update; reboot; error occurs:
graberror
For virtual machine that the grub menu can be accessed, boot into the pervious 372.9 then sudo dracut to create the img file and reboot good into the 372.13.
In the dnf.rpm.log, there are below records.
INFO /etc/dracut.conf.d/virt-drivers.conf:add_drivers+=“xen-blkfront xen-netfront virtio_blk virtio_net”
dracut: WARNING: +=" ": should have surrounding white spaces!
dracut: WARNING: This will lead to unwanted side effects! Please fix the configuration file.

For virtual machine in cloud, have to roll back and do the sudo dracut BEFORE reboot to make it good.
Lucky is all these virtual machines are non-production, it will be a real problem for production one. Is there any setting I put it wrong that will trigger such error? Where should I start to check?

For the dracut errors, you can fix that like my example below:

add_dracutmodules+=" resume "

you will see the space between the first and last quote and the word resume. You can edit that virt-drivers.conf and do the same here. Chances are you have older files before the format change so it would look like +="resume" for example when I had this problem. Once you do this, the dracut will run without errors, and you won’t have to do it yourself anymore.

When it says “virtual machine”, does it mean kernel virtual machine, as in libvirt?

Does this mean all libvirt virtual machines will break as soon as they try to update the new kernel?

Sorry that didn’t state it clearly. I’m running RockyLinux on Virtual Machine when bump into this update problem.

Maybe due to my unclear detail, as I’m running RockyLinux on VM, I can’t find the virt-drivers.conf in /etc/dracut.conf.d, then is it means that I need to create it and add your example in it?

No, I gave my example to show how you fix the problem with that warning message that you get so that you can add the spaces correctly.

My example was for getting resume to work after suspending, but I also had the warning message about problems with spaces.

But I expect it’s most likely something unique to your installation rather than it not working on KVM for everyone else.

Thanks Ian, good to learn something even not directly to problem on hand. Also I suspect that only occur to me. It is just a bit strange to me that several VM bump in same error while update, as those VM are almost just RL installed without tweak (as I my still newbie).
Just hope that it will not occur again in future production machine. Maybe needs to invest more time in setting up a reliable backup strategy :slight_smile:

Were these VM’s cleanly installed with Rocky, or upgraded/migrated from CentOS 8 perhaps?

Centos 8.5 migrated to Rocky 8.5 (when it released) then continuously update automatically. All updates in the pass go well. Without changing anything relate to update or kernel setting then this error occurred that is what bothering me.

Can you confirm if a VM with a clean install of Rocky works properly? I expect this is most likely to do with the migration. Best would be to install from say Rocky 8.5 and then later upgrade to 8.6 and see if problems occur.

Lets summarize:
Symptom: the installation of new kernel (372.13) fails to generate initramfs image. Manual generation is possible.
Apparent reason: dracut does not like the syntax of /etc/dracut.conf.d/virt-drivers.conf
History: Creation of initramfs for 8.5’s kernels and for the 372.9 did succeed.

Logically, there has been an update to dracut that made it more strict/fragile or the new kernel package calls it differently (since you can still manually create the image).
Are there still logs from the earlier kernel installs, i.e. are the warnings new?

Can you uninstall 372.13 from some instance, add the whitespace to the config file, and then reinstall the 372.13? If the reason for failure is the lack of whitespace, then you should get image. If you don’t, then the reason for failure is elsewhere.

Make a VM with 4.18.0-348.e18.0.2.x86_64 then update to 4.18.0-372.13.1.e18.0.2.x86_64 without problem. So better forget migration for future machines :slight_smile: .
I’d migrate from Centos 8.5 by migrate2rocky.sh and go through all the update without problem. Then suddenly an error occur that is the reason bother me. Maybe as such migration should be fade out through time, it will not occur for other users anymore.

The summary is clear. Unfortunately, in VM running 372.9.1, “ls -l /etc/dracut.conf.d/” return “total 0” which means virt-drivers.conf didn’t exist(I’d run “history” and make sure didn’t deleted it in the past)? If this file is a must, then suppose we don’t need to create it manually? So suppose the reason for failure is ,as Ian point out, related to migration.

If it’s not there on a clean install, just move those files to a different location or delete them entirely.

Sorry, can’t follow what you means of “move those files”, as there is nothing in the /etc/darcut.conf.d/, then there is nothing to move or delete. Or you are referring to files in other location?

You said on your Centos server it had files and on the Rocky one not?

No, as Centos Svr was years ago, didn’t know there is file or not. The /etc/dracut.conf.d/ in RL 372.9 is empty.

OK, so on the server that you have problems, remove the files from /etc/dracut.conf.d so remove virt-drivers.conf or whatever else is in there.

I also made a clean Rocky install today, and zero files in that directory.

I got curious.

This is Alma 8 on bare metal that was converted from CentOS 8:

$ ll  /etc/dracut.conf.d
total 8
-rw-r--r--. 1 root root 154 Dec  7  2020 50-network-legacy.conf
-rw-r--r--. 1 root root 229 Jun 29 12:04 dracut-nvidia.conf

$ rpm -qf  /etc/dracut.conf.d/*.conf
file /etc/dracut.conf.d/50-network-legacy.conf is not owned by any package
kmod-nvidia-515.57-1.el8_6.elrepo.x86_64

$ cat /etc/dracut.conf.d/50-network-legacy.conf
# Since rhel-8.3 dracut moved to use NetworkManager
# On existing installations we want to preserve the old scripts
add_dracutmodules+=" network-legacy "

$ sudo dnf provides /etc/dracut.conf.d/*.conf
Last metadata expiration check: 1 day, 23:56:39 ago on Wed 06 Jul 2022 04:37:33 PM EEST.
kmod-nvidia-515.57-1.el8_6.elrepo.x86_64 : NVIDIA OpenGL kernel driver module
Repo        : @System
Matched from:
Filename    : /etc/dracut.conf.d/dracut-nvidia.conf

$ uname -r
4.18.0-372.13.1.el8_6.x86_64

This is Alma 8 VM (on libvirt/KVM) also migrated from CentOS 8

$ ll  /etc/dracut.conf.d
total 4
-rw-r--r--. 1 root root 154 Mar  5  2021 50-network-legacy.conf

$ rpm -qf  /etc/dracut.conf.d/*.conf
file /etc/dracut.conf.d/50-network-legacy.conf is not owned by any package

$ cat /etc/dracut.conf.d/50-network-legacy.conf
# Since rhel-8.3 dracut moved to use NetworkManager
# On existing installations we want to preserve the old scripts
add_dracutmodules+=" network-legacy "

An another bare-metal Alma 8, but installed as Alma 8.4:

$ ll  /etc/dracut.conf.d
total 0

What do we learn from these?

  • One of the CentOS Linux 8 point updates (probably 8.3) did create file /etc/dracut.conf.d/50-network-legacy.conf that does not belong to any RPM-package That file has whitespaces
  • There are no /etc/dracut.conf.d/*.conf-files in base repo packages
  • Additionally (not shown), none of my Alma 8 has word “WARN” in /var/log/dnf.rpm.log
  • The virt-drivers.conf is probably generated by Xen-guest tools (whatever they are)
  • libvirt/KVM guests do not explicitly inject “virtio_blk virtio_net” via these config files
1 Like

I’ve just upgraded Rocky 8.6 host and guest (original clean install)

4.18.0-372.9.1.el8.x86_64
4.18.0-372.13.1.el8_6

and there were no issues with dracut, but I did notice dracut was changed as part of the upgrade

dracut-049-201.git20220131.el8.x86_64
dracut-049-202.git20220511.el8_6.x86_64

In my case

ls -a /etc/dracut.conf.d

is empty as expected.