Rocky Linux 9.3 boot failure: cannot mount /var

I installed Rocky Linux 9.3 to a virtual machine hosted on VMware 7. The partition layout uses two VMDK disks, / for /dev/sda and /var for /dev/sdb. Partition is defined in my kickstart file like below:

clearpart --all --initlabel

part /boot --fstype="xfs" --size=1024 --ondisk=sda
part /boot/efi --fstype="efi" --ondisk=sda --size=600 --fsoptions="umask=0077,shortname=winnt"


part pv.01 --asprimary --fstype="lvmpv" --size=1024 --grow --ondisk=sda
volgroup vg_root pv.01

logvol / --fstype="xfs" --name=root --vgname=vg_root --size=1 --grow
logvol swap --fstype="swap" --name=swap --vgname=vg_root --size=4096

part pv.02 --asprimary --fstype="lvmpv" --size=1024 --grow --ondisk=sdb
volgroup vg_var pv.02

logvol /var --fstype="xfs" --name=var --vgname=vg_var --size=117760 --grow

The kickstart process completed successfully and I installed various components to the system. I then need to change the IP address and boot off the system again. But upon booting, the boot process failed with message like below:

A job running for /dev/mapper/vg_var-var timeout

Subsequently, various system services failed to start up, like auditd.

So far I have tried the following to solve the issue:

  • Boot off of the system in rescue mode
  • Rebuild initramfs file in /boot

None of above solve the issue. In my troubleshoorting effort, neither lvm vgdisplay nor lvm pvscan showed anything. No volume group vg_root nor vg_var showed up.

I boot off the system of the Rocky 9 ISO cd-rom. / and /var can be mounted under /mnt/sysroot and /mnt/sysroot/var without any issue. That’s why I’m able to rebuild the initramfs file. But /var still failed to mount upon rebooting.

Is there anything wrong with my setup?

Chapter 10. Customizing the system in the installer | Red Hat Product Documentation writes:

  • If /usr or /var is partitioned separately from the rest of the root volume, the boot process becomes much more complex because these directories contain boot-critical components. In some situations, such as when these directories are placed on an iSCSI drive or an FCoE location, the system may either be unable to boot, or it may hang with a Device is busy error when powering off or rebooting.
    This limitation only applies to /usr or /var, not to directories under them. For example, a separate partition for /var/www works without issues.

Important
Some security policies require the separation of /usr and /var, even though it makes administration more complex.

Could that explain your case?

Yes I think so. I’d like to backup then restore the VM to a different VM with /var under / volume group. Is there anything different for Rocky Linux 9.3 in terms of backup and restore procedure for grub boot loader configuration?

I don’t backup like that, so cannot advise.

I install packages with Ansible, deploy config for them with Ansible, and have (backup) copies of only the Ansible plays and the user data files.