Diagnosing boot failure

Rocky Linux Help & Support
1

Hi I’m attempting to diagnose a boot problem in Rocky 9.5
This started after I changed the name of the volume group containing the lv my root partition is on.

When booting I get the errors:



test-node2 dracut-initqueue[742]: Warning: dracut-initqueue: starting timeout scripts
test-node2 dracut-initqueue[742]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
test-node2 dracut-initqueue[742]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fmapper\x2frl_test--node2--mgmt-root.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
test-node2 dracut-initqueue[742]:     [ -e "/dev/mapper/rl_test--node2--mgmt-root" ]
test-node2 dracut-initqueue[742]: fi"
test-node2 dracut-initqueue[742]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2frl_test-node2-mgmt\x2froot.sh: "[ -e "/dev/rl_test-node2-mgmt/root" ]"
test-node2 dracut-initqueue[742]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2frl_test-node2-mgmt\x2fswap.sh: "[ -e "/dev/rl_test-node2-mgmt/swap" ]"
test-node2 dracut-initqueue[742]: Warning: dracut-initqueue: starting timeout scripts
test-node2 dracut-initqueue[742]: Warning: Could not boot.
test-node2 systemd[1]: Starting Dracut Emergency Shell...



Warning:
/dev/mapper/rl_test--node2--mgmt-root does not exist
Warning: /dev/rl_test-node2-mgmt/root does
not exist
Warning: /dev/rl_test-node2-mgmt/swap does not exist
Generating "/run/initramfs/rdsosreport.txt"

I am then dropped to a dracut emergency shell.

I’ve made changes to grub, rebuilt initramfs, and few other things I’ve lost track of without success. How should I troubleshoot this? I would really like to understand the issue.

I booted to recovery on a dvd iso chrooted to /mnt/sysroot and pulled this info:

Output of grubby --info=ALL

kernel="/boot/vmlinuz-5.14.0-503.21.1.el9_5.x86_64"
args="rhgb quiet $tuned_paramsgrub_users $grub_users ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.lvm.lv=rl_test-node2-mgmt/root rd.lvm.lv=rl_test-node2-mgmt/swap resume=/dev/mapper/rl_test--node2--mgmt-swap"
root="/dev/mapper/rl_test--node2--mgmt-root"
initrd="/boot/initramfs-5.14.0-503.21.1.el9_5.x86_64.img $tuned_initrd"
title="Rocky Linux (5.14.0-503.21.1.el9_5.x86_64) 9.5 (Blue Onyx)"
id="8b6c9908974e48a2adfe2b455a4324d8-5.14.0-503.21.1.el9_5.x86_64"
index=1
kernel="/boot/vmlinuz-5.14.0-503.19.1.el9_5.x86_64"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.lvm.lv=rl_test-node2-mgmt/root rd.lvm.lv=rl_test-node2-mgmt/swap resume=/dev/mapper/rl_test--node2--mgmt-swap"
root="/dev/mapper/rl_test--node2--mgmt-root"
initrd="/boot/initramfs-5.14.0-503.19.1.el9_5.x86_64.img $tuned_initrd"
title="Rocky Linux (5.14.0-503.19.1.el9_5.x86_64) 9.5 (Blue Onyx)"
id="8b6c9908974e48a2adfe2b455a4324d8-5.14.0-503.19.1.el9_5.x86_64"
index=2
kernel="/boot/vmlinuz-5.14.0-503.16.1.el9_5.x86_64"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.lvm.lv=rl_test-node2-mgmt/root rd.lvm.lv=rl_test-node2-mgmt/swap resume=/dev/mapper/rl_test--node2--mgmt-swap"
root="/dev/mapper/rl_test--node2--mgmt-root"
initrd="/boot/initramfs-5.14.0-503.16.1.el9_5.x86_64.img $tuned_initrd"
title="Rocky Linux (5.14.0-503.16.1.el9_5.x86_64) 9.5 (Blue Onyx)"
id="8b6c9908974e48a2adfe2b455a4324d8-5.14.0-503.16.1.el9_5.x86_64"
index=3
kernel="/boot/vmlinuz-0-rescue-8b6c9908974e48a2adfe2b455a4324d8"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.lvm.lv=rl_test-node2-mgmt/root rd.lvm.lv=rl_test-node2-mgmt/swap resume=/dev/mapper/rl_test--node2--mgmt-swap"
root="/dev/mapper/rl_test--node2--mgmt-root"
initrd="/boot/initramfs-0-rescue-8b6c9908974e48a2adfe2b455a4324d8.img"
title="Rocky Linux (0-rescue-8b6c9908974e48a2adfe2b455a4324d8) 9.5 (Blue Onyx)"
id="8b6c9908974e48a2adfe2b455a4324d8-0-rescue"

fstab:

/dev/mapper/rl_test--node2--mgmt-root    /    xfs    defaults    0  0
UUID=75158f7-30e2-442f -9707-f5ec396f6048     /boot    xfs    defaults   0  0
UUID=8DDA-0693    /boot/efi    vfat    umask=0077,shortname=winnt    0  2
/dev/mapper/rl_test--node2--mgmt-swap    none    swap    defaults    0  0

lvs:

  LV   VG                 Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root rl_test-node2-mgmt -wi-ao---- <213.79g
  swap rl_test-node2-mgmt -wi-ao----  <16.00g
  lv1  test-vg1           -wim------ <148.80t
  lv2  test-vg2           -wim------ <148.80t

pvs:

  PV                 VG                 Fmt  Attr PSize    PFree
  /dev/mapper/mpatha test-vg1           lvm2 a--   <37.20t    0
  /dev/mapper/mpathb test-vg1           lvm2 a--   <37.20t    0
  /dev/mapper/mpathc test-vg1           lvm2 a--   <37.20t    0
  /dev/mapper/mpathd test-vg1           lvm2 a--   <37.20t    0
  /dev/mapper/mpathe test-vg2           lvm2 a--   <37.20t    0
  /dev/mapper/mpathf test-vg2           lvm2 a--   <37.20t    0
  /dev/mapper/mpathg test-vg2           lvm2 a--   <37.20t    0
  /dev/mapper/mpathh test-vg2           lvm2 a--   <37.20t    0
  /dev/sda3          rl_test-node2-mgmt lvm2 a--  <229.79g    0

vgs:

  VG                 #PV #LV #SN Attr   VSize    VFree
  rl_test-node2-mgmt   1   2   0 wz--n- <229.79g    0
  test-vg1             4   1   0 wz--n- <148.80t    0
  test-vg2             4   1   0 wz--n- <148.80t    0
2

There seems to be name inconsistencies, sometimes you have double dashes before and after node2 and other times single dashes.

3

I noticed the double dashes as well but I think that is expected:

device-mapper uses a hyphen (dash) as a separator between vg name “Volume Group” and lv name “Logical Volume” parts of the device name when they are combined within devfs entries. Hence if your VG and/or LV names includes dashes (hyphens) themselves, then device-mapper will add an extra hyphen to distinguish it from the vg-lv separator dash (hyphen) .

Sources: Why do I see double hyphens (dashes) in my Logical volume name ? - Red Hat Customer Portal, thinpool - Doubled hyphens in LVM /dev/mapper names - Unix & Linux Stack Exchange

I have a working node with the same double dashes in fstab and grubby:

/dev/mapper/rl_test--node1--mgmt-root /                       xfs     defaults        0 0
UUID=cae1fa8c-3e81-406b-9ba4-694ce9d89f86 /boot                   xfs     defaults        0 0
UUID=09E9-9938          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
/dev/mapper/rl_test--node1--mgmt-swap none                    swap    defaults        0 0

index=0
kernel="/boot/vmlinuz-5.14.0-503.21.1.el9_5.x86_64"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rl_test--node1--mgmt-swap rd.lvm.lv=rl_test-node1-mgmt/root rd.lvm.lv=rl_test-node1-mgmt/swap rhgb quiet $tuned_params"
root="/dev/mapper/rl_test--node1--mgmt-root"
initrd="/boot/initramfs-5.14.0-503.21.1.el9_5.x86_64.img $tuned_initrd"
title="Rocky Linux (5.14.0-503.21.1.el9_5.x86_64) 9.5 (Blue Onyx)"
id="7dc1ea92ad13450facaa42eab6f6965c-5.14.0-503.21.1.el9_5.x86_64"
index=1
kernel="/boot/vmlinuz-5.14.0-503.19.1.el9_5.x86_64"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rl_test--node1--mgmt-swap rd.lvm.lv=rl_test-node1-mgmt/root rd.lvm.lv=rl_test-node1-mgmt/swap rhgb quiet $tuned_params"
root="/dev/mapper/rl_test--node1--mgmt-root"
initrd="/boot/initramfs-5.14.0-503.19.1.el9_5.x86_64.img $tuned_initrd"
title="Rocky Linux (5.14.0-503.19.1.el9_5.x86_64) 9.5 (Blue Onyx)"
id="7dc1ea92ad13450facaa42eab6f6965c-5.14.0-503.19.1.el9_5.x86_64"
index=2
kernel="/boot/vmlinuz-5.14.0-503.16.1.el9_5.x86_64"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rl_test--node1--mgmt-swap rd.lvm.lv=rl_test-node1-mgmt/root rd.lvm.lv=rl_test-node1-mgmt/swap rhgb quiet $tuned_params"
root="/dev/mapper/rl_test--node1--mgmt-root"
initrd="/boot/initramfs-5.14.0-503.16.1.el9_5.x86_64.img $tuned_initrd"
title="Rocky Linux (5.14.0-503.16.1.el9_5.x86_64) 9.5 (Blue Onyx)"
id="7dc1ea92ad13450facaa42eab6f6965c-5.14.0-503.16.1.el9_5.x86_64"
index=3
kernel="/boot/vmlinuz-0-rescue-7dc1ea92ad13450facaa42eab6f6965c"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rl_test--node1--mgmt-swap rd.lvm.lv=rl_test-node1-mgmt/root rd.lvm.lv=rl_test-node1-mgmt/swap rhgb quiet"
root="/dev/mapper/rl_test--node1--mgmt-root"
initrd="/boot/initramfs-0-rescue-7dc1ea92ad13450facaa42eab6f6965c.img"
title="Rocky Linux (0-rescue-7dc1ea92ad13450facaa42eab6f6965c) 9.5 (Blue Onyx)"
id="7dc1ea92ad13450facaa42eab6f6965c-0-rescue"

but missing from the output of pvs, lvs, vgs:

╭─root@test-node1 ~ 
╰─# pvs                                                                                                                                                         127 ↵
  PV                 VG                 Fmt  Attr PSize    PFree
  /dev/mapper/mpatha test-vg1           lvm2 a--   <37.20t    0 
  /dev/mapper/mpathb test-vg1           lvm2 a--   <37.20t    0 
  /dev/mapper/mpathc test-vg1           lvm2 a--   <37.20t    0 
  /dev/mapper/mpathd test-vg1           lvm2 a--   <37.20t    0 
  /dev/mapper/mpathe test-vg2           lvm2 a--   <37.20t    0 
  /dev/mapper/mpathf test-vg2           lvm2 a--   <37.20t    0 
  /dev/mapper/mpathg test-vg2           lvm2 a--   <37.20t    0 
  /dev/mapper/mpathh test-vg2           lvm2 a--   <37.20t    0 
  /dev/sdi3          rl_test-node1-mgmt lvm2 a--  <229.79g    0 
╭─root@test-node1 ~ 
╰─# lvs
  LV   VG                 Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root rl_test-node1-mgmt -wi-ao---- <213.79g                                                    
  swap rl_test-node1-mgmt -wi-ao----   16.00g                                                    
  lv1  test-vg1           -wima----- <148.80t                                                    
  lv2  test-vg2           -wim------ <148.80t                                                    
╭─root@test-node1 ~ 
╰─# vgs
  VG                 #PV #LV #SN Attr   VSize    VFree
  rl_test-node1-mgmt   1   2   0 wz--n- <229.79g    0 
  test-vg1             4   1   0 wz--n- <148.80t    0 
  test-vg2             4   1   0 wz--n- <148.80t    0
4

I’m noticing that while my root and swap logical volumes are active when booted to recovery, they are not in the dracut shell:

dracut:/# lvm
lvm> lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root rl_test-node2-mgmt -wi------- <213.79g
swap rl_test-node2-mgmt -wi------- <16.00g

Attempts to activate them are unsuccessful:

lvm> lvm vgchange -ay
0 logical volume(s) in volume group “rl_test-node2-mgmt” now active

lvm> lvchange -ay rl_test-node2-mgmt/root
lvm> lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root rl_test-node2-mgmt -wi------- <213.79g
swap rl_test-node2-mgmt -wi------- <16.68g
5

I pulled on this thread a little more and I was able to solve my issue,
I found this when I had vgchange give me verbose output:

lvm> vgchange -ay -v rl_test-node2-mgmt
  Activating logical volume rl_test-node2-mgmt/root.
  activation/volume_list configuration setting defined: Checking the list to match rl_test-node2-mgmt/root.
  No item supplied in activation/volume_list configuration setting matches rl_test-node2-mgmt/root. 
  Not activating rl_test-node2-mgmt/root since it does not pass activation filter.
  Activating logical volume rl_test-node2-mgmt/swap.
  activation/volume_list configuration setting defined: Checking the list to match rl_test-node2-mgmt/swap.
  No item supplied in activation/volume_list configuration setting matches rl_test-node2-mgmt/swap.
  Not activating rl_test-node2-mgmt/swap since it does not pass activation filter.
  Activated 2 logical volumes in volume group rl_test-node2-mgmt.
  0 logical volume(s) in volume group "rl_test-node2-mgmt" now active

I looked at /etc/lvm/lvm.conf and found I had a typo in my volume list under activations. I temporarily removed everything under activations so there would be no restrictions and regenerated my initramfs.

I am now able to boot!

(I did have to add enforcing=0 to my grub kernel parameters for the first boot due to my use of the rescue environment)

1 Like
6

Haven’t used dashes in VG/LV names myself, learned something new.

7

It varies, I’ve seen a lot of double dashes in LVM names under /dev/mapper.