Migration from MD to LVM

RenegadeSithLord · August 2, 2024, 11:40am

I have a system running Rocky 8, that was configured with an MD array for 2 disks. I am migrating the second disk to use LVM (then use a Raid overlay). I’m at the point where I think I have the new LVM configured, but it just doesn’t boot. FIPS is installed on the original (requirement). Some detail, you can see that I’ve mounted the new LVM build:

$ df -h
Filesystem                           Size  Used Avail Use% Mounted on
devtmpfs                             126G     0  126G   0% /dev
tmpfs                                126G     0  126G   0% /dev/shm
tmpfs                                126G   27M  126G   1% /run
tmpfs                                126G     0  126G   0% /sys/fs/cgroup
/dev/md121                            30G   23G  7.3G  76% /
/dev/md118                           150G  110G   41G  74% /cluster
/dev/md119                            30G  247M   30G   1% /tmp
/dev/md128                           187G  169G   19G  91% /repos
/dev/md123                          1018M   40M  978M   4% /home
/dev/md124                            20G  883M   20G   5% /var
/dev/md127                          1018M  338M  681M  34% /boot
/dev/md126                           8.0G  220M  7.8G   3% /var/log
/dev/md122                           8.0G  129M  7.9G   2% /var/log/audit
/dev/md125                           600M  5.8M  595M   1% /boot/efi
/dev/mapper/eddie_lvm-root            30G   23G  7.0G  77% /mount
/dev/mapper/eddie_lvm-boot          1014M  391M  624M  39% /mount/boot
/dev/mapper/eddie_lvm-boot_efi       600M  5.8M  594M   1% /mount/boot/efi
/dev/mapper/eddie_lvm-cluster        130G  114G   16G  88% /mount/cluster
/dev/mapper/eddie_lvm-home          1014M   40M  975M   4% /mount/home
/dev/mapper/eddie_lvm-repos          190G  169G   22G  89% /mount/repos
/dev/mapper/eddie_lvm-tmp             20G  176M   20G   1% /mount/tmp
/dev/mapper/eddie_lvm-var             20G  864M   20G   5% /mount/var
/dev/mapper/eddie_lvm-var_log        8.0G  216M  7.8G   3% /mount/var/log
/dev/mapper/eddie_lvm-var_log_audit  8.0G  125M  7.9G   2% /mount/var/log/audit
tmpfs                                 26G  4.0K   26G   1% /run/user/14559

I can chroot to the new LVM disk:

# chroot /mount
[root@eddie /]# df -h
Filesystem                           Size  Used Avail Use% Mounted on
/dev/mapper/eddie_lvm-root            30G   23G  7.0G  77% /
/dev/mapper/eddie_lvm-boot          1014M  391M  624M  39% /boot
/dev/mapper/eddie_lvm-boot_efi       600M  5.8M  594M   1% /boot/efi
/dev/mapper/eddie_lvm-cluster        130G  114G   16G  88% /cluster
/dev/mapper/eddie_lvm-home          1014M   40M  975M   4% /home
/dev/mapper/eddie_lvm-repos          190G  169G   22G  89% /repos
/dev/mapper/eddie_lvm-tmp             20G  176M   20G   1% /tmp
/dev/mapper/eddie_lvm-var             20G  864M   20G   5% /var
/dev/mapper/eddie_lvm-var_log        8.0G  216M  7.8G   3% /var/log
/dev/mapper/eddie_lvm-var_log_audit  8.0G  125M  7.9G   2% /var/log/audit
devtmpfs                             126G     0  126G   0% /dev

I’ve rebuilt dracut and the boot configuration:

# more grubenv

# GRUB Environment Block

saved_entry=0cdd3299553f4020a37814c21fa2906e-4.18.0-425.3.1.el8.x86_64

kernelopts=root=/dev/mapper/eddie_lvm-root ro crashkernel=auto resume=/dev/mapper/eddie_lvm-swap rhgb quiet page_poison=1 rd

.lvm.lv=eddie_lvm/root rd.lvm.lv=eddie_lvm/boot rd.lvm.lv=eddie_lvm/boot_efi rd.lvm.lv=eddie_lvm/swap vsyscall=none slub_deb

ug=P audit=1 audit_backlog_limit=8192 pti=on fips=0 boot=UUID=69e377b9-8580-40d5-8f36-fe253e20da41

############################################################################################################################

############################################################################################################################

############################################################################################################################

############################################################################################################################

####################################################################################

Even disabling FIPS so that it should boot. The boot process sees the OS and starts the load, but then hands up on “dracut pre-pivot and cleanup hook…”. The message is “Warning: /boot//.vmlinux-4.18.0-425.3.1.el8.x86_64.hmac does not exist. Refusing to continue”. The /boot directory has:

# ls -a
.                                                         initramfs-4.18.0-425.3.1.el8.x86_64kdump.img
..                                                        loader
config-4.18.0-372.9.1.el8.x86_64                          symvers-4.18.0-372.9.1.el8.x86_64.gz
config-4.18.0-425.3.1.el8.x86_64                          symvers-4.18.0-425.3.1.el8.x86_64.gz
efi                                                       System.map-4.18.0-372.9.1.el8.x86_64
grub2                                                     System.map-4.18.0-425.3.1.el8.x86_64
initramfs-0-rescue-0cdd3299553f4020a37814c21fa2906e.img   vmlinuz-0-rescue-0cdd3299553f4020a37814c21fa2906e
initramfs-4.18.0-372.9.1.el8.x86_64.img                   vmlinuz-4.18.0-372.9.1.el8.x86_64
initramfs-4.18.0-372.9.1.el8.x86_64kdump.img              .vmlinuz-4.18.0-372.9.1.el8.x86_64.hmac
initramfs-4.18.0-425.3.1.el8.x86_64.bak.07-30-114347.img  vmlinuz-4.18.0-425.3.1.el8.x86_64
initramfs-4.18.0-425.3.1.el8.x86_64.img                   .vmlinuz-4.18.0-425.3.1.el8.x86_64.hmac

I’v even recreated the .hmac file by running sha512hmac on /boot/vmlinuz-4.18.0-425.3.1.el8.x86_64 with what looked like appropriate output.

My suspicion is that for some reason the grub bootloader isn’t properly pulling in the new /boot and /boot/efi file systems to get the correct configuration. Looking for any ideas that might get me around this.

jlehtone · August 2, 2024, 12:19pm

The UEFI loads bootloader (*.efi, grub.cfg, and grubenv) from /boot/efi
There is code in grub.cfg for GRUB to find /boot and /boot/efi.
Then GRUB loads menu entries from /boot (/boot/loader/entries/*.conf).
Once an entry is selected, kernel and initramfs are loaded from /boot
The kernel get options (including root= and resume= ) from grubenv.

It is the kernel and initramfs that have/start dracut. It will be “wrong dracut”, is something above points to (or picks) wrong volume(s) – by UUID or something.

Note that

blkid
lsblk -f

do list UUID’s.

RenegadeSithLord · August 2, 2024, 12:55pm

I’ll give that a look. I know I’m close, it’s just finding that last needle to get this going. Just need to be sure that the new LVM format can boot without the original disk being present.

jlehtone · August 2, 2024, 1:13pm

Wait! You have ESP (/boot/efi) and /boot on LVs?

That requires that both UEFI and GRUB must read LVM. Can they? The default install still makes those two as “standard” partitions (and rest as LVs).

RenegadeSithLord · August 2, 2024, 1:27pm

No, /boot and /boot/efi are not on LVMs. Those were distinct partitions made so they can be straight MD mirrored after I finish the LVM migration (because it will then become a mirrored LVM pair).

jlehtone · August 2, 2024, 1:33pm

Good. The names on these are misleading:

RenegadeSithLord · August 2, 2024, 1:53pm

Excuse me while I facepalm and realize that the unwanted LVM was mounted while the new partition is using /dev/sdbXX. That would be why it’s not finding the correct data, it’s not on the disk.

Thanks for being that extra pair of eyes I needed to catch my error.

RenegadeSithLord · August 13, 2024, 10:18am

Yeah, using the wrong boot partition was the issue. It boots the OS now. Just need to tweak a few things.

system · October 12, 2024, 10:18am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.