Tried updating CentOS 8 to Rocky Linux using migrate2rocky.sh. Everything seemed fine, hit reboot. Now getting an error:
Failed to start Switch Root. Specified switch root path “/sysroot” does not seem to be an OS tree. os-release file is missing.
Server specs are: Intel Core i7-4770, 2x HDD SATA 2,0 TB Enterprise with Software Raid (mdraid). Nothing special on the CentOS side: Apache, PHP, MariaDB etc.
Any ideas what to do?
If you do go to “edit”-mode in grub menu, you should be able to see the kernel command-line parameters.
Is there “/sysroot”?
I think this a screenshot from the Rocky option, not entirely sure though and can’t check (since lost access again), but I think I didn’t see a /sysroot there.
Looks like there are no options at all.
I have, for example:
root=UUID=de..eb ro resume=UUID=47..98 nouveau.modeset=0 rd.driver.blacklist=nouveau plymouth.ignore-udev
Of which the nouveau.modeset=0 rd.driver.blacklist=nouveau plymouth.ignore-udev was added for NVidia driver and I have removed options rhgb quiet which installer adds by default.
Grub of EL8 does use BLS by default. BLS creates entries in /boot/loader/entries and includes statement:
options $kernelopts $tuned_params
The grub.cfg sources grubenv that has kernelopts (or supplies a default).
Do you have grub entry “rescue”? If yes, try booting with it.
Whether with that or latest kernel you can append to the linux /vmlinuz-4.18.0-348.2.1.el8_5.x86_64 line:
systemd.unit=rescue.target
Option B is to have minimal Rocky 8.5 install image on USB, boot from it and select “rescue”. That should detect the rocky install and give chance to edit files.
I got access to the file system via custom Debian rescue mode, mounted the RAID disks, and checked the /boot/loader/entries:
title Rocky Linux (4.18.0-348.2.1.el8_5.x86_64) 8.5 (Green Obsidian)
version 4.18.0-348.2.1.el8_5.x86_64
linux /vmlinuz-4.18.0-348.2.1.el8_5.x86_64
initrd /initramfs-4.18.0-348.2.1.el8_5.x86_64.img $tuned_initrd
options $kernelopts $tuned_params
id rocky-20211115210229-4.18.0-348.2.1.el8_5.x86_64
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel
Just guessing without real knowledge, but could it be that Rocky can’t load the RAID disks/mounts correctly? Corrupted /etc/fstab somehow? Tried that systemd.unit=rescue.target earlier with no noticeable difference.
/etc/fstab on the mounted disk looks like this:
proc /proc proc defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
tmpfs /dev/shm tmpfs defaults 0 0
sysfs /sys sysfs defaults 0 0
/dev/md/0 none swap sw 0 0
/dev/md/1 /boot ext3 defaults 0 0
/dev/md/2 / ext4 defaults 0 0
/dev/md/3 /biz/ ext4 defaults 0 0
When I started the Debian rescue, ´lsblk´ looked like this:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 2.9G 1 loop
sda 8:0 0 1.8T 0 disk
├─sda1 8:1 0 16G 0 part
├─sda2 8:2 0 512M 0 part
├─sda3 8:3 0 40G 0 part
├─sda4 8:4 0 1K 0 part
└─sda5 8:5 0 1.8T 0 part
sdb 8:16 0 1.8T 0 disk
├─sdb1 8:17 0 16G 0 part
├─sdb2 8:18 0 512M 0 part
├─sdb3 8:19 0 40G 0 part
├─sdb4 8:20 0 1K 0 part
└─sdb5 8:21 0 1.8T 0 part
Is this helps in any way.
[EDIT] I was mistaken
That is not how /etc/fstab should look like. More likely that it would start with:
UUID=***** / xfs default 1 1
[/EDIT]
The ‘kernelopts’ are written in ‘grubenv’ and/or ‘grub.cfg’ which are in /boot/efi/EFI/*/ or in /boot/grub2/ (if you still use legacy mode).
Use blkid or additional options for lsblk to see labels or UUID’s of the filesystems.
Although, you should get RAID up before you mount and write to filesystems.
Initialization of software RAID and root filesystem do happen early in boot by files in initramfs.
The initramfs is generated when new kernel is installed. That too might have gone wrong and should be redone.
Do you still have CentOS kernels? Can you boot to them?
hi,
linux ($root)/vmlinuz-4.18.0-348.2.1.el8_5.x86_64
looks suspicious to me. At least I would expect a root= argument
you could try a line like
linux ($root)/vmlinuz-4.18.0-348.2.1.el8_5.x86_64 root=/dev/md/2 ro
After booting the kernel the initramfs is loaded and after that the root file system is mounted at /sysroot. But how to know what to mount if root= argument is not given?
Also, the error happens too early to look for error in /etc/fstab, IMO.
From your /boot/loader/entries the line
options $kernelopts $tuned_params
is there, so it seems that $kernelopts is empty (?), so maybe regenerating grub.cfg after getting it to boot once helps.
So, I think jlehtone was correct with noting that the kernel arguments are missing and would suggest trying above arguments with root=/dev/md/2 (I got that value from your fstab)
Yes, but seems like it ends up in exact same emergency mode as booting with Rocky kernel.
Tried this. First it seemed like it’s booting up, then stopped for a while. I was about to reboot when started repeating:
“Warning: dracut-initqueue timeout - starting timeout scripts”
I think I have to soon give up on this, since I need to get the server back up.
Sorry it didn’t work out for you. If you can work out what went wrong, I would really love to hear back from you in case there’s some way we can improve migrate2rocky to deal with the issue you’re having.
Had the same error tonight (missing os-release) on an AMD Epyc Server system that couldn’t boot anymore after migration.
Migration path was from CentOS 8.5 to Rocky 8.5.
While AlmaLinux migrations worked on other servers flawlessly - no offense at this point - the migration to Rocky from CentOS did not. However, it might be related to the AMD platform, or possibly not because the Alma migrations happened on Intel. These two (nvme raid 1 as boot devices and AMD platform) were the only major differences between the systems that failed and the other working systems.
I then went through the Alma migration script and one, or all steps that I’ve done seemed to have helped at the end like a reinstall of shim, fwupd, kernel and grub2 and last but not least I regenerated the grub conf, everything from a rescue boot.
Maybe this information is helpful for others.
Can you share your migrate2rocky.log file from the failed migration, please?
The issue that I see here is that you have the remi repo installed which stomps on core OS packages. In this particular case it’s stomping on packages from the powertools repo which is causing migrate2rocky to mistake the remi repo for the powertools repo in CentOS 8.
That said, the os-release file comes from the rocky-release package in the baseos repo and it’s clearly installed here, so I can’t see how os-release ends up missing. Even so, I would say that the remi repo is the most likely cause of your issue.
You can revert the install, disable the remi repo and try again, or you can do the following:
dnf reinstall rocky-release which should replace the os-release file.
On another note, and due to the fact that remi stomps on core packages, your ImageMagick package has been downgraded to the version in epel, which is 6.9.10, as opposed to the version in remi which is 6.9.11. If this is not an issue then I would recommend sticking to the version in epel and getting rid of the remi repo as it is a known cause of problems.
Thank you for response but as I wrote in my initial post is that I fixed the problem by myself already and what I mentioned were the steps that helped in my case.
Also the rocky release rpm package was installed succesfully. I checked it first when I booted from a live system to fix the boot problem. Since the initial error was that /sysroot couldn’t (including the os-release file) be found the error must have something in common with grub, dracut or something along those packages. That’s why reinstalling and/or regenerating the grub conf seem to have done the trick although these packages were covered by the migration script which is surprising…
Regarding Remi repo:
Getting rid of it is impossible since it is of importance.
Remi’s repos only include php and imagemagick. There is nothing else there that overwrites core packages so I doubt very much that repo is the problem at all. Any chances of conflicts are only going to be with php or imagemagick packages system vs remi. These packages do not cause booting issues with the system however. The problem is rather something else that happened or didn’t happen during the migration. Although nothing seemed to show in the log, since grub packages were replaced by Rocky ones, fwupd also, and others mentioned. Reinstalling and remaking the grub config seemed so fix this, Normally installation of new kernels would have done this also, and that was also done during the migration process. Definitely a strange one, but php and imagemagick have nothing to do with that.
Remi’s repositories are safe to use and have been for years.
Spoken to Remi about the issue and he’s removed the problematic package. I would be curious to know if the problem persists now.
Somebody else will have to test this since I converted all my CentOS 8 machines to AlmaLinux (same repo config like the Rocky machine), except for the one Rocky machine, which is now running fine after I manually fixed the boot problem.
But since I was not the first one who had this os-release / sysroot error maybe someone else will run into this and can then check your imagemagick from Remi’s repo theory.
That’s fine, thanks for your feedback about the issue to begin with, though. As I said, we’re always looking to find and fix these edge cases with migrate2rocky.
The issue isn’t with imagemagick, that just happens to be the package that you installed from remi. The issue was with another package that was in remi that clashed with a package in the stock powertools repo, you did not need to have the package installed for it to cause the issue, but rather it was enough to have remi installed and enabled. Remi removed that package from his repo and so it should no longer be an issue now.
Thanks for this information.
I just noticed after your recent post and rereading the log file that the Powertools mapping was wrong.
So, the mapping was wrong because it was caused by a Remi repo package or was that another, additional issue? If it was caused by a Remi package, which was the name of it?
The way that migrate2rocky determines the mapping to the source distro’s repositories is it looks for a particular package name that is unique to that repo. This is because different EL distributions give their repos different names, but a particular repo will almost always have the same package set in it. So in this particular case migrate2rocky looks for the libaec-devel package to figure out which repo to map to powertools, but remi also had libaec-devel in his repo, so migrate2rocky was tricked into thinking that remi was powertools. I spoke to Remi about the issue and he removed the libaec packages from his repo as it’s his policy to not duplicate packages from stock repos anyways and libaec was there in error.
When I looked at your migrate2rocky output I noticed the mapping issue so I immediately keyed onto it. I don’t know for certain if it’s the actual cause of the missing os-release file, but it pays to fix the things that are obvious first, so if you tried again now and reported that the problem still persists then I would work with you to troubleshoot further.
