System Recovery as totally inaccessible

Rocky Linux Help & Support
1

Hi,

I have a Rocky Linux installation that I converted from CentOS before it became a rolling distro.

I am running the Zimbra mail system on the machine and recently I had an issue where there was a kernel panic and MySQL tables became corrupted.

Checking the system which has been running fine for the last few years it seems that it is still booting from the old CentOS kernel.

Personally I am Arch Linux user so I am not that familiar with RPM based distros…

On one side I need to recover the MySQL tables in Zimbra and as attempting to do this I noticed the mysql client process was getting “killed” after launch?

There was also an error stating that a library file could not be loaded by LD because it was too short. Later on google’ing the error and it turns out the file is linked to Android? which confused me totally.

I decided to remove the file and perform a package reinstall. I think I used this command:

yum reinstall $(yum list installed | awk '{print $1}')

Unfortunately now I cannot boot the system at all?

The original CentOS kernel panics claiming that libraries are missing?

If I try to boot the latest Rocky Kernel I get the error message:

error while loading shared libraries libtinfo.so.6 cannot open shared object file: No such file or directory

Is there anyway to recover from this or is the system completely messed up?

In Arch I would normally just perform a “pacstrap” to reinstall things like bash and the base system, then arch-chroot into the root file system and run a reinstallation of everything which normally works fine

Is there any similar way to do this here with Rocky Linux or perhaps there is a recovery utility on an install ISO or something?

2

Yes. When one does boot one of the install images, its bootloader should have an “troubleshoot” entry (in addition to the Install and checksum and install entries). The troubleshoot entry should show a submenu and Rescue within it.

When booted in the rescue mode the image does not start installer, but tries to identify and mount existing filesystems. One does then get a shell and within shell one can chroot to the mounted tree (or access files with the utils that the install image has).

In case you fail to duct-tape the carcass via rescue …

If you had default CentOS install, then it has – just like default Rocky install – filesystems /, /boot, and /home.

  • The /home has user data. It could be left untouched (but backups are recommended)
  • The / has system config (mostly in /etc) and service data (e.g. SQL, containers) somewhere under /var. A backup of those is essential for “Plan B”

Plan B

Make “fresh” install of Rocky. Within, “format” (or recreate) the / and /boot (and /boot/efi if it exists), but only mount the existing /home volume. After install do restore service data from backups.

IMHO, one should have performed Plan B to get Rocky originally – rather than use any (in-place?) conversion from different distro. Rocky does not support conversions, not even from Rocky N to Rocky N+1.

Even CentOS Stream 9 was distinct from CentOS Stream 10. Each of them may roll within – no “point updates” – but one does not roll 9 to 10, AFAIK.

3

I’ll try booting from USB drive and see if the “recovery” method works?

I’m guessing as it looks currently like a linking issue where LD can’t find certain things, it probably just means that a

mkinitcpio -P

hasn’t been performed or perhaps it’s a grub issue where the boot loader needs to be updated

grub-mkconfig -o /boot/grub/grub.cfg

The Zimbra mail system is actually stored in /opt and currently I have a ZFS 1+0 array mounted to store the mail data

Basically /opt/zimbra is on the root drives but the emails themselves are contained on external drives.

It might be if this method fails, I can just backup /opt then reinstall the base OS and Zimbra then replace the /opt/zimbra directory and continue to “repair” the broken system.

For some reason I can’t post links here but there’s actually an official tool (script) called migrate2rocky which I used

4

My bad. I forgot that script and assumed something else.
Yes, conversion of CentOS 8 into Rocky 8 was simple enough to do automatically. In most cases.

5

I’m just about to download an image for Rocky…

I will be installing it from USB disk and the version is going to be 8 Green Obsidian.

I’m confused with what I need.

So I’m looking at the download website and the install instructions which suggests the “Minimal” install.

Will the “minimal” version have the repair tool or do I need the DVD version?

6

The three images are “Boot”, “Minimal” and “DVD”.
All of them have the rescue mode as all of them have the installer and some utilities for it.
Where they differ is what RPM packages are stored in the image:

  • Boot has none as it loads everything from online repos
  • Minimal has just the packages that the Minimal install option would install
  • DVD has all packages

Therefore, no need for the “DVD” image.

7

Thanks!

I’ve downloaded the Minimal iso so I’ll dd that usb key

I’m just checking the “Installation” documentation which I think is the wrong material for my needs but it shows a “troubleshooting” mode which I guess is what I want?

8

So far booted into Troubleshooting mode.

I have root RAID system mounted on /mnt/sysroot (I think that’s the path) as per default

I managed to start networking with auto IP address with:

nmcli con up eno1

From here is it possible to get sshd up and running?

I tried

systemctl restart sshd

but it does not seem to work?

Strange that if I run for dhcp on interface

dhclient eno1

I again get library errors with .so files not found?

I have no idea where to even look at how to repair this now as I really don’t know much about this distro

Will the Arch commands I wrote above work here too?? As in:

mkinicpio -P
grub-mkconfig -o /boot/grub/grub.cfg

I’m not sure really where the errors lie anyway?

9

The first thing that jumps out, is that you didn’t have a Rocky emergency disk sitting next to the computer from when you installed it, my guess is that it’s because it was “upgraded” from centos - a good reason to always do clean install.

Regarding the “troubleshooting” mode of the Rocky install media:

it’s important to remember that it doesn’t log you into the broken underlying o/s; it basically provides an independent shell running in memory (like a “live cd”), so when you type a command in the troubleshooting shell, it runs that command in the troubleshooting shell, not in the underlying o/s. However, you can do a lot with the troubleshooting tool, for example you can mount filesystems from the broken underlying o/s, and look at the files, recover files, change files and so on.

The first step is information gathering, make a note of all underlying disks, partitions, volume groups, and logical volumes. e.g. you could look for your old home directory and save some important files.

The quickest fix is probably just doing clean Rocky install and then copy data from backups, but if you have a lot of time, it can be interesting checking every boot file and tracing where it goes wrong.

There is also a different recovery mode called “emergency” mode, the idea is to boot into the underlying broken o/s, e.g. edit the kernel menu entry in grub and then fix the issue inside the damaged o/s.

10

I don’t want to sound arrogant at all here but I have a full Arch linux posting on recovering a broken system in their forums but that is because I have been using the distro for at least 15 years now.

Obviously I sound like a headless chicken right now because I don’t know Rocky or RPM based distro’s as well so I’m out of my comfort zone here.

Maybe the answer you gave is the right one!

Backup /etc and /opt and just start again…

From the “recovery mode” that I’m in, yes I am in a shell which is the RAMDISK for the USB drive.

Of course I could always chroot into the disk but that means that /dev /proc and /sys are still being used by the USB drive.

From here, I would normally use the

arch-chroot

command which actually mounts these files systems. From a quick Google description:

then I would do a quick “pacman” reinstall of everything…

Regenerate the kenrel initramfs images and then update the GRUB bootloader.

I’m not sure if any of this would work in Rocky as it doesn’t have the “arch-chroot” command and unfortunately I am not sure how the install media works either. I remember from Cent and Fedora that they’re pretty much all GUI or NCurses based, while Arch is entirely a manual install process

No Pacstrap utility either…

Uh… sigh… maybe that’s what I’ll do. Which means I need to buy a new set of disks.

Hmm… In the long run I could just use those new disks to mount /opt and remove it from the general file system so if this happens again at least I have a working /opt directory and it would be fairly safe to reinstall quickly and easily

One quick question does come to mind…

If I chroot into the “live” (busted) system can I still run a:

yum list installed > /path/rpm_pkgs_backup

To at least keep track of installed packages.

Then after reinstall, I can just install anything extra that doesn’t come with the base install

11

IIRC, the Rocky rescue mode does the similar mounts (to /mnt/sysroot).
When chrooted, one has the kernel of the (USB) image, but sees only files of the OS on the disk. The /dev, /proc, /sys are views to kernel, in RAM, not “real” files.

Something in that reinstall process did fail, leaving some packages in inconsistent state. Therefore, some (library) files are missing (or wrong version). All binaries that depend on missing libraries will obviously fail.

The binaries of the rescue image (when you are not chrooted) do have all their libs, so they do work. However, to (re)install RPM packages one would have to use the yum that is within the chroot.

At least the yum works (at least some operations). Without package manager the install of anything is a no-go. (I once managed to remove zlib package and after that no RPM could be installed …)

Note. The yum is a symlink to dnf.

On that list of currently installed packages:

  • If the reinstall did remove a package without installing it again, it will not be in the list
  • Some packages may have been replaced with packages that have different name
12

I don’t like that at all; overwriting hundreds of files when there’s nothing wrong with 99.9% of them, and it probably goes wrong half way through anyway; instead you can use rpm to “verify” a package, it simply checks every file in the package.

For the specific missing file:

rpm -ql ncurses-libs

I can see

/usr/lib64/libtinfo.so.6
/usr/lib64/libtinfo.so.6.2

so it’s worth checking if those files exist on the old disks, but my guess is that even if you fix that, some other bigger error will pop up.

To find the original database error, it’s worth checking the database server log on the old disk, e.g. the day before it went wrong, it might have interesting warnings leading up to the crash.

13

I think I’m more inclined to “play it safe” and use your suggestion initially of just restarting on a clean build then migrating everything on top.

Forgive my responses if they sound blunt, I do really appreciate everyone’s input. To put in context, I have autism (and ADHD) and currently really stressed out and anxious over this. It really feels like learning an entirely new system even though it’s still Linux.

At this stage on Monday I’m gona order 2x new SSD’s which should be with me on Tuesday.

I think the plan here will ultimately be to use the “recovery mode” to format one of the drives in EXT4 then copy:

/root

/etc

/opt

… and even /var across - maybe /var/log only as I think that’s all I really need from that!

Then basically write all installed packages to a file.

At this stage being totally paranoid, best to remove all drives apart from the 2x root drives.

They’re in a RAID mirror array so that’s all good and fine with a structured file system using UEFI boot (separate /boot partition as I guess is standard nowadays with modern systems)

Hopefully I can then just get the installer to use the same hierarchy to perform the “clean” install.

I can then install the Zimbra suite

From there I can boot back into “rescue mode” off the USB drive and migrate /etc and /opt across again.

I have in mind to create a ZFS mirror for the two new disks and mount those specifically as /opt so that it’s separate from the main fs.

After that import the /opt/zimbra/store fs, which is already on a ZFS mirror back into the (hopefully) now working system

Then continue repairing the damaged mysql tables within Zimbra if that even works?

Hopefully my mail system will then be back online again and then I can see about what to do next with it

14

Just a quick reference to the “dumped packages” file…

Once I have a reinstall, I can then probably just create a new dump then simply do something like a

diff -y rpm_old rpm_new

then:

yum install pkg1 pkg2 pkg3

…etc

which is probably more sensible then trying to do a complete package reinstall from “dump file”

15

The way it normally works is you install Rocky to the new SSD, ignoring (or unplugging) the old hard drives. As part of the install you can use “DNF Groups” to decide which packages to install, e.g. “Server without GUI”, when it’s finished, you have perfectly matching packages ready to be used, you can then add additional packages if you need them (from the official Rocky repos).

At that point, you can mount the old drives (maybe temporary) to pull of user data (not old o/s files). You don’t want to contaminate you brand new Rocky build.

Look out for ZFS, I don’t think it’s supported.

16

Thanks!

Just got this from the Rocky docs:

Looks like it’s there…

It was weird, 20 years ago I started with Sun Solaris then transitioned to *BSD. Mainly FreeBSD and OpenBSD.

We’ll see… I’m just gona go prep the system today then hopefully next week I’ll have a recovered base.

Really worried about the Zimbra suite as OpenSource support finished for that a long time ago when the company changed hands.

Anyway hopefully it will all work out…

17

Really interesting thing just happened.

Got to the machine and ran a chroot

So from live environment

chroot /mnt/sysroot

I had this error thrown up:

ERROR: ld.so object 'lib/libsystem_ext.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored

This is from the 8.10 usb minimal install iso?

I’m wondering if this error is what is actually the issue on the root fs on disk and not the usb medium?

I can still access the fs so there’s no problem there.

18

The documentation about ZFS appears to be buried in “LXD Server”. Two chapters higher up has the intro about LXD

19

it usually means you’re underlying o/s has been messed around with; ‘/etc/ld.so.preload’ does not exist by default, so someone must have added it; it’s used to preload shared object files (it’s not usually what you want); in this case I think it’s saying “ignored”, as in I can’t load it, but I don’t care.

20

Hi everyone,

sad news today!

I performed a fresh installation of Rocky 8 and tried to install Zimbra on top.

Unfortunately it doesn’t work at all… I read that many people are complaining that Zimbra cannot be installed on Rocky.

With this being the case… I think I found an old CentOS 8.5 img which I will try to install from

I don’t know if I will ever be able to recover my mail system at this point but we’ll see

I really need to get away from Zimbra as much as possible. I found an alternative being Carbonio but not sure if that project is still alive or not? :frowning: