You are in emergency mode. After logging in, type “journalctl -xb” to view system logs, “systemctl reboot” to reboot, “systemctl default” or “exit” to boot into default mode.
Cannot open access to console, the root account is locked. See sulogin(8) man page for more details.
Greetings @ide.nurhalim ,
Ooph. I’m sorry to hear that. I’m guessing this is the 9.1 → 9.2 update. We did discover a bug that exists upstream if you are running LVM. If you know you are using LVM, let me know the work-around is easy to do.
If you don’t know, or if you don’t have LVM then you will need to investigate the system logs. Since you have a locked root account, you will have to investigate with a live CD/USB. If you need more information about this, I can write up a few things to try.
~Stack~
What bug is that with lvm, my systems setups are using lvm and I haven’t noticed any issues?
Upstream made a change in how devices are detected. If a device has an extra _ at the end of it’s IDNAME in /etc/lvm/devices/system.devices , then it won’t boot. Removing the _ should make it boot again.
When the initial installation of my root account I may lock, I forgot about it, if that happens how can I run the command?
Greetings @ide.nurhalim
If you don’t have a root password, you may need to reset it. Try the Method #3 password reset here:
Also, before you exit out of the system (after touch /.autorelabel) check that there isn’t an _ after the IDNAME
cat /etc/lvm/devices/system.devices
If there is, edit the file and remove it.
Hope that helps.
I am having the same issue. However, I do know my root password, which I entered and saw the journalctl -xb output. With my limited understanding, it seems to point to an issue with dev/mapper which is timing out.
I´ve been using many different Linux distros for years as my daily drivers, but I gotta admit my shortcomings as a system admin. I just don’t know what to do about this one right now. I am googling my way out as usual… but can’t find much info right now. I can boot successfully using the previous kernel which is how I am typing this right now.
cat /etc/lvm/devices/system.devices show this:
# LVM uses devices listed in this file.
# Created by LVM command lvmdevices pid 3908 at Sat Aug 6 19:09:27 2022
VERSION=1.1.3
IDTYPE=sys_wwid IDNAME=eui.6479a72160163232 DEVNAME=/dev/nvme0n1p3 PVID=F8LLtaFytE2tnkBxeLg8X5sQUVP0gYUF PART=3
IDTYPE=sys_wwid IDNAME=eui.6479a74580200134 DEVNAME=/dev/nvme2n1p3 PVID=01n4N27WELBW0edct0uaKNgUubZEIGji PART=3
my fstab looks like this:
#
# /etc/fstab
# Created by anaconda on Sat Aug 6 19:09:27 2022
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/rl_localhost--live00-root / xfs defaults 0 0
UUID=c62da415-c05b-41c2-ac3e-0d9c2d01da7a /boot xfs defaults 0 0
UUID=444D-85F8 /boot/efi vfat umask=0077,shortname=winnt 0 2
/dev/mapper/rl_localhost--live00-home /home xfs defaults 0 0
/dev/mapper/rl_localhost--live00-swap none swap defaults 0 0
I have a production server for the hospital, because like this I am really crazy and have not found a solution, so I use a new server and restore the database
I just fixed my problem. I did not have a leading space or underscore as suggested by other users. However, look here https://bugzilla.redhat.com/2208039
Try removing the system.devices file. I’m not implying this will work for everyone, but it did for me:
rm /etc/lvm/devices/system.devices
Then restart, and it should boot.
I honestly don’t know what I was doing, I don’t even know how lvm works to be honest. I was willing to format and do a fresh install… so, I had nothing to lose. Luckily for me, it worked!
thank u, i will try again later
This is what I found while running the update:
Running scriptlet: kernel-core-5.14.0-284.11.1.el9_2.x86_64 705/705
Devices file sys_wwid t10.ATA_____RMX-256GB_______________________________AA202207040135______ PVID hsfbatPjDgode28LSWAOhAV3Vx164XF1 last seen on /dev/sdb6 not found.
Devices file sys_wwid t10.ATA_____RMX-256GB_______________________________AA202207040135______ PVID hsfbatPjDgode28LSWAOhAV3Vx164XF1 last seen on /dev/sdb6 not found.
Running scriptlet: kernel-modules-5.14.0-284.11.1.el9_2.x86_64 705/705
Hi all,
We’ve added a section to the release notes for 9.2 which speaks about this issue as well as some workarounds.
Hopefully this helps! Current Release 9.2 - Documentation
Thank you so much for this! Tried so many things before finding this and this is the only thing that worked for me.
@danielp statement was the only one that worked for me. Here is the details i can offer to this situation. Looking at the Rocky teams comment they stated the following
If you are impacted by this bug, then until the bug fix is released by RHEL and Rocky, please review and follow one of the following options before upgrading and rebooting your system:
- After upgrading the
lvm2package, but prior to rebooting, regenerate/etc/lvm/devices/system.devicesby runninglvmdevices --update. Confirm that the contents of/etc/lvm/devices/system.devicesis changed to match the new format before rebooting the system.- Prior to rebooting after upgrading, rename the
/etc/lvm/devices/system.devicesfile (e.g.,mv /etc/lvm/devices/system.devices{,.bak}) and, once the system has rebooted, runvgimportdevices --allto regenerate the file in the new format.A correctly formatted
system.devicesfile for Rocky Linux 9.2 will not have repeated underscores in the device’s IDNAME field.
I am doing packer.io builds on AWS and would get the error of the system being locked out which was strange as this was a stable build before. I added a line in the build to get version info after full yum update and I had the following
NAME="Rocky Linux"
VERSION="9.2 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.2 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.2"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"
and started with a base AMI of: ami-0fe64c0692c69d851 (note was using the most recent but due to all the errors went down a version)
After the LVM creation portion of the build i had a cat on system.devices to see if it existed and if anything was in it. It output the following:
# Created by LVM command vgcreate pid 33528 at Tue Jul 25 14:07:14 2023
VERSION=1.1.2
IDTYPE=sys_wwid IDNAME=nvme.1d0f-766f6c3037313836373630343065626565336134-416d617a6f6e20456c617374696320426c6f636b2053746f7265-00000001 DEVNAME=/dev/nvme1n1 PVID=7O00WcnQ4mNxWHpjXpXkse99rLBnhowr
IDTYPE=sys_wwid IDNAME=nvme.1d0f-766f6c3035303838636538663462626538363632-416d617a6f6e20456c617374696320426c6f636b2053746f7265-00000001 DEVNAME=/dev/nvme2n1 PVID=32SH2Xsr0jpfOYvUQy0tmqaSm1e7hvVf
Note there are no underscores found and there looks to be no spaces. If i let the build just finish out just like that, it will not boot in amazon.
If i perform a lvmdevices --update after the LVM creation it does not solve this problem.
It is only when I specifically rm system.devices that once I spin up the completed build all the lvm mounts actually mount.
when i look for system.devices i get that it does not exist like expected
[root@ip-10-2-7-27 rocky]# cat /etc/lvm/devices/system.devices
cat: /etc/lvm/devices/system.devices: No such file or directory
I can restart and completely halt the system and every time it starts back up. After performing vgimportdevices --all i get the following:
# Created by LVM command vgimportdevices pid 1652 at Tue Jul 25 14:36:58 2023
VERSION=1.1.1
IDTYPE=sys_wwid IDNAME=nvme.1d0f-766f6c3034666461306538633339626337636366-416d617a6f6e20456c617374696320426c6f636b2053746f7265-00000001 DEVNAME=/dev/nvme1n1 PVID=7O00WcnQ4mNxWHpjXpXkse99rLBnhowr
IDTYPE=sys_wwid IDNAME=nvme.1d0f-766f6c3032386164396637623761386664633765-416d617a6f6e20456c617374696320426c6f636b2053746f7265-00000001 DEVNAME=/dev/nvme2n1 PVID=32SH2Xsr0jpfOYvUQy0tmqaSm1e7hvVf
Everything works.
So after offer that verbose amount of detail i just want to say that it does not seem to be limited to just IDNAME’s with underscores/spaces because as shown above my system.devices would like fine like @danielp’s in our build processes but then not start. Also the first option that Rocky Team offered did not work for us.
Hope this someone looking for more detail.
This Workaround worked for me. I got into this issue after a power outage. Thanks!
Workaround:
$ rm -f /etc/lvm/devices/system.devices