I am running RL 8.5 and just did a dnf update. When I rebooted I get the message indicated. I tried the previous kernel but that didn’t make a difference. I’ve waited for more than 20 minutes and the job doesn’t finish. Any suggestions on how to debug this?
Thanks in advance.
I narrowed it down to systemd-tmpfiles-setup-dev.service
I saw a post somewhere about someone having the problem when the root directory was no longer owned by root. I checked that with stat -c “%U %G” and the ownership is correct.
I thought it might be SELinux (which is enabled), so I appended enforcing=0 to the end of the command line.
I figured it out. This article describes the problem:
It turns there is a problem with udev initialization that has to do with groups. I use NIS and had nis first for groups in nsswitch.conf. This causes a problem because at the time this service starts rpcbind is not yet started. This lead to a failure in starting udev Kernel Device Manager, which in turn led to a failure in creating the /dev static nodes. At least that is my understanding. I fixed it, as suggested in the above link, by putting files first in the group line of nsswitch.conf.
Debugging this was kind of hard. What ultimately helped was stepping through the spawned processes one-by-one and carefully looking at the log messages. It didn’t work to try to boot into emergency or rescue mode because the failure happened earlier in boot than those things.
To do that I added:
plymouth.enable=0 systemd.confirm_spawn=true systemd.debug-shell=1 systemd.log_level=debug systemd.unit=multi-user.target
to the end of my boot line. Having the debug shell and confirming what was happening step-by-step was helpful. These boot entries came from:
https://www.thegeekdiary.com/how-to-debug-systemd-boot-process-in-centos-rhel-7-and-8/
2 Likes