Login Failure After RL9.1 Update

Rocky Linux General
1

Login to any user including system user fails at tty and via ssh after today’s update that includes kernel-core-5.14.0-162.12.1.el9_1.0.1.x86_64
Reverting to prior kernel and all is OK.

This is the only log information I could find:

Jan 29 08:43:58 bagend systemd[2925]: user@1007.service: Failed to execute /usr/lib/systemd/systemd: Permission denied
Jan 29 08:43:58 bagend systemd[2925]: user@1007.service: Failed at step EXEC spawning /usr/lib/systemd/systemd: Permission denied
Jan 29 08:43:58 bagend systemd[1]: user@1007.service: Main process exited, code=exited, status=203/EXEC
Jan 29 08:43:58 bagend systemd[1]: user@1007.service: Failed with result 'exit-code'.
Jan 29 08:43:58 bagend systemd[1]: Failed to start User Manager for UID 1007.```
2

Strange, I don’t seem to have these symptoms. I just updated to the new kernel now, have rebooted and been able to connect as root as well as regular user via ssh.

Can you double-check permissions for systemd:

[root@rocky9 ~]# ls -lha /usr/lib/systemd/systemd
-rwxr-xr-x. 1 root root 1.9M Jan 23 21:07 /usr/lib/systemd/systemd

maybe also check /var/log/audit/audit.log in case of some selinux stuff.

3

So w/o rebooting yet it does appear to be selinux related, mainly that it is enabled. I have only had one kernel since I installed in December. When I converted my system to UEFI boot I used grub-editenv to add selinux=0 to the kernel parameters. I also edited /etc/default/grub to add the same to the commandline arguments there. I checked the grubenv file there and it contains the kernel option selinux=0 also. There are the two entries under /boot/loader/entries the prior kernel has selinux=0 and omits rhgb and quiet while the new entry adds back rhgb, quiet and omits selinux=0.
So I know how to fix this for now but what is preventing my choices from persisting on new kernel installs?

4

Usually if I disable selinux its in /etc/selinux/config. Never ever done it via grub. I stopped disabling selinux a long time back and prefer to use it.

5

Just running grub2-mkconfig fixed the new kernel commandline in the /boot/loader/entries

I rebooted w/o issues now. Until the next kernel update I won’t know if the issue of parameter carry over to the newest is resolved.

6

The other alternative is check /etc/default/grub and ensure on the GRUB_CMDLINE_LINUX has the selinux=0 entry. Usually though, you don’t need to do that, just edit /etc/selinux/config and change to disabled in there.

7

The entry was present in default/grub.

just edit /etc/selinux/config and change to disabled in there.

As I understand it that route to disabling selinux is being deprecated in future implementations and the kernel parameter will be the only means to disable. But, that aside we still don’t know why the kernel update scripts did not pick up my defaults. I haven’t scoured all the logs yet so the answer may be there.

8

So reviewing the dnf logs it appears that the upgrade did not complete as there was an error message and missing information thereafter. I was using Cockpit for managing and I suspect it presented the reboot option prior to package cleanup operations completing. I"m attempting to do a dnf history redo 26 for that last update and I get these errors:

Cannot find rpm nevra "libXpm-3.5.13-7.el9.x86_64".
  Cannot find rpm nevra "libtiff-4.4.0-2.el9.x86_64".
  Cannot find rpm nevra "dbus-x11-1:1.12.20-6.el9.x86_64".
  Cannot find rpm nevra "dbus-daemon-1:1.12.20-6.el9.x86_64".
  Cannot find rpm nevra "tracker-3.1.2-2.el9.x86_64".
  Cannot find rpm nevra "libtracker-sparql-3.1.2-2.el9.x86_64".
  Cannot find rpm nevra "kernel-headers-5.14.0-162.6.1.el9_1.0.1.x86_64".
  Cannot find rpm nevra "glibc-headers-2.34-40.el9.x86_64".
  Cannot find rpm nevra "glibc-devel-2.34-40.el9.x86_64".
  Cannot find rpm nevra "firefox-102.6.0-1.el9_1.x86_64".

That’s a truncated list of the 65 packages in the transaction. Any insight?

9

Those look like old rpm’s no longer found locally, so not sure that relates to the last transaction. For example, current Firefox is

[root@rocky9 ~]# dnf info firefox
Last metadata expiration check: 2:39:08 ago on Mon 30 Jan 2023 11:17:08 AM CET.
Available Packages
Name         : firefox
Version      : 102.7.0

I would expect it to show the later versions if it was the correct transaction to replay.

10

You’re right. Ugh, had to leave for work so I rushed through and obviously missed the obvious. Also have to reread the man page for dnf to make sure i’m using the syntax correctly.

11

So I reread man dnf and I did have the syntax correct.

If I enter:
dnf history info 26
I get the last transaction with the kernel distinguished as 12.1
If I enter:
dnf history redo 26
I get this completely erronious listing of the replaced packages in this transaction as well as listing the kernel distinguished as 6.1 for install
I ran a separate reinstall of kernel-core-5.14.0-162.12.1.el9_1.0.1.x86_64
and again the install missed my command line options to remove rhgb and quiet. I had to run grub2-mkconfig to correct the kernel parameters.

So I did a comparison with Fedora 37 grub2 packages and found that I was missing 3.

dnf install grub2-tools-extra
dnf install grub2-tools-efi
dnf install grub2-efi-x64-modules

I’ll have to do another reinstall to see if these fix the issue.

12

I don’t have those 3 grub packages. But I have also found that installing kernel, or kernel-core isn’t enough to get grub to generate the config. I simulated this by adding rhgb quiet to my /etc/default/grub. Just trying to figure out now what else could have called it, or a reinstall doesn’t completely run everything like it did during install.

The initial install/upgrade did do everything as intended though or as far as I see.

13

The installer calls grub2-mkconfig. A snippet from /var/log/anaconda/storage.log:

INFO:program:Running in chroot '/mnt/sysroot'... grub2-mkrelpath /boot/grub2
INFO:program:/grub2
DEBUG:program:Return code: 0
INFO:anaconda.modules.storage.bootloader.grub2:bootloader.py: used boot args: resume=UUID=af95761b-0fb7-4fb1-ba06-3d80b9d34042 
INFO:program:Running in chroot '/mnt/sysroot'... grub2-set-default 864fb640b2c749e7a611d2e2305d1f2d-5.14.0-70.5.1.el9_0.x86_64
DEBUG:program:Return code: 0
INFO:program:Running in chroot '/mnt/sysroot'... grub2-mkconfig -o /boot/grub2/grub.cfg

The installation of kernel packages does call /bin/kernel-install:

[root@alma9 ~]# rpm -q --scripts kernel-5.14.0-162.12.1.el9_1

[root@alma9 ~]# rpm -q --scripts kernel-core-5.14.0-162.12.1.el9_1
postinstall scriptlet (using /bin/sh):

if [ `uname -i` == "x86_64" -o `uname -i` == "i386" ] &&
   [ -f /etc/sysconfig/kernel ]; then
  /bin/sed -r -i -e 's/^DEFAULTKERNEL=kernel-smp$/DEFAULTKERNEL=kernel/' /etc/sysconfig/kernel || exit $?
fi
mkdir -p /var/lib/rpm-state/kernel
touch /var/lib/rpm-state/kernel/installing_core_5.14.0-162.12.1.el9_1.x86_64
preuninstall scriptlet (using /bin/sh):
/bin/kernel-install remove 5.14.0-162.12.1.el9_1.x86_64 /lib/modules/5.14.0-162.12.1.el9_1.x86_64/vmlinuz || exit $?
if [ -x /usr/sbin/weak-modules ]
then
    /usr/sbin/weak-modules --remove-kernel 5.14.0-162.12.1.el9_1.x86_64 || exit $?
fi
posttrans scriptlet (using /bin/sh):
if [ -x /usr/sbin/weak-modules ]
then
    /usr/sbin/weak-modules --add-kernel 5.14.0-162.12.1.el9_1.x86_64 || exit $?
fi
rm -f /var/lib/rpm-state/kernel/installing_core_5.14.0-162.12.1.el9_1.x86_64
/bin/kernel-install add 5.14.0-162.12.1.el9_1.x86_64 /lib/modules/5.14.0-162.12.1.el9_1.x86_64/vmlinuz || exit $?

[root@alma9 ~]# rpm -q --scripts kernel-modules-5.14.0-162.12.1.el9_1
postinstall scriptlet (using /bin/sh):
/sbin/depmod -a 5.14.0-162.12.1.el9_1.x86_64
if [ ! -f /var/lib/rpm-state/kernel/installing_core_5.14.0-162.12.1.el9_1.x86_64 ]; then
	mkdir -p /var/lib/rpm-state/kernel
	touch /var/lib/rpm-state/kernel/need_to_run_dracut_5.14.0-162.12.1.el9_1.x86_64
fi
postuninstall scriptlet (using /bin/sh):
/sbin/depmod -a 5.14.0-162.12.1.el9_1.x86_64
posttrans scriptlet (using /bin/sh):
if [ -f /var/lib/rpm-state/kernel/need_to_run_dracut_5.14.0-162.12.1.el9_1.x86_64 ]; then
	rm -f /var/lib/rpm-state/kernel/need_to_run_dracut_5.14.0-162.12.1.el9_1.x86_64
	echo "Running: dracut -f --kver 5.14.0-162.12.1.el9_1.x86_64"
	dracut -f --kver "5.14.0-162.12.1.el9_1.x86_64" || exit $?
fi

I bet that the /bin/kernel-install does not call grub2-mkconfig nor read /et/defaults/grub.

14

That is as I understand what the /boot/grub2/grubenv file is for. To populate that file you use the commands grub2-editenv or grubby. Which I did use to set up the initial environment for the 6.1 kernel and supposedly it is read on the next kernel update but is not.
I’ll have to do some comparison with my fedora install by exploring the install scripts as jlehtone has subsequently posted to see if it sheds some light.

15

One notable difference between el8 and el9 is in kernel options:

el8:

[root@alma8 ~]# cat /boot/loader/entries/*-4.18.0-425.3.1.el8.x86_64.conf 
title AlmaLinux (4.18.0-425.3.1.el8.x86_64) 8.7 (Stone Smilodon)
version 4.18.0-425.3.1.el8.x86_64
linux /vmlinuz-4.18.0-425.3.1.el8.x86_64
initrd /initramfs-4.18.0-425.3.1.el8.x86_64.img $tuned_initrd
options $kernelopts $tuned_params
id almalinux-20221108191600-4.18.0-425.3.1.el8.x86_64
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel

el9:

[root@alma9 ~]# cat /boot/loader/entries/864fb640b2c749e7a611d2e2305d1f2d-5.14.0-162.12.1.el9_1.x86_64.conf 
title AlmaLinux (5.14.0-162.12.1.el9_1.x86_64) 9.1 (Lime Lynx)
version 5.14.0-162.12.1.el9_1.x86_64
linux /vmlinuz-5.14.0-162.12.1.el9_1.x86_64
initrd /initramfs-5.14.0-162.12.1.el9_1.x86_64.img
options root=UUID=6b..ac ro resume=UUID=af..42 rd.driver.blacklist=nouveau
grub_users $grub_users
grub_arg --unrestricted
grub_class almalinux

That is, while el8 got the options from grubenv or grub.cfg (the $kernelopts),
the el9 has them hard-coded in each entry.

How the entries (separate files when BLS is used) are created and modified? How the different utilities do it? Ultimately, are they consistent?

16

The kernel install scripts are here:
/usr/lib/kernel/install.d/
The one of particular interest is 20-grub.install
Comparing that with the one in Fedora37 there are some distinct differences.

RL9.1

        if [[ "x${GRUB_ENABLE_BLSCFG}" = "xtrue" ]] || [[ ! -f /sbin/new-kernel-pkg ]]; then
            if [[ -f /etc/kernel/cmdline ]]; then
                read -r -d '' -a BOOT_OPTIONS < /etc/kernel/cmdline
            elif [[ -f /usr/lib/kernel/cmdline ]]; then
                read -r -d '' -a BOOT_OPTIONS < /usr/lib/kernel/cmdline
            else
                declare -a BOOT_OPTIONS

                read -r -d '' -a line < /proc/cmdline
                for i in "${line[@]}"; do
                    [[ "${i#initrd=*}" != "$i" ]] && continue
                    [[ "${i#BOOT_IMAGE=*}" != "$i" ]] && continue
                    BOOT_OPTIONS+=("$i")
                done
            fi

Fedora37

        if [[ "x${GRUB_ENABLE_BLSCFG}" = "xtrue" ]] || [[ ! -f /sbin/new-kernel-pkg ]]; then
            if [[ -f /etc/kernel/cmdline ]]; then
                if [[ /etc/kernel/cmdline -ot /etc/default/grub ]]; then
                    # user modified /etc/default/grub manually; sync
                    grub2-mkconfig -o /etc/grub2.cfg
                fi
                read -r -d '' -a BOOT_OPTIONS < /etc/kernel/cmdline
            elif [[ -f /usr/lib/kernel/cmdline ]]; then
                read -r -d '' -a BOOT_OPTIONS < /usr/lib/kernel/cmdline
            else
                declare -a BOOT_OPTIONS

                read -r -d '' -a line < /proc/cmdline
                for i in "${line[@]}"; do
                    [[ "${i#initrd=*}" != "$i" ]] && continue
                    [[ "${i#BOOT_IMAGE=*}" != "$i" ]] && continue
                    BOOT_OPTIONS+=("$i")
                done
            fi

So RL9 does not recognize /etc/default/grub or the grubenv file, why?
How do I populate /etc/kernel/commandline?

17

So I found this in the RHEL 9 documentation.

This is stupid.

18

I was wondering that too. The /etc/kernel/cmdline in my system was written last month.

With inconclusive google hits, I tried blunt instrument: grep -r cmdline /etc
Look at function update_bls_cmdline() in /etc/grub.d/10_linux

1 Like
19

So jlehtone, that was the hint that finally solved it for me. In the end I created /etc/kernel/cmdline by hand and entered the options line parameters from the working kernel .conf in /boot/loader/entries/

I uninstalled the latest kernel and reinstalled it via dnf upgrade and the correct parameters are now present and I can move on.

20

I was wondering too
thanks!