Weird SSH Issue on 9.3, SSH login fails until another user logs in

Rocky Linux General
1

Hello,

Clean install from 9.3 and did a dnf update then created sudo user account. Everything is fine and I go through the rest of the setup (a virtualization host with Gnome GUI) then reboot. Now I can’t login via SSH to my sudo user. I enable SSH login for root and that doesn’t work either.

Now for the strange part. I create a new user and they work via SSH and now suddenly my sudo user and root user work too (via SSH!

I reboot and again the sudo user doesn’t work via SSH, however if I login to SSH via the temp user it works and now I can also login as the sudo user again… this is the same console window and the password for the sudo user is copied and pasted from the clipboard…

So in summary:

  1. server is rebooted and sudo user can’t login via ssh (password was pasted)
  2. same console window is used to login using temp user and it works (password was typed), then I exit temp user
  3. same console window is used and sudo user can now login (password was still in clipboard from step one)

Below are the relevant entries with notes from /var/log/secure

Apr  4 22:22:21 SERVER sudo[39588]: pam_unix(sudo:session): session opened for user root(uid=0) by PROBLEM_USER(uid=0)
Apr  4 22:22:21 SERVER sudo[39588]: pam_unix(sudo:session): session closed for user root
Apr  4 23:17:03 SERVER su[39533]: pam_unix(su:session): session closed for user root
Apr  4 23:17:03 SERVER sudo[39529]: pam_unix(sudo:session): session closed for user root
Apr  4 23:17:05 SERVER sshd[39227]: Received disconnect from 10.51.254.11 port 58510:11: disconnected by user
Apr  4 23:17:05 SERVER sshd[39227]: Disconnected from user PROBLEM_USER 10.51.254.11 port 58510
Apr  4 23:17:05 SERVER sshd[39204]: pam_unix(sshd:session): session closed for user PROBLEM_USER
#######I exited the SSH session here for the night########
Apr  4 23:25:31 SERVER gdm-password][681578]: gkr-pam: unable to locate daemon control file
Apr  4 23:25:31 SERVER gdm-password][681578]: gkr-pam: stashed password to try later in open session
Apr  5 08:12:31 SERVER sshd[1215766]: Accepted password for PROBLEM_USER from 10.51.254.11 port 51582 ssh2
Apr  5 08:12:31 SERVER sshd[1215766]: pam_unix(sshd:session): session opened for user PROBLEM_USER(uid=1000) by (uid=0)
#######I logged in via SSH successfully (there was no reboot)########
Apr  5 08:12:36 SERVER polkitd[36831]: Registered Authentication Agent for unix-process:1215988:3677996 (system bus name :1.170 [/usr/bin/pkttyagent --process 1215988 --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Apr  5 08:12:36 SERVER polkitd[36831]: Unregistered Authentication Agent for unix-process:1215988:3677996 (system bus name :1.170, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Apr  5 08:12:43 SERVER gdm-password][1216136]: gkr-pam: unable to locate daemon control file
Apr  5 08:12:43 SERVER gdm-password][1216136]: gkr-pam: stashed password to try later in open session
Apr  5 08:17:22 SERVER polkitd[5280]: Loading rules from directory /etc/polkit-1/rules.d
Apr  5 08:17:22 SERVER polkitd[5280]: Loading rules from directory /usr/share/polkit-1/rules.d
Apr  5 08:17:22 SERVER polkitd[5280]: Finished loading, compiling and executing 10 rules
Apr  5 08:17:22 SERVER polkitd[5280]: Acquired the name org.freedesktop.PolicyKit1 on the system bus
Apr  5 08:17:24 SERVER sshd[5586]: Server listening on 0.0.0.0 port 22.
Apr  5 08:17:24 SERVER sshd[5586]: Server listening on :: port 22.
#######The server was rebooted########
Apr  5 08:17:30 SERVER systemd[5900]: pam_unix(systemd-user:session): session opened for user gdm(uid=42) by (uid=0)
Apr  5 08:17:30 SERVER systemd[5899]: pam_unix(systemd-user:session): session opened for user PROBLEM_USER(uid=1000) by (uid=0)
Apr  5 08:17:30 SERVER gdm-launch-environment][5836]: pam_unix(gdm-launch-environment:session): session opened for user gdm(uid=42) by (uid=0)
Apr  5 08:17:30 SERVER vncsession[5847]: pam_unix(tigervnc:session): session opened for user PROBLEM_USER(uid=1000) by (uid=0)
#######VNC Server and Gnome GUI are are present, is this why a gdm session is opened without me doing anything?########
Apr  5 08:17:31 SERVER polkitd[5280]: Registered Authentication Agent for unix-session:c1 (system bus name :1.27 [/usr/bin/gnome-shell], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Apr  5 08:17:41 SERVER polkitd[5280]: Registered Authentication Agent for unix-session:1 (system bus name :1.62 [/usr/bin/gnome-shell], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Apr  5 08:17:48 SERVER unix_chkpwd[7386]: password check failed for user (PROBLEM_USER)
Apr  5 08:17:48 SERVER sshd[6798]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.51.254.11  user=PROBLEM_USER
Apr  5 08:17:50 SERVER sshd[6798]: Failed password for PROBLEM_USER from 10.51.254.11 port 40510 ssh2
Apr  5 08:18:01 SERVER unix_chkpwd[7387]: password check failed for user (PROBLEM_USER)
Apr  5 08:18:03 SERVER sshd[6798]: Failed password for PROBLEM_USER from 10.51.254.11 port 40510 ssh2
#######After the reboot I tried logging in via SSH twice, via copying and pasting the password, it failed.########
Apr  5 08:18:33 SERVER sudo[7446]:  PROBLEM_USER : TTY=pts/0 ; PWD=/home/PROBLEM_USER ; USER=root ; COMMAND=/bin/su
Apr  5 08:18:33 SERVER sudo[7446]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Apr  5 08:18:33 SERVER su[7452]: pam_unix(su:session): session opened for user root(uid=0) by (uid=0)
#######Here I logged in via VNC to Gnome-session as PROBLEM_USER then sudo su in preparation to create a new user but decided to test something else########
Apr  5 08:18:40 SERVER sshd[6798]: Connection closed by authenticating user PROBLEM_USER 10.51.254.11 port 40510 [preauth]
Apr  5 08:18:40 SERVER sshd[6798]: PAM 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.51.254.11  user=PROBLEM_USER
Apr  5 08:18:55 SERVER sshd[7493]: Accepted password for TEMP_USER from 10.51.254.11 port 46026 ssh2
Apr  5 08:18:55 SERVER systemd[7502]: pam_unix(systemd-user:session): session opened for user TEMP_USER(uid=1001) by (uid=0)
Apr  5 08:18:55 SERVER sshd[7493]: pam_unix(sshd:session): session opened for user TEMP_USER(uid=1001) by (uid=0)
Apr  5 08:19:03 SERVER sshd[7525]: Received disconnect from 10.51.254.11 port 46026:11: disconnected by user
#######Here I logged in via ssh as a previously created test user and it worked, then exited########
Apr  5 08:19:03 SERVER sshd[7525]: Disconnected from user TEMP_USER 10.51.254.11 port 46026
Apr  5 08:19:03 SERVER sshd[7493]: pam_unix(sshd:session): session closed for user TEMP_USER
Apr  5 08:19:13 SERVER sshd[7588]: Accepted password for PROBLEM_USER from 10.51.254.11 port 36620 ssh2
#######Then is the same console session tried the PROBLEM_USER and it worked....########

Any ideas or is more info needed?

LUKS2 is in use on root and swap parition. Tiger VNC server is installed and configured. ZFS is used. Fail2ban is used (and will ban me after failed attempts, but even with fail2ban removed the login issue is still present). Dell’s OpenManage is installed. This has happened across 2 clean installs.

Thank you.

1 Like
2

I’ve found the cause… it wasn’t the Rocky install specifically but it was strange anyway.

I did a clean install of Rocky 9.3 on a different machine and after extensive testing found it was an issue with the “Konsole” version/session/settings I was using when logging via SSH to the Rocky 9.3 system.

The weird part is that if I paste the password after a fresh reboot of the Rocky system it doesn’t work, but if I type it in manually it does… and here is the really strange part: after I login successfully via typing in the password manually I “exit”, hit the up arrow to log back in and this time can paste the password using the same clipboarded password from the failed attempt… it’s repeatable over and over.

I then used tmux on the same computer and putty from another but those worked fine with pasting the same password on the first attempt… back to konsole and the strange issue persists. So then I boot up a Rocky 8.9 install and the copy and paste work just fine from Konsole, reboot and it still works just fine…

At this point it seems to be something specific to Konsole/my profile and Rocky 9.3 since it works elsewhere… but I don’t see how this could be happening…

Anyway, I have a work around now but still am curious where I can look next to explain the issue.

Thank you!