I am testing Rocky Linux as an alternative for CentOS to test homelab/work related stuff. I usually run Debian.
SSH connection is taking aaaaages and the solutions found all over the internet didn’t work.
Just like the Debian VMs, I do:
Set up the VM on Proxmox
Install SSH
ssh-keygen
nano ssh./authorized_keys to add my pc public key
From now on I ssh to the box without password.
With Rocky Linux tho something is off:
Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)
debug1: No credentials were supplied, or the credentials were unavailable or inaccessible
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)
I did set both GSSAPIAuthentication no and UseDNS no but it is still taking ages to authenticate.
When running Ansible playbook against the server, this delay is killing me haha
Are you sure Kerberos credential lookup is the part that is taking the most time? You can run output of ssh -vvv through ts (from moreutils) which prepends a timestamp to the output as sub-second resolution:
Yes, I tried that -vvv and it gets stuck on kerberos part even so I disabled it.
I’ve SSH many servers, work, homelab, this is the first time with such weird behavior.
Is this taking around 30 seconds or so? If so it’s possibly a DNS issue on the remote server trying to do an rDNS lookup of your IP address. In my experience this is the most common cause of slowness ssh’ing into a server.
Not the only one, of course, but it’s always where I start looking!
It isn’t taking thaaaat long.
I run Pi-Hole + Unbound Recursive DNS as my DNS for years.
Never had such issue so I don’t think it’s it.
I usually run Debian for homelab, no issues. I have ran CentOS in the past, no issues. My router run FreeBSD, no issues.
The company runs AWS Cloud so I need a YUM Linux to test stuff to transfer to AWS Linux (YUM).
I found out this Rocky Linux is a strong alternative to CentOS. The tests were fine, Ansible playbook was fine, just this seconds delays with SSH is killing me haha
I’ll check everything again but yeah, SSH isn’t this headache
Thanks for mentioning moreutils coz searching for ts was getting me nowhere haha
It is getting stuck here, it takes good seconds there before proceeding.
So right now, my /etc/ssh/sshd_config with no changes:
KerberosAuthentication no
GSSAPIAuthentication no
UseDNS no
ChallengeResponseAuthentication no
It is taking around 20s to grant me with the terminal access.
Like I mentioned before, between 10 or so DEB based servers and FreeBSD (router), this VM is the only one with such behavior. I have excluded network issues or Proxmox for the matter.
I can still use it in the worst case scenario of course, it is just this delay that sucks.
Will do some more investigation but yeah, I have no much idea where to start from coz ssh just works haha
I use key (id_rsa) for authentication over password but it still takes time.
H4kuna
what about the debug from the server end, youve not shown us whats happening their
its important to look at the logs at the server side rather than the client, as its the server thats the issue.
i have seen this exact behaviour, but not on any of my Rhel based installs, including rocky, where i have seen it was a debian bullseye install.
Thing is its not ssh per say that is causing the issue its logging is as root that is causing the issue.
so if i ssh in as say admin, no issue, but if i su - then 20 second delay.
so if you are logging in directly as root, you will also see that.
I also run all my machines under proxmox, so we have that in common.
so create another user say admin and see if that has the same issue.
it may be related to this issue, and i remember now i fixed it by adding a cron job to update the write permissions on that /var/run file mentioned in the report. https://bugzilla.redhat.com/show_bug.cgi?id=1923086
there is a report on debian that is similar as well, #770135 - systemd: ssh logins considerably delayed (until PAM timeout) when systemd is upgraded but the system not rebooted - Debian Bug report logs
and finaly the cause of the issue i saw which is lxc debian under proxmox is fixed as follows. Fix Debian 11 bullseye slow SSH login and sudo on LXC Proxmox · GitHub
To be fair I am tired of this lmao
And a quick search tells I am not alone with this problem with Rocky, like I said before “SSH isn’t this mess”
I have a bunch of Linux servers, it is the first time having this problem.
I never ssh as root, I always use create an user, set the keys, authorized_keys and only then ssh the VMs/baremetal
I did try to ssh as root for the sake of “I tried”, same thing.
I did check the logs and there is no error, journalctl -t sshd does show main: sshd: ssh-rsa algorithm is disabled but that is all.
Anyway, thank you for the help guys, I will try to find another YUM alternative that won’t give me headache with a simple SSH and like I said, I am not alone.
not journalctl the sshd -DDDD -p666 output.
you posted the ssh -vvv but not the server log where it hangs.
it has to be something with your setup as i have several rocky/centos/rhel installs on my proxmox and they are not experiencing this at all.
regards peter
I tried AlmaLinux and the same issue so now I am pissed lol
Even so I had this line on the server side: GSSAPIAuthentication no
Nothing seems to be working and the whole file is being ignored.
I did some more digging and tried ssh -o GSSAPIAuthentication=no HOST
No issues but the goddamn line is already in place under /etc/ssh/sshd_config
idk man, this must be a YUM based system thing
To make the changes permanent, I had to add this line on the client side: .ssh/config
Host *
GSSAPIAuthentication no
Dear lord… 15y plus using DEB based Linux, managed a bunch of CentOS in the past tho, I have never encountered this problem.
what path are you using for your sshd_config file.
is there an error in the file that is making sshd ignore it.
start with a clean file
this is what i use but it will only allow root to use certificate based auth
#############
cat /etc/ssh/sshd_config
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTHPRIV
PermitRootLogin no
AuthorizedKeysFile .ssh/authorized_keys
PermitEmptyPasswords no
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
PermitRootLogin without-password
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem sftp /usr/libexec/openssh/sftp-server
################
There are no errors or it would fail after a restart.
I don’t touch that file, I setup the Linux home lab server, SSH to it and that’s it. I’ve had zero reasons to touch that file.
I do the opposite, root isn’t allowed to SSH at all with exception of my OPNSense Firewall/Router as fail-safe in case I lock my user, only my user SSH the servers with certificate based auth.
well if you have not added the items to that file that we have discussed through this ie
GSSAPIAuthentication no
then it wont fix it.
GSSAPIAuthentication no is set at the server end
It can be set at both ends. There are options for GSSAPIAuthentication present in ssh_config which is client-side, and sshd_config which is server side.
Debian defaults to yes on the client-side but no on the server-side. Rocky defaults to no on the client-side, but yes on the server-side.
It’s certainly far easier to apply some configuration options client-side, simply that it means I apply it once on my laptop, rather than configure 50-100 servers.
Not entirely sure, what OS is installed on the machine connecting to Rocky in this instance, but obviously there could have been the chance GSSAPIAuthentication was yes at both ends, hence why he solved it by overriding in .ssh/config.
i guess it depends on how things are setup
if you have 20 sysadmins all accessing the servers from there workstations it pays to have it setup in the servers build scripts.
True, there could also be business policies in place that set certain options server-side, and I may personally wish to disable certain options on my side because I may or may not need to use what was set server-side. Each can then have individual configuration based on their requirements.
