Problem with ssh server on rocky 8.5

Hi All,

Could you please look on sshd_config file and tell me what could cause problem?
I can login after removing key from .ssh/known_host but after few seconds connection is broken

-bash-4.2$ ssh 192.168.90.55
ssh: connect to host 192.168.90.55 port 22: Connection refused

after few try as above i got

-bash-4.2$ ssh rocky
The authenticity of host ‘rocky (192.168.90.55)’ can’t be established.
ECDSA key fingerprint is SHA256:yGVO3a2TFv7+oDscv2RqxXayi3P2LTemM3uSJPpvhfQ.
ECDSA key fingerprint is MD5:70:d7:01:38:5b:92:17:10:ea:fa:39:5b:3e:e9:77:ab.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘rocky,192.168.90.55’ (ECDSA) to the list of known hosts.
tomasz@rocky’s password:
Activate the web console with: systemctl enable --now cockpit.socket

This system is not registered to Red Hat Insights. See https://cloud.redhat.com/
To register this system, run: insights-client --register

below is sshd_config

Thank you

#	$OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

# If you want to change the port on a SELinux system, you have to tell
# SELinux about this change.
# semanage port -a -t ssh_port_t -p tcp #PORTNUMBER
#
Port 22
Protocol 2
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# This system is following system-wide crypto policy. The changes to
# crypto properties (Ciphers, MACs, ...) will not have any effect here.
# They will be overridden by command-line options passed to the server
# on command line.
# Please, check manual pages for update-crypto-policies(8) and sshd_config(5).

# Logging
#SyslogFacility AUTH
#SyslogFacility AUTHPRIV
SyslogFacility AUTH
LogLevel INFO

# Authentication:

#LoginGraceTime 2m
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

PubkeyAuthentication yes

# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile	.ssh/authorized_keys

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
PermitEmptyPasswords no
PasswordAuthentication yes

# Change to no to disable s/key passwords
#ChallengeResponseAuthentication yes
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
#KerberosUseKuserok yes

# GSSAPI options
GSSAPIAuthentication yes
GSSAPICleanupCredentials no
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no
#GSSAPIEnablek5users no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
# WARNING: 'UsePAM no' is not supported in Fedora and may cause several
# problems.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes

# It is recommended to use pam_motd in /etc/pam.d/sshd instead of PrintMotd,
# as it is more configurable and versatile than the built-in version.
PrintMotd no

#PrintLastLog yes
TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
ClientAliveInterval 30
ClientAliveCountMax 99999
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Accept locale-related environment variables
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS

# override default of no subsystems
Subsystem	sftp	/usr/libexec/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#	X11Forwarding no
#	AllowTcpForwarding no
#	PermitTTY no
#	ForceCommand cvs server

Your pasting of the sshd_config file turned into a bit of a mess, but I did copy-paste it into vim and having taken a very quick look at it nothing really leaps up and says “look at me!”

Unfortunately it’s a bit difficult to be sure of exactly what you have in sshd_config because of the aforementioned mess, so there’s no guarantee. :slight_smile:

The next thing to do is to try sshd using the default sshd_config file as-shipped with Rocky Linux and see if that solves the problem. If it does, then make only one change at a time and when the the bad things happen, the last thing you changed is the problem.

If it still doesn’t work when you use the default sshd_config file, then sshd_config is not your problem.

Hashtag on sshd_config means comment, but in here it is a header style?
I often like to look at the stripped down version, with no comments or empty lines with:

grep "^[^#]" filename

Posting within code-block does not use hashtag as style command.

Apparently, EL9 will have /etc/ssh/sshd_config.d/ for dropping in customizations.
EL8 has already /etc/ssh/ssh_config.d/

-bash-4.2$ grep “[1]” sshd_config
Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTH
LogLevel INFO
PermitRootLogin yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
IgnoreRhosts yes
PermitEmptyPasswords no
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials no
UsePAM yes
X11Forwarding yes
PrintMotd no
TCPKeepAlive yes
ClientAliveInterval 30
ClientAliveCountMax 99999
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem sftp /usr/libexec/openssh/sftp-server

Thank you


  1. ^# ↩︎

Did you try sshd using the default sshd_config file as-shipped with Rocky Linux? Did that solve the problem? If not, did that change the problem? (Meaning, did any error or status messages change in any way?)

This will allow you to determine if sshd_config is actually the issue or not.

making changes in sshd_config doesnt change the symptoms
the symptoms are that the connection is suddenly broken. it could be one time per hours or multiple times during ten minutes.
there are three servers 10.11.20.35, 192.168.90.30 and 192.168.90.55
problem with connection is on 192.168.90.55

when was able to login I logged to 192.168.90.55 from both others servers and nmaped 192.168.90.55 from 10.11.20.35 and got
PORT STATE SERVICE
22/tcp open ssh
111/tcp open rpcbind
3389/tcp open ms-wbt-server

when connection is broken nmap shows
PORT STATE SERVICE
2323/tcp open 3d-nfsd
7000/tcp open afs3-fileserver

interesting is that connection from 10.11.20.35 is always broken when nmap shows open ports 2323 and 7000 but very rarely connection from 192.168.90.30 is broken (but it also happens).

do you have any idea what is going on here?

Thank you

If the default sshd_config doesn’t make the problem go away then it’s probably not sshd_config that’s the problem.

ssh -vvv whateveryourloggininto

That might tell you something.

Are these services configured on the 192.168.90.55 box? I find it hard to believe that they would just appear when the connection goes down. Sounds more like to me, that there is already a machine/device using that same IP, and they are fighting for the network sometimes one replies, and sometimes another, and the port differences being two machines colliding with IP conflict.

Easiest would be to shutdown the Rocky server and then try to nmap the 192.168.90.55 address and see if it replies with anything, including ping.

I might be wrong, but it looks like this kind of situation. Could also check firewall configurations if there is a firewall in between these network segments.

BINGO!
Thank you !
tomasz

I’m not the administrator of these networks. I’m the only user which is trying to help to find source of problem… here is answer of administrators when I send your findings.

We believes that one of the maintenance me has a device that he uses occasionally.
Perhaps that is where the conflict lies. We can change IP’s if you would like.

1 Like

Yup would be good to get them to change the ip and you can use that new ip for the machine.

In that sense DHCP is awesome: all addresses are allocated by one party and it is their misconfiguration, if there are collisions.

Don’t some methods, like the manual ad hoc ip address add actually check for conflicts?

arp is your friend here.
record the MAC address of your system Ethernet card
Command: ip a
(one of mine: link/ether 00:1a:4a:16:01:97 brd ff:ff:ff:ff:ff:ff)
MAC address 00:1a:4a:16:01:97
On another system, ping the system in question, then run the arp command
If your system in question MAC address is not on the list, bob’s your uncle, something else has the IP address.

arp result example:

$ arp
Address HWtype HWaddress Flags Mask Iface
ip72.somedomain.com ether 00:1a:4a:16:01:7c C ens4
192.168.9.12 ether 00:1a:4a:16:01:93 C ens3
ip75.somedomain.com ether 00:1a:4a:16:01:7c C ens4
ip77.somedomain.com (incomplete) ens4
192.168.9.50 ether d4:ae:52:bd:e4:60 C ens3
192.168.9.30 ether 00:1a:4a:16:01:a9 C ens3
_gateway ether 00:1c:73:00:00:80 C ens4

(Edit: these are really quit fun to track down, hopefully arp finds it, otherwise something like the shark packet trace thingy will need to come out and play.)

Guys,
problem solved. here is the answer of it administrators:
‘There was a piece of lab equipment that had the same IP’

Thank you !!!

1 Like