Upgrade linux9 openssh-server problem

Rocky Linux General
1

After upgrading linux9, I have been using mac’s terminal to log in, these are no problem, when I log in using software such as xshell, the session will end immediately. I am using the key file login method and the following is the log when restarting

.
Jul 25 22:15:48 localhost systemd[1]: Stopping OpenSSH server daemon...
Jul 25 22:15:48 localhost systemd[1]: sshd.service: Deactivated successfully.
Jul 25 22:15:48 localhost systemd[1]: Stopped OpenSSH server daemon.
Jul 25 22:15:48 localhost systemd[1]: Stopped target sshd-keygen.target.
Jul 25 22:15:48 localhost systemd[1]: Stopping sshd-keygen.target...
Jul 25 22:15:48 localhost systemd[1]: OpenSSH ecdsa Server Key Generation was skipped because all trigger condition checks failed.
Jul 25 22:15:48 localhost systemd[1]: OpenSSH ed25519 Server Key Generation was skipped because all trigger condition checks failed.
Jul 25 22:15:48 localhost systemd[1]: OpenSSH rsa Server Key Generation was skipped because all trigger condition checks failed.
Jul 25 22:15:48 localhost systemd[1]: Reached target sshd-keygen.target.
Jul 25 22:15:48 localhost systemd[1]: Starting OpenSSH server daemon...
Jul 25 22:15:48 localhost systemd[1]: Started OpenSSH server daemon.
1 Like
2

Reading the error log for some reason when the server daemon tries to figure out what kind of key it is suppose to be using it fails all the checks so it can’t use a key to login. the config you are using I assume prohibits logging in using any other method so it doesn’t check so the program just reaches an end state and stops.

Need to figure out why the program is failing those checks, is there a further log based specifically on the keygen service? Or perhaps looking at the sshd-keygen.target file it is mentioning may lead to something worthwhile.

3 
[root@mx ~]# cat /usr/lib/systemd/system/sshd-keygen.target
[Unit]
Wants=sshd-keygen@rsa.service
Wants=sshd-keygen@ecdsa.service
Wants=sshd-keygen@ed25519.service
PartOf=sshd.service

systemctl restart sshd-keygen@ed25519.service

sshd-keygen@ed25519.service - OpenSSH ed25519 Server Key Generation
     Loaded: loaded (/usr/lib/systemd/system/sshd-keygen@.service; disabled; vendor preset: disabled)
     Active: inactive (dead) since Tue 2022-07-26 08:17:12 CST; 6min ago
  Condition: start condition failed at Tue 2022-07-26 08:24:04 CST; 7s ago
             └─ ConditionFileNotEmpty=|!/etc/ssh/ssh_host_ed25519_key was not met
    Process: 1784 ExecStart=/usr/libexec/openssh/sshd-keygen ed25519 (code=exited, status=0/SUCCESS)
   Main PID: 1784 (code=exited, status=0/SUCCESS)
        CPU: 46ms

Jul 26 08:17:12 RK3399 systemd[1]: Starting OpenSSH ed25519 Server Key Generation...
Jul 26 08:17:12 RK3399 systemd[1]: sshd-keygen@ed25519.service: Deactivated successfully.
Jul 26 08:17:12 RK3399 systemd[1]: Finished OpenSSH ed25519 Server Key Generation.
Jul 26 08:20:47 RK3399 systemd[1]: OpenSSH ed25519 Server Key Generation was skipped because all trigger condition checks failed.
Jul 26 08:22:37 RK3399 systemd[1]: OpenSSH ed25519 Server Key Generation was skipped because all trigger condition checks failed.
Jul 26 08:23:19 RK3399 systemd[1]: OpenSSH ed25519 Server Key Generation was skipped because all trigger condition checks failed.
Jul 26 08:23:38 RK3399 systemd[1]: OpenSSH ed25519 Server Key Generation was skipped because all trigger condition checks failed.
Jul 26 08:24:04 RK3399 systemd[1]: OpenSSH ed25519 Server Key Generation was skipped because all trigger condition checks failed.
4

I think this is the problem after OpenSSL upgrade;
How can I fix this problem?

[root@RK3399 ssh]# /usr/sbin/sshd -p 9710 -d
debug1: sshd version OpenSSH_8.7, OpenSSL 3.0.1 14 Dec 2021
debug1: private host key #0: ssh-rsa SHA256:l295Ah3BM282/Ng34lZr4sqtss5e1A7KSkdLecrGgmA
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:8OP7g5nvaxbbmNSZ2lrSpFoLS9q7RcD+7hefbtl45js
debug1: private host key #2: ssh-ed25519 SHA256:M56mANe3ppZv9sCacin6+9yANdYeJDWHX8erWCV3WnY
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-p'
debug1: rexec_argv[2]='9710'
debug1: rexec_argv[3]='-d'
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 9710 on 0.0.0.0.
Server listening on 0.0.0.0 port 9710.
debug1: Bind to port 9710 on ::.
Server listening on :: port 9710.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: sshd version OpenSSH_8.7, OpenSSL 3.0.1 14 Dec 2021
debug1: private host key #0: ssh-rsa SHA256:l295Ah3BM282/Ng34lZr4sqtss5e1A7KSkdLecrGgmA
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:8OP7g5nvaxbbmNSZ2lrSpFoLS9q7RcD+7hefbtl45js
debug1: private host key #2: ssh-ed25519 SHA256:M56mANe3ppZv9sCacin6+9yANdYeJDWHX8erWCV3WnY
debug1: inetd sockets after dupping: 3, 3
Connection from 1.116.42.97 port 63454 on 10.0.0.100 port 9710 rdomain ""
debug1: Local version string SSH-2.0-OpenSSH_8.7
debug1: Remote protocol version 2.0, remote software version nsssh2_6.0.0013 NetSarang Computer, Inc.
debug1: compat_banner: no match: nsssh2_6.0.0013 NetSarang Computer, Inc.
debug1: SELinux support disabled [preauth]
debug1: permanently_set_uid: 74/74 [preauth]
debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: curve25519-sha256@libssh.org [preauth]
debug1: kex: host key algorithm: ssh-rsa [preauth]
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none [preauth]
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none [preauth]
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32 [preauth]
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32 [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: SSH2_MSG_KEX_ECDH_INIT received [preauth]
mm_answer_sign: sign: error in libcrypto
debug1: do_cleanup
debug1: Killing privsep child 1898
5

After pure installation of rockylinux9, I put the public key in .ssh/authorized_keys, log in and select the key to log in, and the prompt message is “The selected key is not registered in the remote host”

6

Looking at this part of the log it says that the keygen service is not starting because it can’t find the key

" ConditionFileNotEmpty=|!/etc/ssh/ssh_host_ed25519_key was not met "

is your key located there? if not you will either want to put the key there or edit some configs to point to your key.

Also The service says it is disabled by default and is currently still disabled

" …@.service; disabled; vendor preset: disabled) "

though this is only to make sure that the service restarts on boot.
sudo systemctl enable sshd-keygen@.service
the above command will at least solve that much of the problem

7

You can try it in your system, and by default the service will fail.

8

in the middle of spinning up a system will take me awhile. but I’ll see what I can find.

9

Hello, I’m facing the same problem now after update to 9.1. I would appreciate if you could tell how you solved the problem. Thanks.

Status:
sshd-keygen@ed25519.service - OpenSSH ed25519 Server Key Generation
Loaded: loaded (/usr/lib/systemd/system/sshd-keygen@.service; disabled; vendor preset: disabled)
Active: inactive (dead)
Condition: start condition failed at Thu 2022-12-01 19:01:11 CET; 48s ago
└─ ConditionFileNotEmpty=|!/etc/ssh/ssh_host_ed25519_key was not met

10

From what I see on the thread the host keys may be a red herring.

Please describe your observable symptoms.

11

This is what I get in the log file. I’m no longer able to connect to server via ssh. It is configured to use key.

Dec 1 20:19:13 server systemd[1]: OpenSSH ecdsa Server Key Generation was skipped because all trigger condition checks failed.
Dec 1 20:19:13 server systemd[1]: OpenSSH ed25519 Server Key Generation was skipped because all trigger condition checks failed.
Dec 1 20:19:13 server systemd[1]: OpenSSH rsa Server Key Generation was skipped because all trigger condition checks failed.
Dec 1 20:19:13 server systemd[1]: Reached target sshd-keygen.target.
Dec 1 20:19:13 server systemd[1]: Starting OpenSSH server daemon…
Dec 1 20:19:13 server systemd[1]: Started OpenSSH server daemon.

12

Thank you for taking your time to help me.

13

Do you think these explain the inablility to log in?

systemd[1]: OpenSSH ecdsa Server Key Generation was skipped because all trigger condition checks failed.
systemd[1]: OpenSSH ed25519 Server Key Generation was skipped because all trigger condition checks failed.
systemd[1]: OpenSSH rsa Server Key Generation was skipped because all trigger condition checks failed.
systemd[1]: Reached target sshd-keygen.target.

No, they don’t. I get that same output from my machine and it is what I expect to get. That just tells that the machine does already have sshd host keys.

What you want to know is why communication between sshd and ssh client does fail.

Give option -v to ssh client. You should get much more verbose view on how the client experiences the connection attempt.

On the server, do look at the messages that sshd did log during the attempt. Probably the latest:

journalctl -xeu sshd

In case errors are not logged by sshd.service, look at:

journalctl -e
# or
less /var/log/secure
less /var/log/messages
14

I’m very grateful that you pointed out my mistake while trying to solve the problem. I was really thinking that the error message I mentioned above was the reason that I wasn’t being able to connect to server via ssh. Looking to the log files I couldn’t identify any other error message regarding sshd.
At last I took a look at the firewall and became aware that the ssh service port was changed back to the default 22 after the update. I wasn’t expecting that. After I change it back to my custom port the connection was established as before.
Thank you very much for your valuable help.

15

What means did you use to make sure that your port is not changed back to the default on the next update? If you just set the port under the “Run Time” status and don’t save it to “Permanent” then the settings won’t persist after a reboot.

16

Indeed. Change of stored firewall config by package update sounds outright scary. Such “feature change” would surely break many setups and should therefore be prominently documented in release notes. (Not that I had looked at them carefully.)

17

This is a feature of the Firewald GUI that is present in most graphical installs [Mate | Gnome] for RH since RH7. If you edit your firewall connections via the CLI then I think it avoids the confusing overhead of the GUI.

18

The firewalldpackage has its default config in /usr/lib/firewalld.
If one does use GUI applets or CLI tool (firewall-cmd), they both add customizations into /etc/firewalld. Content in /etc/firewalld overrides content in /usr/lib/firewalld.
If one manually adds config, one should put it into /etc/firewalld, where firewalld, GUI applets, and CLI tool should see the additions.

Package update can and does replace content in /usr/lib/firewalld. The expectation is that package update will not erase/modify content in /etc/firewalld and that syntax will not change. Hence, the customization of sshd-port should have remained unchanged and in effect.

Should the package update have modified /etc/firewalld, then all systems with custom firewall would have been affected by the update. The other option is that the customization was not properly made, and possibly still isn’t. Hence, future updates can break it again.

19

I changed the port number in the file ssh.xml which resides in the directory /usr/lib/firewalld/services/
after the change I reboot the system a couple of times to see if it persists. So far the change remains stable. My knowledge in these matters is very limited, I will pay attention next time and report if I see changes during updates.

20

Yes, that is not the Way. Copy that file into /etc/firewalld/services/