Issues with SSH login using VEEAM 11

curto · December 2, 2023, 3:59am

Guys, not sure what is going on here and i could use some help

2 x remote sites running VEEAM 11 backing up to a local Linux (Open Media Vault) server.

This is running fine and has been so for approx 12 months - both sites run a number of different backup and replication jobs and these are fine.

I have now convinced the clients to implement offsite backups (previously they would take home external USB drives that had backups stored on them)

I host a large Rocky 9.2 server with approx 100TB of disk storage on it.

I have establish point to point VPNs between the clients site and my firewall using wireguard - this is working fine and connections can be made between the various machines.

I am setting up a VEEAM Backup copy job set to run after every backup on each of the clients sites.

I have installed Putty on the clients Windows Backups servers and from there i can SSH to the machine hosted on my site - thus proving firewalls etc are all good.

When i setup the backup copy jobs they run fine for a period of time (usually about a week or so) and then they all start failing - this is happening from both clients sites so i am assuming the problem is on my Rocky host rather than at their end.

From another Linux box on each clients site i can SSH backwards and forwards to my Rocky box and have scripts that run that perform a weekly rsync using passwordless login (with SSH-Keygen) and these continue to run fine.

However the daily backup copy jobs fail - when i look at the logs on the server - i see something like this

Received disconnect from 192.168.0.42 port 61149:11: Session closed [preauth]
Dec 2 14:12:29 mediastore8 sshd[3456598]: Unable to negotiate with 192.168.0.42 port 61148: no matching host key type found. Their offer: ssh-rsa,ssh-dss [preauth]
Dec 2 14:12:29 mediastore8 sshd[3456596]: Unable to negotiate with 192.168.0.42 port 61147: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 [preauth]

and this is repeated

If i go into the Veeam console and tell it to reinstall the agents on the Rocky box - the jobs will start running again and do so for another week or so before they start failing.

My rocky server is on a UPS and i have verified through the logs that there have been no unexpected reboots etc.

I have the firewall disabled on Rocky

Any ideas where i can go to try and narrow this one down ?

Craig

FrankCox · December 2, 2023, 4:35am

I have seen something like this before when a router was failing. It works for a while then stops passing some traffic (but not everything) then works again after a while or after it gets rebooted.

The ultimate solution was to take it out and shoot it and put a new one into place. No issues after that.

curto · December 2, 2023, 4:39am

Thanks for the quick response.

I can swap the router/firewall for a different physical unit - with relative ease - however based on the fact that the Weekend replication (which is approximately 2TB of data) goes through with no issue - even whilst the smaller backup copy jobs are failing makes me feel this is not the issue.

I will swap the unit tomorrow as you suggest and see if it changes anything though as it is not a big job.

Craig

curto · December 2, 2023, 4:54am

Below is the code of a running session (actually two clients at the same time) - this works fine after redoing the client setup in Veeam without any reboots or starts of servers etc. This is what i have had to do every two weeks or so whilst trying to troubleshoot this issue. It will then run fine for the next week or so

Dec  2 15:14:39 mediastore8 sshd[3492622]: pam_unix(sshd:session): session closed for user root
Dec  2 15:14:39 mediastore8 sshd[3492624]: Disconnected from user root 192.168.0.42 port 53474
Dec  2 15:14:39 mediastore8 sshd[3492624]: Received disconnect from 192.168.0.42 port 53474:11: Connection terminated by the client.
Dec  2 15:13:28 mediastore8 sshd[3494848]: pam_unix(sshd:session): session closed for user root
Dec  2 15:13:28 mediastore8 sshd[3494850]: Disconnected from user root 172.16.49.6 port 58766
Dec  2 15:13:28 mediastore8 sshd[3494850]: Received disconnect from 172.16.49.6 port 58766:11: Connection terminated by the client.
Dec  2 15:13:28 mediastore8 sshd[3494850]: Close session: user root from 172.16.49.6 port 58766 id 0
Dec  2 15:13:28 mediastore8 sshd[3494604]: pam_unix(sshd:session): session closed for user root
Dec  2 15:13:28 mediastore8 sshd[3494606]: Disconnected from user root 172.16.49.6 port 58762
Dec  2 15:13:28 mediastore8 sshd[3494606]: Received disconnect from 172.16.49.6 port 58762:11: Connection terminated by the client.
Dec  2 15:13:22 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 1
Dec  2 15:11:15 mediastore8 sshd[3490976]: pam_unix(sshd:session): session closed for user root
Dec  2 15:11:15 mediastore8 sshd[3490978]: Disconnected from user root 172.16.49.6 port 57689
Dec  2 15:11:15 mediastore8 sshd[3490978]: Received disconnect from 172.16.49.6 port 57689:11: Connection terminated by the client.
Dec  2 15:10:47 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494850]: Starting session: subsystem 'sftp' for root from 172.16.49.6 port 58766 id 0
Dec  2 15:10:47 mediastore8 sshd[3494848]: User child is on pid 3494850
Dec  2 15:10:47 mediastore8 sshd[3494848]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Dec  2 15:10:47 mediastore8 sshd[3494848]: Accepted password for root from 172.16.49.6 port 58766 ssh2
Dec  2 15:10:47 mediastore8 sshd[3494848]: Connection from 172.16.49.6 port 58766 on 172.16.100.26 port 22 rdomain ""
Dec  2 15:10:47 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:47 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:46 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:46 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:46 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:46 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:46 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 1
Dec  2 15:10:46 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Close session: user root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494606]: Starting session: command for root from 172.16.49.6 port 58762 id 0
Dec  2 15:10:46 mediastore8 sshd[3494604]: User child is on pid 3494606
Dec  2 15:10:46 mediastore8 sshd[3494604]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Dec  2 15:10:46 mediastore8 sshd[3494604]: Accepted password for root from 172.16.49.6 port 58762 ssh2
Dec  2 15:10:46 mediastore8 sshd[3494604]: Connection from 172.16.49.6 port 58762 on 172.16.100.26 port 22 rdomain ""
Dec  2 15:10:46 mediastore8 sshd[3494602]: Unable to negotiate with 172.16.49.6 port 58761: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 [preauth]
Dec  2 15:10:46 mediastore8 sshd[3494602]: Connection from 172.16.49.6 port 58761 on 172.16.100.26 port 22 rdomain ""
Dec  2 15:09:19 mediastore8 sshd[3493168]: pam_unix(sshd:session): session closed for user root
Dec  2 15:09:19 mediastore8 sshd[3493170]: Disconnected from user root 192.168.0.42 port 53518
Dec  2 15:09:19 mediastore8 sshd[3493170]: Received disconnect from 192.168.0.42 port 53518:11: Connection terminated by the client.
Dec  2 15:09:19 mediastore8 sshd[3493170]: Close session: user root from 192.168.0.42 port 53518 id 0
Dec  2 15:09:19 mediastore8 sshd[3492937]: pam_unix(sshd:session): session closed for user root
Dec  2 15:09:19 mediastore8 sshd[3492939]: Disconnected from user root 192.168.0.42 port 53514
Dec  2 15:09:19 mediastore8 sshd[3492939]: Received disconnect from 192.168.0.42 port 53514:11: Connection terminated by the client.
Dec  2 15:09:06 mediastore8 sshd[3492939]: Close session: user root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:06 mediastore8 sshd[3492939]: Starting session: command for root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:06 mediastore8 sshd[3492939]: Close session: user root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:06 mediastore8 sshd[3492939]: Starting session: command for root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:06 mediastore8 sshd[3492939]: Close session: user root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:06 mediastore8 sshd[3492939]: Starting session: command for root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:06 mediastore8 sshd[3492939]: Close session: user root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:06 mediastore8 sshd[3492939]: Starting session: command for root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:06 mediastore8 sshd[3492939]: Close session: user root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:06 mediastore8 sshd[3492939]: Starting session: command for root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:06 mediastore8 sshd[3492939]: Close session: user root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:05 mediastore8 sshd[3492939]: Starting session: command for root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:05 mediastore8 sshd[3492939]: Close session: user root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:05 mediastore8 sshd[3492939]: Starting session: command for root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:05 mediastore8 sshd[3492939]: Close session: user root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:05 mediastore8 sshd[3492939]: Starting session: command for root from 192.168.0.42 port 53514 id 0
Dec  2 15:09:05 mediastore8 sshd[3493170]: Starting session: subsystem 'sftp' for root from 192.168.0.42 port 53518 id 0

Craig

curto · December 5, 2023, 10:21pm

Just as a further update - once the job has been redefined (actually the remote server in the job) the backup copy jobs are running fine again. Based on past iterations i can expect them to continue to operate for approx 2 weeks until they suffer the same errors.

Will continue updating this thread

regards

Craig

system · February 3, 2024, 10:21pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.