Thanks @R_O_C_K_Y_L_I_N_U_X !
I used several ways to look for the intrusion(s). So far I have not been able to find any…
I have only 4 users on this server (incl root), so it’s relatively easy to check each of them using the secure logs.
I cannot find any ssh connection for that time period I have logs, which was successful, apart from my own checks. When correct, this seems to indicate that ssh was perhaps not the used way in?
I will look at the messages log too.
Did you confirm by inspecting /etc/passwd that there are only 4 users?
My concern is that an attacker could have added user(s) which you might not be aware of.
Tony
Very good point! I checked the /etc/passwd file again and selected those lines having a login shell which includes /bin/. This is what I found:
[root@server ssh]# more /etc/passwd | grep /bin/
root:x:0:0:root:/root:/bin/bash
sync:x:5:0:sync:/sbin:/bin/sync
clamav:x:1001:1001:Clam Antivirus:/home/clamav:/bin/false
mockbuild:x:1002:1002::/home/mockbuild:/bin/bash
[root@server ssh]#
I did remove my known users from the list.
I expect “sync” to be OK? SInce this has no normal shell.
What is definitely not created by me is user “mockbuild” !!
[root@server ssh]# cd /home/mockbuild/
[root@server mockbuild]# ll
total 0
drwx------. 2 mockbuild mockbuild 74 Jul 31 2022 Desktop
[root@server mockbuild]# cd Desktop/
[root@server Desktop]# ll
total 8
-rwx------. 1 mockbuild mockbuild 253 Jul 31 2022 turboprint-control.desktop
-rwx------. 1 mockbuild mockbuild 255 Jul 31 2022 turboprint-monitor.desktop
[root@server Desktop]#
These are created in 2022, which sounds strange and seems to be related to a linux printer software I have used many years (turboprint). They are added to each user of the system.
But there should definitely not be a user “mockbuild” which can login to the system.
My plan is to remove this user incl home directory. I have already removed the turboprint desktop links.
Is there something else I should check before removing the user and its home directory?
I cannot find any process related to mockbuild:
[root@server Desktop]# ps -ef | grep mockbuild
root 215302 166302 0 23:25 pts/0 00:00:00 grep --color=auto mockbuild
[root@server Desktop]#
I plan to remove the user by using:
$ userdel -Z -r mockbuild
Does this sound good enough?
The server has no ssh connection for now, so mockbuild should not be able to login now I suppose?
I have also put " AllowUsers" plus the username that that is allowed to use ssh in the ssh config file.
Your suggestions are highly appreciated!
My first thought is still that you should, per my previous message, run for each of your users:
grep "session" /var/log/secure* | grep mockbuild | wc -l
grep "session" /var/log/secure* | grep mockbuild | more
(this time using “mockbuild” as the user being checked).
But now that you have identified this user mockbuild (and quoting from Google Gemini): “mock is the standard tool in RHEL-based distributions for building RPMs inside isolated chroot environments. When building or compiling software using mock, the tool automatically executes build steps under the internal mockbuild user account to prevent root access during compilation.”
Gemini suggests checking the status of mockbuild via the following commands:
To determine if the mockbuild user has been weaponized as a backdoor, run these checks:
Check account lock status:
Bash
sudo passwd -S mockbuild
If it shows Password locked (or L), direct password login is disabled.
Check /etc/shadow entry:
Bash
sudo grep mockbuild /etc/shadow
If the password field (second field) contains ! or *, no password can authenticate against it.
Check for SSH authorization:
Bash
sudo ls -la /var/lib/mockbuild/.ssh/ /home/mockbuild/.ssh/ 2>/dev/null
Look for an authorized_keys file.
Check for elevated permissions:
Bash
sudo grep -rn "mockbuild" /etc/sudoers /etc/sudoers.d/
Takeaway: If
/etc/shadowshows a locked password (!), there are no SSH keys in its home directory, and no sudo privileges assigned, the account cannot be logged into directly from the outside, even with/bin/bashset as its shell.
Thanks again!
This is what got:
[root@server Desktop]# passwd -S mockbuild
mockbuild LK 2022-07-18 0 99999 7 -1 (Password locked.)
[root@server Desktop]# grep mockbuild /etc/shadow
mockbuild:!!:19191:0:99999:7:::
[root@server Desktop]# ls -la /var/lib/mockbuild/.ssh/ /home/mockbuild/.ssh/ 2>/dev/null
[root@server Desktop]# grep -rn “mockbuild” /etc/sudoers /etc/sudoers.d/
[root@server Desktop]#
I guess this is indeed a leftover of an earlier test I did (2022) with a chroot environment. Which I never finished/used. This seems to be a false alarm . Good news.
Regarding the first test with mockbuild:
grep “session” /var/log/secure* | grep mockbuild | wc -l
grep “session” /var/log/secure* | grep mockbuild | more
This gave 0 as result. Good
The same is done with my known users and that gives legitimitet logins only for those users, plus one login with a name which “includes the name of one of my real users”:
/var/log/secure-20260712:Jul 7 11:57:03 server sshd-session[1250110]: Invalid user tesovallado from 185.116.107.41 port 40516
/var/log/secure-20260712:Jul 7 11:57:04 server sshd-session[1250110]: Failed password for invalid user tesovallado from 185.116.107.41 port 40516 ssh2
/var/log/secure-20260712:Jul 7 11:57:05 server sshd-session[1250110]: Connection closed by invalid user tesovallado 185.116.107.41 port 40516 [preauth]
An this was blocked by the system.
So nothing here either, whch is good news I guess.
As this user mockbuild cannot login, should I just leave it and not remove it?
It was worth making a deeper forensic analysis of your server, but at this point, the best advice that you received was from @iwalker who wrote earlier:
" A compromised server can no longer be trusted. Best thing to do is start with a new clean server."
I agree with Ian’s advice: backup and restore.
He had other good advice, including:
"Ensure for the future, that updates are applied to the server regularly so that this sort of thing doesn’t happen again. "
Tony
Thanks again Tony and all other experts who helped me!
This is a great forum with lots of expertise and people who want to help!
I will build a server from scratch using RL 10.
I already update my server at least once every month. However that proved to be insufficient (or there is a hidden bug in the OS which someone exploited…).
/Ralf
Den 26 juli 2026 16:37:25 CEST, Tony Patti notifications@rockylinux.discoursemail.com skrev:
Most likely someone successfully brute-forced the password for one of your users if you are still allowing password authentication from anywhere.
That is garbage advice. Ports other than 22 for SSH are found within less than a day.
If you don’t have time to keep on top of things yourself, aka you are not a full-time professional admin, then I would STRONGLY recommend the use of dnf-automatic with full automatic patching and an out of hours reboot where required.
bear in mind that SSH keys for authentication can be a worse security nightmare than password based authentication. The problem is that you have zero control over whether your users are protecting the SSH key with a passphrase, and if your users are apt to pick passwords with low entropy then they are apt to leave their SSH key unsecured.
The best two options are first to have a mechanism to enforce secure passwords, and secondly implement MFA. SSH keys should only be used for authentication by trusted users, basically for admin purposes. If you need passwordless SSH between machines for ordinary users then Kerberos has been the answer for well over three decades now.
Thanks for the advice.
I have considered this already many years ago, but experienced a perhaps “once in a lifetime” bad kernel update where I had to go back to the previous kernel to get the server working again. It was good I was sitting next to the server and could act immediately.
That made me reluctant to use auto-updates. But perhaps I should reconsider that auto-updating again.
Just out of curiosity, how many “full time professionals” use this auto-updating?
I don’t use auto update. I prefer to log into my servers regularly and check for updates manually purely because I don’t want my servers automatically updating and potentially breaking something making it more difficult to find out what caused it.
If you have a lot of servers, then you can set up your own Foreman server with Katello and once synced with public repositories you can see here when your servers have updates available and then apply the updates. Easier to manage and maintain and easy visibility without checking all servers individually/manually.
Historically, I would agree with you. However, we now live in an entirely different world when it comes to cybersecurity, which should have you re-evaluating the risks. What is more likely: a bad update or a hack, and what are the consequences and remediation in both scenarios?
I would note you can configure dnf-automatic to only download security updates if they exist and send an email that it has done so. You can also have it automatically apply them and optionally reboot if required. You can basically choose your risk profile. You will probably get very tired very quickly of the near-daily updates you need to apply, however.
To be honest, the chances of a bad update are, in my decades of experience, extremely low and the consequences way lower than those of a hack. I run an HPC system with hundreds of nodes, and we are moving, where possible, to fully automate patching for security vulnerabilities. The risks of being hacked vastly outweigh those of a bad update.
It’s not that clear cut. For example, in banking/finance they will not apply updates immediately to production systems without testing them first on staging environments going from dev → test → prod. Applying an update directly to production could break their banking/finance applications and cause huge outages for clients. Irrespective of whether updates in most cases don’t cause problems, they will still err on the side of caution.
You are free to choose what method suits you best. Unfortunately, automatic updates doesn’t apply to all situations and whilst you may disagree that is fine. There are others that will also disagree with your viewpoint based on certain scenarios.
Due to the changes in security etc, there are different/additional products can be used in front of such systems to ensure even if they aren’t up-to-date, that they aren’t taken advantage of. Be it a WAF, Crowdstrike or other products on the market. Obviously if you don’t have the money for the additional protection, then you have to decide on what you will do.
I’ve been doing updates on my Linux servers for over 20 years, manually and I don’t intend to change it. I’m not bored of it either 
Now get off my lawn, youngling 
. Linux user since 1992 here.
The threat environment has fundamentally changed in the last three months. I didn’t say automatic updates were applicable in all scenarios. However, unless you have a dedicated team that can apply updates and patches on a DAILY basis, you should be seriously considering it.
For example, let’s take a bank and assume you use Oracle as your DB; yesterday, CVE-2026-61211 was disclosed. You simply don’t have even a day to do dev → test → prod without risking your entire banking IT system being hacked and taken over.
Or you are running an e-commerce site on WordPress and CVE-2026-63030 drops, you are going to have to patch WordPress right there and then or risk a catastrophic outcome.
Anyone who thinks that the old methods are still valid is, and I don’t say this lightly, a fool. Everyone needs to fundamentally readdress all assumptions around the cybersecurity threat and come up with new ways of working that get systems patched much faster than previously.
Professionally, we are likely going to drop Rocky Linux and switch back to AlmaLinux. The patch release delay for Rocky is simply too great; the pricing for RL Pro from CIQ is ridiculous (it’s more expensive than genuine RHEL from Red Hat!!!). So we are looking at AlmaLinux with KernelCare and Extended update support. Deciding if we can afford LibCare too. That’s a five-figure sum we need to find out of next year’s budget (which starts on Saturday) that was not on the plan, but the current situation is unsustainable. Being an HPC system, we are horribly exposed to LPE’s.
I used to do manual updates because of the bad kernel issue. The last one I had was in Solaris 9 but it was severe. I since switched to an automatic (dnf-automatic) approach.
I last used an offline Satellite server approach to distributing my updates. All of my systems updated from an offline server, workstations every night, most servers on the weekend. At the beginning of the week, I would pull in the latest updates. I would immediately update my system to verify what was being updated and then would manually update a couple more to verify no immediate issues. The workstations would then update automatically and I would have a few days to verify no issues before the servers went. Our cadence was 1 month at the time. I think for the month of July, Red Hat released 10 updates to the kernel, 1 every 3 days or so with 1 day having 2 updates. I would expect a weekly update cadence now. To paraphrase men in black - There are always vulnerabilities on your system. You assume they have not been found by anyone. Updates are those that are found by the vendor, been turned in by the good guys or are KEVs. Updating is only part of the solution. I do not panic if a particular update is not immediately needed - fire drill level - as they can wait until the next update run.
Changing ports is hardly garbage. It is absolutely considered a best practice. Beyond using port 22, you MUST scan for ports on that network or system. This makes it, even in automation, more work. As I stated, this is security through obscurity but it does reduce the attack plane. It also makes detection easier for ISP/Security Operators as port scanning is much easier for an IDS to detect then monitoring traffic on just port 22.
As for Public Key Authentication - It is FAR superior to passwords for the exact weaknesses you described. A user can easily share a password, write it down, give it away, expose it on clear text communication etc. Not having a passphrase on a key only weakens restricting who uses it and if it gets out of the user’s control. Passwords are no better and even far worse then keys for any of the justifications presented to use a password.
This is why we are going towards passkeys instead of passwords all over the Internet. They are that much more secure!
I generally tend to change the SSH port from time to time, not always though. Lately I’ve preferred to just use firewalld and only allow access to SSH from specific IP addresses. That will also reduce the risk somewhat since if it’s a VPS at a provider, then the attacker would need to be in our office 
and if they are in our office, they’d also need an SSH public key to get on the system as well as password auth is disabled
Like I said in actual practice they will find it in less than a day on average.
You are dead wrong about keys as well. If a user is going to do all those bad things, you think they are not going to share their key? When you have been on the end of a major breach precisely because users didn’t secure their keys you might change your view. In the meantime, you are a badly informed fool of a Took. The best method to secure against those sorts of issues is MFA not SSH keys, and even MFA is far from a silver bullet.