Hi
This. morning I read in the logwatch report from last night:
A total of 1 possible successful probes were detected (the following URLs
contain strings that match one or more of a listing of strings that
indicate a possible exploit):
/?file=php://filter/convert.base64-encode/resource=/etc/passwd HTTP Response 200
What can I do to lock-out such possible intruders?
I did login via direct access to machine following the very clear procedure outlined here:
How to Reset a Lost Root Password on Rocky Linux | CIQ Knowledge Base
And I changed root pasword and regained control. I have a fully updated RL 9.8 system - I updated the machine directly after regaining control.
What should I do next to secure my system and check for any installed bots or other malicious software?
Many thanks for all tips!
Was SSH access publicly available? If so, then I would suggest reinstalling your server from zero and restore your data.
Even if SSH access was blocked, there’s no guarantees that something else is amiss. A compromised server can no longer be trusted. Best thing to do is start with a new clean server.
Ensure for the future, that updates are applied to the server regularly so that this sort of thing doesn’t happen again. Also ensure things like selinux are enabled as it would have most likely blocked that kind of access to /etc/passwd.
Thanks iwalker for your quick reply!
Yes, SSH was publicly available…I get everyday 2-3000 attempts to get access. But root is not a user which can be used to login in SSH. There is only one allowed user with a strange name and an ever stranger password. But I see that the login attempts are for instance using the name of my ISP as a login account (not valid though…), so whoever is involved, he/they is/are doing some research…
I have for now closed SSH, to be on the safe side.
I guess I still need to start from scratch and intall the server from scratch, which i really don’t like just now. Are there any other ways, perhaps not as good as reinstalling, that I cold try first and do “something”?
Regarding SELinux, I checked the SELinus status and this is:
SELINUX=enforcing
…
SELINUXTYPE=targeted
Still the /etc/passwd was accessed during the break-in …
Does the info from Logwatch above reveals anything about what was actually done with /etc/passwd?
When I check it, it “seems” to contain the users/processes that used to be in that file. Was it just someone who wants to show he/she/it is skilled to do this?
I would also look at doing some best practice configurations with sshd. I would definitely use a port other than 22. It is harder to find an open port if you have 10000 or so to look through. Most free scanning tools typically go to the first 1000 ports or so. Script kitties know to look at port 22. If possible I would also setup MFA on the user logging in through ssh. Google authenticator works well with RL. Only 1 user, in my case is allowed to ssh on my system. Lastly, if able, I would close my SSH port through my router unless needed. I can control mine remotely and it is open only when needed.
Thanks @randallb !
I do use another port than the standard port 22 (a much higher arbitrary number). But still someone managed to find it and misuse it.
I now closed the ssh port on the router, so nobody can get to the server with ssh.
I also do not use a password. Use public key authentication which is more secure. And yes a lot of this is security through obscurity.
Good luck with the system.
Thanks @randallb !
I’ll look into ssh and MFA. That should indeed be more secure.
Look carefully at the “php wrapper” filter on this page
https://github.com/swisskyrepo/PayloadsAllTheThings/blob/master/File%20Inclusion/Wrappers.md
if you had anything secret buried in a php file, they would be able to see it. In addition they can look at other well known files, like the list of user accounts.
Instead of using logwatch, it would be better to analyse the logs from that day (and the day before). My guess is that you’d see some strange urls, maybe starting with getting ‘phpinfo()’ and ‘phpmyadmin’ and so on. If you have a web server runnning you need to check both system logs and web server logs, and I would also save the attack as a big text file for future reference.
Although it says “php” in the log, it might be some other process and it would be good to know how/why selinux didn’t stop it. You could do a quick check anyway, write a two line php script to get the contents of the file with the user accounts, and see if you get an sealert.
Thanks @gerry666uk !
I will check the logs this weekend and look for SE -alerts.
Unfortunately I don’t know how to use php…
Would you mind outlining those “two php lines” to be used on the log files?
Checking at the “week-end” is ok if you have good log retention, but you should really check (and save) the same day, because logs can rotate away, and it can be hard scrolling backwards over days and days of logs trying to find where it went wrong. The first thing is to relate the logwatch entry with the real entry in the real log, which could be a system log or a web server log. If you’re not running a web server, I don’t know how you’d be seeing a url starting with ‘php’, unless it’s using php from the command line. You could at least check if you have php installed.
I hope this comment helps to provide a bit of additional context.
The quoted log entry, presumably from Apache, returning status “200” does NOT NECESSARILY mean that the password file was sent via http. The HTTP GET command (and the full Apache log entry) probably looks something like this:
GET index.php?file=php://filter/convert.base64-encode/resource=/etc/passwd
and what the status 200 is indicating is that the index web page (in this example) was returned (as normal). If you still can extract the full log entry, i recommend you do so, to better understand, and presumably you would test, and confirm what is actually returned, using the same URL to the same website.
In other words, my sense from this thread is that this Apache log entry may be coincidental, and that the real issue the OP faced is SSH publicly open, and if you say there are an average of 2500 attempts per day, that is approaching one million SSH attempts on an annual basis. I don’t see any mention of firewall rules, but if it is possible to identify only allowed SSH traffic (e.g. based on source IP address) that would be a great improvement.
Disabling ssh password authentication and allowing keys only would go a long way to prevent unauthorized logins since there’s really nothing left for the attacker to brute-force.
PasswordAuthentication no
You can also use the AllowUsers directive to allow only certain usernames.
AllowUsers larry moe curly
It’s also possible to allow password authentication only from certain IP addresses and require keys from any other address.
The directive to do that can look like this:
Match Address 192.168.0.*
PasswordAuthentication yes
This allows a user to log in with a password from the local network (assuming it’s 192.168.0.* and assuming that you have set PasswordAuthentication no as shown above) but requires a key from any other IP address.
Thanks @R_O_C_K_Y_L_I_N_U_X !
Yes, I tend to agree with you about “sending the /etc/passwd file”.
I tested the same command
from another computer on my LAN through my webserver https://mysite and only my default home page was shown. I could not see anything else was returned or downloaded.
I also checked logs from what the server did when I did this and the SELinux log says (using https://mysite:9090):
"SELinux is preventing /usr/libexec/power-profiles-daemon from getattr access on the file /etc/passwd. "
(16 times incl my own test yesterday - no info on specific dates though )
“SELinux is preventing /usr/sbin/unix_chkpwd from read access on the file /etc/shadow”
(26 times incl my own test yesterday - no info on specific dates though )
So it looks like SELinux prevented the reading of the /etc/passwd file.
The intrusion probably happened through ssh (which is closed for now).
Thanks @FrankCox !
Very helpful tips!
When setting up ssh more securely, I will certainly look into this!
Until then I have closed ssh completely.
Need to look into your solutions first, before re-starting ssh again.
Thanks @gerry666uk !
Yes, I will and am looking through my logs now. It looks like logs are indeed rotated, but saved in separate files:
-rw-------. 1 root root 2793968 Jul 25 13:13 secure
-rw-------. 1 root root 8067619 Jun 27 23:59 secure-20260628
-rw-------. 1 root root 7826589 Jul 4 23:59 secure-20260705
-rw-------. 1 root root 10965227 Jul 11 23:59 secure-20260712
-rw-------. 1 root root 6284983 Jul 18 23:59 secure-20260719
So I am looking through the files from 22-23 july as the logwatch showed the “intrusion” at 23 july.
I updated the system at July 19th. And assuming the real intrusion was not through Apache (see comment from @R_O_C_K_Y_L_I_N_U_X , then I may have to look from July 19th I guess?
I tried to find events related to “passwd” as the intrusion lead to a change i the root password (or are there other ways to change/delete root password??) , but I cannot find anything when I use:
$ more /var/log/secure | grep passwd
As recommended by @iwalker I tried to find a corresponding entry in the httpd logs, using
$ more /var/log/httpd/access_log | grep passwd
The result is:
192.161.49.2 - - [23/Jul/2026:16:50:40 +0200] “GET /..%2F..%2F..%2F..%2F..%2F..%2Fetc%2Fpasswd HTTP/1.1” 404 196 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:146.0) Gecko/20100101 Firefox/146.0”
This IP adres in Alabama/USA has a "Abuse Confidence Score: 100%"
But as said, this is probably not the “real” intruder as I expect it to got access via ssh…
Anything else I should look for? Or should I look in another system log?
BTW I do have php 8.4 installed and I am running a website too.
Many thanks!
Look in /var/log/secure and the rotated files for connections via SSH. You can grep it for sshd and you should see attempts to login be it successful or failed. You can then find out which user they successfully connected as that allowed them to reset the root password.
Assuming that the root password was changed, then they must have gotten into the server. If the previous root password was a simple one then they could have taken advantage of that. Other possibilities if the server wasn’t regularly updated and they abused a vulnerability.
I realize that this returned status 404, but I wanted to point out that, for the purpose of enhancing the security of your web server, in Apache’s httpd.conf you can block all “automated directory transversal attacks” which are attempting to read “sensitive system files” via a case-insensitive RewriteRule along these lines, and focusing on the fact that attacker(s) are utilizing “%2F” to hide the fact that they are passing a “slash” within the URL:
<IfModule mod_rewrite.c>
RewriteEngine On
# Catch any literal %2F (case-insensitive) in the raw HTTP request line
RewriteCond %{THE_REQUEST} %2[fF] [NC]
RewriteRule ^ - [F,L]
</IfModule>
instead of status 404, this would return status “403 Forbidden”.
While this is just one example, I want to suggest this as a useful tool in your toolkit, as you see in your logs other automated attacks.
Thanks @iwalker !
I ran the commands below and apart from my own tests using the only approved ssh user “xxxxx”, done the 23rd (after the discovery) and today the 25th (via LAN - router ssh port closed), I get no match:
[root@server log]# more secure-20260719 | grep -e “sshd” | grep -e “session opened”
[root@server log]# more secure-20260712 | grep -e “sshd” | grep -e “session opened”
[root@server log]# more secure-20260705 | grep -e “sshd” | grep -e “session opened”
[root@server log]# more secure-20260628 | grep -e “sshd” | grep -e “session opened”
[root@server log]# more secure | grep -e “sshd” | grep -e “session opened”
Jul 23 22:35:41 server sshd-session[64296]: pam_unix(sshd:session): session opened for user xxxxx(uid=1003) by xxxxx(uid=0)
Jul 23 22:36:58 server sshd-session[64473]: pam_unix(sshd:session): session opened for user xxxxx(uid=1003) by xxxxx(uid=0)
Jul 23 22:46:24 server sshd-session[65419]: pam_unix(sshd:session): session opened for user xxxxx(uid=1003) by xxxxx(uid=0)
Jul 25 15:03:09 server sshd-session[155049]: pam_unix(sshd:session): session opened for user xxxxx(uid=1003) by xxxxx(uid=0)
[root@server log]#
Of course, the intrusion could have happened before June 21st but I do login into the server regularly and use the su command to check/update the system. I did not have any login issues (>wrong password) before July 23rd when I checked the server after I noticed the warning in the logwatch mail.
This would mean that the intruder must have used a vulnerability.
Then this is a bigger problem… Is it still there or have recent updates removed that vulnerability? Are there any notes on what changes have been made the last days to solve such a vulnerability? I noticed 2 kernel updates lately.
Thanks @R_O_C_K_Y_L_I_N_U_X !
Yes, I have added this to my httpd.conf file!
Because grep thru some individual files gives you nothing, can i suggest it is easier if you use filename wildcard in your grep command, additionally you can search for specific known usernames (using “root” below as example), so you might try commands like these for more insight into your log analysis:
grep "session" /var/log/secure* | grep root | wc -l
grep "session" /var/log/secure* | grep root | more
Basically, for analysis purposes, i am suggesting that you inspect your /etc/passwd file, where the last (7th) parameter in the /etc/passwd file is the user’s default login shell or executable program run when the account logs in. As such, you can identify (build a list of) specific users which might have connected to your system, especially knowing that your list of users might have been modified by an attacker.
Hope these general ideas might help you,
Tony