Hello!
I’m not sure if I should post this here or at RHEL, but I just did a brand new install of Rocky Linux 8. I updated everything, installed EPEL and fail2ban, and now the box is pinging China. China’s on my list of geo-blocked locations due to hacking activity.
The only reason I can think of for my box to be pinging China is as a prelude for attempting to contact a command-and-control server. Can somebody please enlighten me as to other reasons?
Thank you
But it sounds like you already installed unsupported software on top of the official Rocky 8.x, e.g. EPEL so it’s hard to say if this is Rocky default. Is there a way to see the actuvity using command line only?
It is fail2ban. When it blocks an address it sends an icmp unreachable back to the source. A packet capture would confirm.
That’s a valid hypothesis, but it proved false. I hadn’t opened the box up to ssh from the Internet yet. Here are the results from fail2ban-client and the log:
[ ~]$ sudo fail2ban-client status sshd
Status for the jail: sshd
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
[ ~]$ sudo cat /var/log/fail2ban.log
2022-05-24 10:11:44,349 fail2ban.server [7860]: INFO --------------------------------------------------
2022-05-24 10:11:44,349 fail2ban.server [7860]: INFO Starting Fail2ban v0.11.2
2022-05-24 10:11:44,350 fail2ban.observer [7860]: INFO Observer start...
2022-05-24 10:11:44,356 fail2ban.database [7860]: INFO Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2022-05-24 10:11:44,358 fail2ban.database [7860]: WARNING New database created. Version '4'
2022-05-24 10:13:38,979 fail2ban.server [7860]: INFO Shutdown in progress...
2022-05-24 10:13:38,979 fail2ban.observer [7860]: INFO Observer stop ... try to end queue 5 seconds
2022-05-24 10:13:39,021 fail2ban.observer [7860]: INFO Observer stopped, 0 events remaining.
2022-05-24 10:13:39,060 fail2ban.server [7860]: INFO Stopping all jails
2022-05-24 10:13:39,061 fail2ban.database [7860]: INFO Connection to database closed.
2022-05-24 10:13:39,061 fail2ban.server [7860]: INFO Exiting Fail2ban
2022-05-24 10:13:39,187 fail2ban.server [7925]: INFO --------------------------------------------------
2022-05-24 10:13:39,187 fail2ban.server [7925]: INFO Starting Fail2ban v0.11.2
2022-05-24 10:13:39,187 fail2ban.observer [7925]: INFO Observer start...
2022-05-24 10:13:39,190 fail2ban.database [7925]: INFO Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2022-05-24 10:14:42,411 fail2ban.server [7925]: INFO Shutdown in progress...
2022-05-24 10:14:42,412 fail2ban.observer [7925]: INFO Observer stop ... try to end queue 5 seconds
2022-05-24 10:14:42,481 fail2ban.observer [7925]: INFO Observer stopped, 0 events remaining.
2022-05-24 10:14:42,513 fail2ban.server [7925]: INFO Stopping all jails
2022-05-24 10:14:42,514 fail2ban.database [7925]: INFO Connection to database closed.
2022-05-24 10:14:42,514 fail2ban.server [7925]: INFO Exiting Fail2ban
2022-05-24 10:14:42,641 fail2ban.server [7974]: INFO --------------------------------------------------
2022-05-24 10:14:42,642 fail2ban.server [7974]: INFO Starting Fail2ban v0.11.2
2022-05-24 10:14:42,642 fail2ban.observer [7974]: INFO Observer start...
2022-05-24 10:14:42,645 fail2ban.database [7974]: INFO Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2022-05-24 10:14:42,646 fail2ban.jail [7974]: INFO Creating new jail 'sshd'
2022-05-24 10:14:42,660 fail2ban.jail [7974]: INFO Jail 'sshd' uses systemd {}
2022-05-24 10:14:42,660 fail2ban.jail [7974]: INFO Initiated 'systemd' backend
2022-05-24 10:14:42,661 fail2ban.filter [7974]: INFO maxLines: 1
2022-05-24 10:14:42,678 fail2ban.filtersystemd [7974]: INFO [sshd] Added journal match for: '_SYSTEMD_UNIT=sshd.service + _COMM=sshd'
2022-05-24 10:14:42,678 fail2ban.filter [7974]: INFO maxRetry: 5
2022-05-24 10:14:42,678 fail2ban.filter [7974]: INFO findtime: 600
2022-05-24 10:14:42,678 fail2ban.actions [7974]: INFO banTime: 600
2022-05-24 10:14:42,678 fail2ban.filter [7974]: INFO encoding: UTF-8
2022-05-24 10:14:42,679 fail2ban.jail [7974]: INFO Jail 'sshd' started
I installed EPEL only so I could install fail2ban. Nothing else from EPEL was installed from that repo.
The screenshot was from my external firewall, I don’t know how to see what process is originating the ping on the box itself. Do you? I’d be happy to run that command.
Could have been dnf contacting mirrors for updates. Technically you could find out by doing:
watch -n1 'netstat -tanp'
when a connection opens and you see an IP that looks similar or the same of what you have reported above, the p parameter passed to netstat will show the process ID. You can then use ps command to find out what that process is if it doesn’t already say next to it what it is.
Assuming you downloaded Rocky from a legitimate and trusted source, then it’s a bit extreme to suggest that it’s some kind of hacking/command and control thing. Bit of paranoia there I think. I expect if you aren’t in the Asia Pacific region, then it would rule out the chance of you connecting to a Chinese mirror anyway assuming that the mirrors are selected via geographic location.
That said the process list should help, as well as listing the packages installed on your system as well - but then if something was installed outside of yum/dnf/rpm then it wouldn’t appear there. A fully ps listing would show all processes running on the system as well.
Thank you. I’m familiar with netstat, but IIRC, that’s only useful for showing listening or established connections. I was operating on the assumption that some piece of software was waiting to be able to ping a specific IP before initiating an outbound connection. I wasn’t able to find anything showing how to see outbound pings in netstat, and when I tested it by using your command in one putty session and then opening a new one to do a ping, I didn’t see the ping process.
I did download Rocky from a trusted source and then I verified the hash. I did a standard gui server install and selected the “Criminal Justice Information System” security policy during install. The only two things I did after that were the EPEL install and fail2ban (which I wish was available on the standard repo so I wouldn’t have to install a new repo. I don’t like doing source-code compile because it complicates keeping packages up to date.)
Perhaps I am being paranoid, but there are plenty of stories going around right now in all of the security blogs of supply-chain attacks (solar winds, Log4J) and github attacks, I just want to ensure that I’m not bringing a trojan into my network.
Nope, I’m in the US. Which is one reason why I’m trying to figure out why my machine is pinging Chinese IPs.
Thanks. I used to download source and compile, but it played hell with keeping the systems up to date. I now only use dnf. I wish there was a way to tell what processes were initiating the pings, dangit.
You could use wireshark on Rocky and record the session late and then analyse it. This could shed some light on what is doing the ping at least. Obviously would mean running it until the ping was made. Not sure how often it was doing it from your original firewall screenshot, but if regularly shouldn’t be too hard to find it.
Okay, I guess I’m going to have to do the Wireshark thing. To help isolate the problem, I downloaded a fresh instance of Rocky 8.6 and again verified the SHA256. I did a clean install using the DISA GUI STIG (since 8.6 removed the CJIS Security Policy) and did a ‘sudo dnf -y update’ then waited for six hours. No scary traffic. I then did ‘sudo dnf -y install dnf-automatic’ and ‘sudo systemctl enable --now dnf-automatic.timer’ and waited over night. The box is now doing the scary stuff again (see screen capture). China, the Russian Federation and Kazakhstan? Why on earth would the box be attempting to ping these because of dnf-automatic?
(Note: I have not installed EPEL or fail2ban at this point. Only Rocky Linux 8.6 and dnf-automatic from the base repo).
I think I’ll try this with a clean install of RHEL 8.6 and see if I get the same results to isolate whether this is a Rocky behavior or an upstream behavior. I’ll report the results here.
Can you explain what that means?
dnf-automatic looks like a python thing, there’s a config file called ‘/etc/dnf/automatic.conf’, can you have a quick look at it? You may also be able to check the python source code.
It was a good idea where you installed pure Rocky to test this.
There should be a way to monitor ICMP without wireshark, e.g. using firewalld logging, or some kind of network monitoring. I seem to remember it’s not obvoius how to do it using tcp dump (something to do with tcp not being icmp).
EDIT;
I didn’t realize wireshark is in the official Rocky appstream repo, so no problem to use that.
Can confirm I got the same behavior on one of my tests when I was working on a kickstart config. Pointed my kickstart at dl.rockylinux.org/pub/rocky/8.5 and captured this traffic.
Your firewall is listing pings as “echo-reply (ICMP)”. The other ICMP traffic will be seen in the capture.
Does this vm have static NAT configured on the firewall? The traffic looks port scan related.
When you do the initial install, you have the option of selecting a security policy for the installation configuration. It’s in the bottom right quadrant of the install screen. You have different policies from which to choose. I used to use the FBI’s CJIS policy, but now it’s gone. So this time I used the DoD’S DISA GUI STIG policy.
Sorry, I’m home for the Memorial Day weekend, so I can’t look into files. I did, however, run the test with a clean RHEL 8.6 install with no dnf-automatic, and it’s doing the same thing. I’ve created a ticket with RedHat.
Actually, if you look at one of my earlier posts, you’ll see that my firewall distinguishes between the outbound pings and the inbound replies. I’m not worried about the echo replies, because as you said, they’re the result of ping scans. That’s not the issue.
This VM does have static NAT to the Internet, but ping is currently the only inbound thing I’m allowing.
Since this is going to be a production server, I re-loaded the VM with a copy of RHEL 8.6, with a fully-supported subscription during the install. I didn’t even have to run the dnf-automatic, the clean install with nothing else started doing this.
Since we have support, I’ve started a ticket with RedHat, and already given them the logs they asked for. More to follow after the Memorial Day weekend.
Does your firewall allow icmp type 0 and icmp type 8 only? (ping request and reply) Or does it allow all icmp?
Thank you @IDtheTarget for identifying this issue and sharing it here. I also support a large number of RHEL VM’s that have the STIG baseline applied. Am eagerly awaiting your follow up, and curious what Red Hat has to say about this.
I was able to monitor a range of machines today; RHEL7, RHEL8, Rocky 8 on real hardware and Rocky 8 on vm. None of the machines were using DISA GUI STIG, so I don’t know if that makes a difference.
tcpdump -i enp1s0 "icmp"
I was surprised to see almost zero icmp traffic; The only time it lit up was when I ran a ping from a second terminal (and/or another machine).
You bring up some really good questions. I’m digging into the firewall to see exactly what is allowed through. The firewall setting just says “Accept ICMP requests”. Based on your earlier suggestion, I’m looking into if the box is simply replying to ICMP requests.
First thing I did was to look at the Geo protection policy. The setting shows to Drop (not Block, which would send a reply), both to and from China (and other hostile countries).
Second thing I did was to look at the IP addresses that the Geo policy was dropping, and see if there were incoming packets from those IPs. There are not.
Third, I tried to gather info to answer your question about ICMP types. Unfortunately my log only shows “ICMP”, not type. Which makes me realize that I was assuming that the drops were ping. Your question reminded me that there are other types of ICMP traffic. So now I’m looking into how to watch the pings as they happen at the network layer.
Thanks for helping me to consider alternatives. Sometimes I jump to conclusions in my hurry to fix things. Red Hat is still reviewing my logs and say that they’re bringing in a network expert, I’ll report back when I hear more or figure out the ICMP details.
Thanks!
One of the things I wondered about is if this is actually a corruption of my internal DNS. That this box was making an innocent query to a legitimate host, but a corrupt DNS server could be changing the IP. So I changed the resolv on this box to be 1.1.1.1, and I’m still getting the same issue. More to follow.
