Interesting those IPs in your screen shot are listed at
Looks like a lot are port scans or unauthorised connection attempts to port 4899
I haven’t been able to recreate this, but I would recommend adding some auditd rules to help diagnose which process is doing this:
echo “-a always,exit -F arch=b64 -S socket -F a0=2” > /etc/audit/rules.d/socket.rules
echo “-a always,exit -F arch=b32 -S socket -F a0=2” >> /etc/audit/rules.d/socket.rules
echo “-a always,exit -F arch=b64 -S socket -F a0=10” >> /etc/audit/rules.d/socket.rules
echo “-a always,exit -F arch=b32 -S socket -F a0=10” >> /etc/audit/rules.d/socket.rules
Reboot, then any time a process opens an IP socket (including a raw socket for ICMP), you’ll see an entry in /var/log/audit/audit.log
Match up your firewall ICMP timestamps to your audit log timestamps, and you should theoretically find your process.
Just a comment about tcpdump. It does capture ICMP, ESP, UDP and well as TCP and other protocols as well. I just tested it myself on a digitalocean cloud based VM, I generated some ping traffic to the VM and tcpdump recorded it along with all the other stuff that was happening at the time. I just did:
tcpdump -i eth0 -w /var/log/packetdump
That captured everything going in and out of eth0. Inspecting the packetdump file with wireshark, my own pings are the only ping traffic I see but I didn’t run it long as the server is busy running two web sites and the file was growing quite fast.
I didn’t get any choice on security policy for the install as it was a pre canned Digitalocean image for Rocky Linux that they provide. I don’t see any untoward traffic over the few minutes I left it running. It has two third party repos, EPEL and the REMI repo. I block ssh and cockpit traffic from everywhere except for certain known address ranges. I don’t block access to the web sites at present but they do have some level of protection via the cloudflare WAF system.
Anyway, I am curious to see if there is any unexpected outgoing ICMP. I have left the following running:
tcpdump -i eth0 icmp -w /var/log/packetdump
I will report back after it has been left for a few hours and see if anything shows up.
After running the tcpdump for a day capturing all ICMP traffic, I can see lots of incoming icmp echo with the expected icmp echo-reply from my server. The only server initiated outbound ICMP appears to be ICMP Type 3 code 13 (destination unreachable - administratively filtered). I’m wondering if those are triggered by attempted connections to ports that are blocked by firewalld. Need to do some more research.
I did some further testing. From my laptop I used nmap to connect to a number of TCP ports on the server that have nothing listening and are also blocked by firewalld. For each TCP SYN packet sent by nmap, there is an ICMP Type 3 code 13 immediately sent back to my laptop.
But if there’s nothing listening and they’re blocked by firewalld, how were you able to “connect”? Would it not just time out?
I’m assuming the laptop is in your house, and the server is in the “cloud”?
and did that ICMP originate from the digitalocean server?
So in the context of the original post, you are saying that any Rocky device that receives a SYN would reply with an ICMP?
This is an example of what tcpdump running on the DO server sees for TCP 33 or any other blocked port…
Laptop IP ---------> DO Server IP TCP 33 SYN
DO Server IP -----------> Laptop IP ICMP Type 3 Code 13.
This happens for every TCP port blocked by firewalld on the DO server. I’m not sure why the SYN packets are not just silently dropped. To me it seems to be revealing information to the port scanner unnecessarily. It may be firewalld that is doing it, I’m not sure. I don’t see this behavour on another server running an old version of CentOS, which doesn’t use firewalld, just iptables.
This seems consistent with the RHEL8 documentation, have a look at this section. Under “Blocking ICMP requests without providing any information at all” there’s an example of how you can configure firewalld to DROP instead of just block for everything in a zone.
Yes, this appears to be normal, and also helps with IPv6.
In the context of the original post, it would be important to clarify whether these pings are merely replying to incoming requests, or if they’re sprouting from the local machine for no apparent reason.
Thanks for that. I hadn’t seen that before. So after
firewall-cmd --permanent --set-target=DROP
firewall-cmd --reload
I don’t see the outgoing icmp type 3 code 13 any more but instead nmap tries three tcp SYN packets before giving up. So it does provoke some extra SYN attempts.
This is what a part of “default” ruleset (created by firewalld) does look like:
# nft -a list ruleset inet
table inet firewalld { # handle 14
...
chain filter_INPUT { # handle 135
type filter hook input priority filter + 10; policy accept;
ct state { established, related } accept # handle 139
ct status dnat accept # handle 140
iifname "lo" accept # handle 141
jump filter_INPUT_POLICIES_pre # handle 143
jump filter_INPUT_ZONES_SOURCE # handle 145
jump filter_INPUT_ZONES # handle 147
jump filter_INPUT_POLICIES_post # handle 149
ct state { invalid } drop # handle 151
reject with icmpx type admin-prohibited # handle 152
}
...
}
Note that the policy of this chain is accept – if no rule handles a packet, then it gets through.
However, the last rule in the chain (handle 152) does match every packet and drops them – politely.
The “polite” means sending back a “I’m not talking to you” ICMP packet.
When you just drop you don’t answer in any way. The other party that tries to connect (nmap, with TCP) has to wait until the TCP timeout for (delayed) answers that never will come. The nmap, being what it is, might try more than once “just in case”.
In nmap, there’s also an option; --max-retries
I decided it would be good to have a way to monitor all connections of a specific type, both coming in and going out. e.g. “log all icmp connections”, because tools like ‘netstat’ and ‘ss’ don’t show (or log) e.g. a one millisecond connection. I initially looked at kernel syscalls related to networking, but then remembered that the firewall already knows all about networking. In an earlier post I suggested using firewalld logging, assuming it would be easy (having used iptables in the past).
But, it seems firewalld is very much focused on ‘incoming’ traffic, and in this case we want outgoing as well. I looked into the rich rules language and the “direct” interface, but couldn’t work out how to do it with the Rocky default nftables backend. At this point, I’m not even sure if firewalld is a good choice for a big production server, might be better to just use nftables instead?
Anyway, here are the commands I ended up with:
log all outgoing icmp connections, including replies to incoming probes
Important note, this command creates a temp rule, it will go away when you reboot or reload firewalld, don’t try to save it unless you know the risk.
as root
get rid of any temp rules currently in memory
firewall-cmd --reload
look at the output chain
nft list chain inet firewalld filter_OUTPUT
add the rule, this MUST be all on one line
nft add rule inet firewalld filter_OUTPUT ip protocol icmp log prefix \"tag_icmp: \" level info
issue a command that will NOT log anything
ncat -z -v -w 1 --udp 8.8.8.8 53
issue a command that WILL log something
ping 8.8.8.8
issue another command that will log
ping www.google.com
look at the tail of the log
journalctl -f -e -k
ping your device from somewhere else
check the log again
leave it running for ages and see if it logs anything
clean up - lose the temp rule
firewall-cmd --reload
you could change it slightly to also log incoming, but the original post is to do with Rocky initiating the connection.
The OP needs to check that the ICMPs are not actually replies to attempted TCP connections to ports blocked by firewalld. This is what I suspect is happening. Anyway, the easy way to tell is that they are ICMP type 3 code 13. I’m not certain, but the OP’s log screen shot looks very much like the log screen you get from the Checkpoint Smart console. If it is you can double click on any log message to get full details but only the OP can do that for us.
The logging explained above makes it clear, so if anyone is concerned, the first step is to enable logging and then post any anomily here. In my own tests, I’m not seeing anything of concern.
If the suspect comms can’t be captured on the suspect box, then you should probably move to paranoia level 1 and assume the hacker has interfered with your ability to detect them using normal tools.
Using a suspect machine to diagnose it’s own infection might work but that’ll depend on how sophisticated the infection is and you can never know.
If I was genuinely worried about it, I’d start again with a clean install and do AIDE audits on the target box using another clean air-gapped box to scan the target offline at various points including after OS install, after 3rd party install, after comms detected.
Also, if your router (uncompromised as showing the data) has port-mirroring you might use that to capture all data over a period on a seperate box with NIC in listen-only mode. Alternatively, a tap or hub. And if you’re going to do that you might as well put snort on the listen-only box.
Applying Occam’s razor, I haven’t seen any proof or indication so far that Rocky Linux or RHEL should be compromised out of the box, which would indeed be a gigantic issue. What I have seen though, is plenty of things indicating that what’s actually happening here is simply an ICMP reply to these machines, which is indeed outgoing traffic, but only because it was prompted by that remote system in the first place.
As the RHEL documentation linked in my previous post explains, the default behavior is not to completely drop incoming ICMP requests, therefore this would be expected behavior.
Maybe this was different in RHEL7. Maybe every single RHEL8 system in the world is rooted by Chinese Intelligence. Maybe the moon is made of cheese.
If you reply to this thread, I suggest that you please do so because you’re adding new and valuable information to answering the question that the thread addresses.
If the machine is directly exposed to the internet, it’s normal that it would reply to icmp requests, but many people will have an edge firewall, set to block icmp before it even gets to Rocky, or they could configure Rocky to drop packets. The important thing is that Rocky is not initiating the requests.
Whether it’s a good idea to drop icmp is a different topic, for example you can ping ‘www.google.com’ right now.
Sorry I haven’t been back in touch. You were completely correct. A deeper dive into the packets showed that this was, in fact, a reply to an ICMP request instead of originating from my box.
I sincerely apologize for inciting drama, I should have done more investigation before posting here.
Thank you!