Rocky-8: FirewallD reload extreme slow with hughe IPSET list

Rocky Linux Tech Help
1

I use a huge IPSET list with around 30k of known bad ip addresses.
However, every firewall-cmd --reload takes ~19 minutes to reload.

Resources are plenty, so no bottleneck here.

My Rocky-8 system is the latest version, same goes for FirewallD (strict from the @System repo). FirewallD uses this version, firewalld-0.9.11-9.

I’ve read multiple threads on the internet, that firewallD had some bugs in its older versions.

Can anyone confirm this or does anyone know a workaround?

2

Have you checked it with RL9?

3

I’ve seen similar posts somewhere …
Found it/them: Firewalld-0.9.11-1 update drastically slowed ipset importing on AlmaLinux 8 · Issue #186 · AlmaLinux/almalinux-deploy · GitHub

The ‘ipset’ is a “netfilter thingy”. The nf_tables has also “named set”. That creates question:

  • Are ipset and nf_tables set totally separate?
  • Are ipset and nf_tables set equally slow to populate?
  • Can both be populated via FirewallD, or must one use nftables.service / other script?
  • If both are possible, does FirewallD affect the speed (of reload)?
4

I did a couple of seconds ago:

Here the Rocky-9 specs (software side):
Rocky-9: 9.4
FirewallD: firewalld-1.3.4-1.el9

A reload took around 6 seconds (Rocky-8 around 19 minutes), listing the ipset info cmd took less than 1 sec, in Rocky-8 ~18 minutes.

5

The versions of upstream sources that RHEL packages are based on:

Distro kernel nftables ipset firewalld
el8 4.18.0 1.0.4 7.1 0.9.11
el9 5.14.0 1.0.9 7.11 1.3.4

(Backports may affect performance too.)

6

With “IPSET” I don’t mean that package … I mean the function inside of FirewallD and its xml files (and yes, it uses nftables).

Anyway, I’m somewhat shocked. RHEL 8 (and all its derivates) are enterprise products … why is this even an issue???

Those big ip lists are one of the best options to secure an edge server.

7

How are you using the list of IP’s? Are you parsing it and then doing something like:

firewall-cmd --zone=myzone --add-source=$ip --permanent

or loading it some other way?

8

Even in el9 the man firewall-cmd does point to man 8 ipset even though the “ipset” it has created is an nftables set. What does FirewallD of el8 actually do? Do you see your “ipset” in nft list ruleset, or do the rules point to ipset ipset?

That you have to ask from Red Hat. I do agree that such performance regression is not what one expects.

9

If it is “FirewallD XML”, then it is probably something like:

# cat /etc/firewalld/ipsets/bad.xml
<?xml version="1.0" encoding="utf-8"?>
<ipset type="hash:net">
  <entry>10.2.9.16</entry>
  <entry>10.3.0.10</entry>
  <entry>10.5.0.34</entry>
</ipset>

and it is up to FirewallD to parse that.

(Static) config for nftables.service could eat:

set bad {
	type ipv4_addr
	flags interval
	elements = {  10.2.9.16, 10.3.0.10, 10.5.0.34 }
}
10

firewall-cmd --permanent --ipset=known_malicious_ips --add-entries-from-file=ban_me_i_am_bad.txt

—> done after some seconds

time firewall-cmd --reload
success

real	18m14,621s
user	0m0,211s
sys	0m0,031s
11

With
nft list ruleset

I see all 30k ip’s in less of a second.

Reloading firewalld is the problem

12

Maybe the ipset thing is slower? I dunno, I do it a different way. For example, I get a list of IP’s from Cloudflare or my CDN, and then I parse over them and add them to firewalld using the add-source option. I suppose you could try that method and see if it’s any faster. My IP lists are shorter than yours so I don’t have any speed issues.

13

This presumably creates/modifies file /etc/firewalld/ipsets/known_malicious_ips.xml
The ban_me_i_am_bad.txt “should contain an entry per line”, and inserting text to xml is relative trivial.

Now firewalld reads the XML file and writes to kernel (via nft library?). Sounds trivial, but time disagrees.

One could dump the active ruleset to file with nft list ruleset, disable firewalld.service, and make nftables.service load the ruleset from file. If that is faster, then the issue is clearly in the firewall-nft transaction.

14

This is one of the reasons I don’t use firewalld. The biggest problem with firewalld is that a complete reload is required when changing/adding/removing rules form what I have experienced. If you can modify firewalld rules without reloading them from scratch I would consider using it. With iptables you can update, add and delete rules on the fly leaving all the other rules untouched intact and protecting the server.

15

You can, just add the rule without adding --permanent on the end. So effectively, if I was adding port 6443, I would do this:

firewall-cmd --zone=myzone --add-port=6443/tcp
firewall-cmd --zone=myzone --add-port=6443/tcp --permanent

The first line makes it active now, the second makes sure we save it to be used when reloaded/restarted. If we just used the first line and reloaded/restart firewalld, the port would no longer be there and we’d have to add it again. Doing it like the above doesn’t require reload to be used. There maybe a better method, but none that I know of right now.

1 Like
16

This makes the rule active without restarting firewalld? In the same manner you can do a
iptables -I INPUT blah blah ACCEPT ?

17

firewalld uses nftables, not iptables. But if you prefer iptables, go for it and use that instead of firewalld.

18

Yes. The firewall-cmd can change both the current active rules in the kernel (without --permanent) and update config (with --permanent) that is used on boot to load rules into kernel.
There is even possibility to give a timeout parameter for the former. You can add a rule that is automatically removed in, say 5 minutes. Within that time you see whether the rule works as expected, but are not permanently locked out from your system if the rule is “bad”.

The iptables (and nft) do change the current active rules in the kernel. They don’t change the stored config. (The iptables in Rocky is only a nft-translator that speaks nf_tables to the kernel.)

Furthermore, if you modify rules with iptables or nft while FirewallD “manages” the rules, it is “same manner” as your girlfriend rummaging your wallet while you are not watching – your assumptions about the contents of your wallet might become invalid.

19

When I manage with iptables my girlfriend has no idea what has been changed nor does she have access to modify anything or take anything out of my wallet. The “Config” of the firewall is dynamic and changing all the time. With very few static items.

20

When I was messing with firewalld and added rules (thinking they would be active on the fly) then checked active rules they do not display as active until firewalld is restarted.