I found the source.
The firewalld default options was changed, but not noted in the release notes
After set NftablesTableOwner=no in /etc/firewalld/firewalld.conf, now the flush will works again.
The external flush will be needed, because firewall-cmd can only add or remove one entry from set, but not flush the set.
NftablesTableOwner
If set to yes, the generated nftables rule set will be owned exclusively by firewalld. This prevents other entities from mistakenly (or maliciously) modifying firewalld’s rule set. If you intentionally modify firewalld’s rules, then you will have to set this to “no”. Defaults to “yes”.
The RHEL 9.5 release notes does have in its New Features:
You can use both firewalld and nftables services simultaneously
The firewalld and nftablessystemd services are available to use simultaneously. Previously, users could enable only one of these services at a time. With this enhancement, these systemd services no longer conflict with each other.
I’d guess that the NftablesTableOwner was added/enabled in order to avoid the conflict.
But this will not be true. Because under 9.4 and earlier, both(firewalld and nft) can modified the nftables by default. After the update only firewalld can it by default. So the documentation looks like incomplete. I will open an ticket on upstream for it.
This new option adds two flags for the table(s) that FirewallD creates: owner and persist.
The nf_tables has apparently had flags for a while. Its documentation:
Flag
Description
dormant
table is not evaluated any more (base chains are unregistered).
owner
table is owned by the creating process.
persist
table shall outlive the owning process.
Creating a table with flag owner excludes other processes from manipulating it or its contents. By default, it will be removed when the process exits. Setting flag persist will prevent this and the resulting orphaned table will accept a new owner, e.g. a restarting daemon maintaining the table.
That basically means that nftables.service (or anyone else) cannot modify the table owned by firewalld.service. As long as FirewallD does not touch other tables than the one it creates, both services can “safely” co-exists. (I’d guess one can still cause trouble though.)
Important
To prevent the different firewall-related services (firewalld, nftables, or iptables) from influencing each other, run only one of them on a RHEL host, and disable the other services.
Clearly not updated to match the 9.5’s options (but we have to remember that RHEL EUS/EEUS subscribers can pin to old versions, which makes uniform documentation more challenging)?