Is nwfilter supposed to work or no?

I cannot get nwfilter to work properly.

I have a kvm host. Defined a bridge, kvm guest joined to the bridge. Everything seems to work.

However, I would like to filter traffic to/from guest on the host, but for the life of me, cannot figure out if nwfilter works on rocky at all.

I assign a filter on the guest VM network interface in XML: <filterref filter="testing"/>. As soon as i start the guest VM with this, there is no traffic in or out. Even though the filter testing currently contains:

  <filterref filter='allow-arp'/>
  <filterref filter='allow-ipv4'/>
  <filterref filter='allow-dhcp'/>

which should allow almost anything. No TCP connections in or out seem to get through. However, icmp ping works out of the VM to anywhere.

Is nwfilter supposed to not work?

They do seem to create some ebtables entries, ebtables-save shows currently:

*filter
:INPUT ACCEPT
:FORWARD ACCEPT
:OUTPUT ACCEPT

*nat
:PREROUTING ACCEPT
:OUTPUT ACCEPT
:POSTROUTING ACCEPT

:libvirt-I-vnet0 ACCEPT

:libvirt-O-vnet0 ACCEPT

:I-vnet0-ipv4 ACCEPT

:O-vnet0-ipv4 ACCEPT

:I-vnet0-arp ACCEPT

:O-vnet0-arp ACCEPT

-A PREROUTING -i vnet0 -j libvirt-I-vnet0
-A POSTROUTING -o vnet0 -j libvirt-O-vnet0
-A libvirt-I-vnet0 -p IPv4 -j I-vnet0-ipv4
-A libvirt-I-vnet0 -p ARP -j I-vnet0-arp
-A libvirt-O-vnet0 -p IPv4 -j O-vnet0-ipv4
-A libvirt-O-vnet0 -p ARP -j O-vnet0-arp
-A I-vnet0-ipv4 -p IPv4 --ip-src 0.0.0.0 --ip-dst 255.255.255.255 --ip-proto udp --ip-sport 68 --ip-dport 67 -j ACCEPT
-A I-vnet0-ipv4 -j ACCEPT
-A O-vnet0-ipv4 -p IPv4 --ip-proto udp --ip-sport 67 --ip-dport 68 -j ACCEPT
-A O-vnet0-ipv4 -j ACCEPT
-A I-vnet0-arp -j ACCEPT
-A O-vnet0-arp -j ACCEPT

iptables-save:

*filter
:INPUT ACCEPT [3003:315045]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2145:143170]
COMMIT
*security
:INPUT ACCEPT [2253:149759]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2146:143418]
COMMIT
*raw
:PREROUTING ACCEPT [4082:570579]
:OUTPUT ACCEPT [2149:143938]
COMMIT
*mangle
:PREROUTING ACCEPT [4084:570683]
:INPUT ACCEPT [3007:315253]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2150:144162]
:POSTROUTING ACCEPT [2150:144162]
COMMIT
*nat
:PREROUTING ACCEPT [2100:437092]
:INPUT ACCEPT [272:16324]
:POSTROUTING ACCEPT [311:20734]
:OUTPUT ACCEPT [311:20734]
COMMIT

What am i missing?

One potential thing is nftables. Kernel of Rocky has nftables. At least the ‘iptables’ is just a wrapper that translates iptables-syntax rules into nftables (and back). I have no idea whether libvirtd&co use iptables-syntax or do they use nftables directly for some rules.

You can see the actual in-kernel ruleset with:

sudo nft list ruleset

As @jlehtone said, Rocky 8.5 like RHEL 8.x uses firewalld with nftables, this behaviour can be changed though by editing /etc/firewalld/firewalld.conf:

# FirewallBackend
# Selects the firewall backend implementation.
# Choices are:
#       - nftables (default)
#       - iptables (iptables, ip6tables, ebtables and ipset)
FirewallBackend=nftables

the backend can be reverted to iptables like it was in RHEL7. If you are unable to get it to work with firewalld or libvirt forces uses of iptables, then you can always revert firewalld by editing that file and restarting it. You may wish to completely reboot the server though as I’ve had issues before under Debian/Ubuntu when changing firewalld from iptables to nftables as they default to iptables.

Yes and no, AFAIK. Yes, the firewalld.service can call “iptables” instead of directly using the nftables-library that nft uses. That is, firewalld can generate rules in “iptables syntax” or in “nftables syntax”.

However, the “iptables” is not the iptables-legacy that talks to netfilter subsystem within kernel. The “iptables” is a disguised version of nft that takes “iptables syntax” and talks to nftables subsystem within kernel in “nftables syntax” just like the nft does. Granted, nftables subsystem does use some features of netfilter subsystem, rather than reimplementing everything.

Where does this show, if anywhere? Logical scenarios are that (A) libvirt talks to firewalld and (B) both libvirtd and firewalld talk to kernel. In case A access to kernel is consistent. In case B there are “two chefs on the soup”. In both cases all communication could be in one syntax (iptables?), or there is a mix that parts of ruleset mimics “good old” netfilter chains and part are “plain nft”.

There is an extra twist: bridged traffic is not filtered by default. (It has never been filtered by default in RHEL systems). There was time, when it was not possible at all. Then there was third-party kernel module that made bridged traffic visit the netfilter. Module like that became native, but RHEL explicitly disabled it in config. Now the module is no longer loaded automatically when you create a bridge.

By that, KVM must load the kernel module and re-enable the filtering in config (after default disable), if nwfilter is defined. That is the only explanation why anything (bridged) gets filtered.

1 Like

@jlehtone very informative. It would be good to get to a situation where everything talks via firewalld, that way it doesn’t matter what the backend would be, be it nftables or iptables or whatever else may appear in the future. I’ve seen this for example with docker, it would create itself iptables rules even when firewalld was active with nftables. Had it used firewalld syntax for adding rules, it wouldn’t have mattered which was being used. Didn’t dig any further to see if I could change it, but did notice it at least.

Again, yes and no. The firewalld.service is indeed the default, but there is also nftables.service that Red Hat recommends for “serious use” (since FirewallD still lacks support for some “usual scenarios”).

If I don’t use firewalld and service xyz talks only to FirewallD, then what do I do?

In earlier EL (6 or 7) the libvirtd injects rules directly to backend, into “INPUT/FORWARD” chains. At least once I did modify libvirtd-scripts (point of rule insertion) so that my unrelated custom rule remained first in chain. In EL8 libvirtd creates named chains that are more controllable? (I have mainly bridged setups without any VM filter on host-side.)

I admit I am a bit overwhelmed at the nft ruleset I am seeing (more familiar with iptables). Rules seem to be created in nftables, but they seem to have no effect.

For instance, want to allow port 3389 to a guest VM. In guest bridged network conf i have:

<filter name='testing' chain='root'>
  <filterref filter='allow-rdp'/>
</filter>

virsh nwfilter-dumpxml allow-rdp

<filter name='allow-rdp' chain='ipv4' priority='-700'>
  <rule action='accept' direction='in' priority='100'>
    <ip protocol='tcp' dstportstart='3389'/>
  </rule>
</filter>

#nft list ruleset
(relevant sections)

  chain POSTROUTING {
                type filter hook postrouting priority srcnat; policy accept;
                counter packets 6867 bytes 877794 jump POSTROUTING_direct
                oifname "vnet1" counter packets 6808 bytes 874671 jump libvirt-O-vnet1
        }
chain libvirt-O-vnet1 {
                ether type ip counter packets 89 bytes 8348 jump O-vnet1-ipv4
                ether type arp counter packets 2293 bytes 105478 jump O-vnet1-arp
                counter packets 4416 bytes 759465 accept
        }
        chain O-vnet1-ipv4 {
                ether type ip udp sport 67 udp dport 68  counter packets 0 bytes 0 accept
                ether type ip tcp dport 3389  counter packets 0 bytes 0 accept

Somehow the packet does not reach the dport 3389 rule at all?

Log:
FINAL_REJECT: IN=virbr2 OUT=virbr2 PHYSIN=eno2 PHYSOUT=vnet1 MAC=[edited] SRC=[edited] DST=[edited] LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=57617 DPT=3389 WINDOW=65535 RES=0x00 SYN URGP=0

nft ruleset could be very compact. Alas, firewalld tends to be very verbose, with intricate hierarchies of chains. That is for achieving machine actionability, I presume.

POSTROUTING is after all filtering steps. Where are rules that could write to logs?
(I don’t recall any “log” rules in the default ruleset.)

The default/final filter rule is typically:

reject with icmpx type admin-prohibited

Furthermore, there can be multiple chains on same hook with different priorities and the effective result takes a lot to grok.

Yea, that is what i meant. Firewalld+libvirt seems to have achieved a very complicated set of rules.

Logging in this case is achieved by setting Logdenied=unicast in firewalld.conf which seems to add a log rule just before every icmp reject.

Furthermore, dumping just the firewalld’s ruleset an the firewalld+libvirt ruleset … getting diff of those is not easy, not to mention that the reject could be in plain firewalld part.

As a test, I decided to forget the nwfilter side of things and try to create rules for bridge directly according to this: Bridge filtering - nftables wiki

However (no doubt resulting from some mistake or assumption i made), when i do:

# nft add table bridge filter
# nft add chain bridge filter forward '{type filter hook forward priority 0; }'
# nft add rule bridge filter forward ct state established,related accept
Error: Could not process rule: Protocol error
add rule bridge filter forward ct state established,related accept
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

can anyone shed a light on the last error (google is not my friend)?

Replying to myself here, in case someone else stumbles upon the same issue. nft ct state rules are not supported by the 4.8 kernel that is shipped with Rocky Linux 8 by default. Installing kernel-lt 5.4 from EPEL repo fixes it.

2 Likes