Firewalld with docker, wireguard and fail2ban explanation

Rocky Linux General
1

Hi

I try to figure out how firewalld works. I’ve read a lot all over the internet but it doesn’t get in my head how firewalld really works. I need a bit advice/help from you guys. It’s my first time to use firewalld.

I have actually mainly 4 zones active:
public:

public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens3
  sources: 
  services: ssh http https wireguard
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

docker:

docker (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: docker0
  sources: 
  services: 
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

wireguard:

wireguard (active)
  target: default
  icmp-block-inversion: no
  interfaces: wg0
  sources: 
  services: 
  ports: 
  protocols: 
  forward: no
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

drop:

drop (active)
  target: DROP
  icmp-block-inversion: no
  interfaces: 
  sources: IP
  services: 
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

DOCKER

So let’s start with docker first.
I would like to ban an ip for the docker zone. I tried already firewall-cmd --add-rich-rule='rule family=ipv4 source address=IP reject' --zone=public --permanent also for the docker zone. Both of them doesn’t work so I give the drop zone a shot at the moment firewall-cmd --zone=drop --add-source=IP --permanent. I always reload firewalld after that firewall-cmd --reload.

WIREGUARD

Currently I’m setting up wireguard and created the wireguard zone for that. The access to the server works but the peers doesn’t have any internet connections. Might be forwarding IP and masquerade doesn’t really work. So I enabled on both zones (public & wireguard) masquerade but this doesn’t seem to effect anything. I run sysctl -w net.ipv4.ip_forward=1 which has also no effect. I’m a bit stuck here as well.

fail2ban

banaction = firewallcmd-rich-rules[actiontype=]
banaction_allports = firewallcmd-rich-rules[actiontype=]

Ban works but nothing is listed under zones.

Drop everything first

I’m also thinking about to drop everything first and allow certain ports (ssh, http, …) only also for the docker zone. What needs to be done for that with all the zones? Default zone wouldn’t be public anymore and I would change it to drop zone?

I hope you guys can enlighten my brain a little :slight_smile:

2

If packet is dropped, then that is it. The default is to have first some rules that allow packets that match, say ssh, and then the “match all that were not allowed” that rejects (politely drops).

Forwarding requires three things:

  • net.ipv4.ip_forward = 1 so that kernel does attempt to forward packets. Some zones enable this by default
  • Routes that point some packets to go through, rather than to enter the host. This is usually automatic, although “default route” can be tricky
  • Filter rules that allow packets to be forwarded. In RL8.6 and RL9 the firewalld finally supports forwarding with policies. Before that one had only rich rules and (now deprecated) direct rules. A policy states what is allowed and denied to travel from zone A to zone B.

Masquerade is sNAT. The sNAT changes source address on packets. Let say client A (in LAN) sends packet to server B (in WAN). They are not on same subnet, so packet goes to through router R. R has addresses in LAN and WAN. Therefore, A sends packet to Rlan. Router forwards packet to B.

  • If there is no NAT, then B receives packet “from A”, and it must have a route “to LAN via Rwan” in order to send reply correctly
  • If R masquerades traffic that goes to WAN, then it replaces source address with Rwan. B receives packet “from Rwan” and replies. R sees the packet as reply and restores, i.e. sends reply with “from B to A” to A
  • If R masquerades also traffic that goes to LAN, then it must modify that packet to say “from Rlan to A”. The A had send a packet to B, but only Rlan tries to talk with it. That fails
    So no, do not masquerade both directions.

Wireguard is VPN. Does it update “default route” on client, when VPN tunnel is established? That is quite common. If so (default route is via tunnel subnet and VPN server), then the client will send everything except wireguard connection (and link-local stuff) through the tunnel subnet and server must forward traffic.

A “zone” is a list of machines.
You have four lists:

IF packet is from IP, THEN do A
ELSE IF packet is from machine that connects through ens3, THEN do B
ELSE IF packet is from machine that connects through docker0, THEN do C
ELSE IF packet is from machine that connects through wg0, THEN do D

We do not care through which interface packets that have “from IP” do come from, they will be allowed or denied based on rules on zone A (which in your case is “drop” that drops everything). No additional rich rules are required.

A service (like fail2ban) has three options:

  1. Assume that firewall rules are already correct. (Not true for fail2ban, which changes rules dynamically.)
  2. Modify rules in the kernel directly
  3. Ask FirewallD to modify rules in the kernel. Even there could me many approaches

Option 2 is obviously worst.

FirewallD knows both the (–permanent) config that it loads from files on start, and the active config that it has sent to kernel. You can check the actual rules that are in the kernel with:

sudo nft list ruleset
1 Like
3

zone A (which in your case is “drop” that drops everything)

Is not the public zone, zone A?
In my point of view everything hits zone public first and gets directed to the specific zone depends on the request.

Ok, if I understand this correctly, then adding the IP to the drop zone should also reject the request to the docker zone?
If so, why does the “IP” still get through to the docker zone and get blocked there via fail2ban.
Or I leave the drop zone as it is and don’t use it at all. How can I block the IP for the docker zone and why would that be that way?

Why does the port have to be open in the public zone for wireguard and not in the wireguard zone itself, which contains the interface? I do not understand the philosophy of the zones. Only if the public zone would be “A”. Where I should also block the IP for the docker zone. That would make sense for me.

To get an internet connection for wireguard peers, shouldn’t it work that way then?

firewall-cmd --new-zone=wireguard --permanent
firewall-cmd --add-interface=wg0 --zone=wireguard --permanent
firewall-cmd --add-service=wireguard --zone=public --permanent
firewall-cmd --add-masquerade --zone=public --permanent
firewall-cmd --add-forward --zone=public --permanent
firewall-cmd --reload
sysctl -w net.ipv4.ip_forward=1

Wireguard is VPN. Does it update “default route” on client, when VPN tunnel is established?

What exactly do you mean? How can I check that?

Fail2ban now also lists the bans under public zone. I also looked in nftables (nft list ruleset) before and it was listed there as well. So it actually worked, but it would be handy to use just one frontend to check. Either nftables or firewalld.
If I wasn’t using Docker I would probably just use nftables, this seems easier. But I have read it has some problems with Docker isolation etc…

4

Lets start with simple INPUT-rules without “zones”:

chain INPUT
1. allow established connections to continue
2. allow from A to SSH
3. allow from B to SSH
4. allow from B to HTTP
5. allow from C to HTTP
6. reject everything else

That looks simple, but gets quickly big when you add sources and/or services, and when services require many rules. The latter part we could manage by use of chains for services:

chain S
1. allow to SSH

chain H
1. allow to HTTP

chain INPUT
1. allow established connections to continue
2. from A jump S
3. from B jump S
4. from B jump H
5. from C jump H
6. reject everything else

Now all ports and helpers about HTTP(S) is in one place, rather than repeated for B, C, (and possible future clients). However, B is still twice in INPUT. Lets try something bit different:

chain onlyS
1. allow to SSH
2. reject everything else

chain onlyH
1. allow to HTTP
2. reject everything else

chain SandH
1. allow to HTTP
2. allow to SSH
3. reject everything else

chain INPUT
1. allow established connections to continue
2. from A goto onlyS
3. from B goto SandH
4. from C goto onlyH
5. reject everything else

This is “zone-based” ruleset.
The difference between “jump” and “goto” is that jump returns to caller, but goto one higher.
In previous ruleset packet (from B) did jump to S on rule 3. If it was not to SSH, then it did return to INPUT to test rule 4.
When packet from B goto chain SandH, it will not return to INPUT. Not even if we had no catch-all “reject” in SandH.

Each packet belongs to only one zone (in INPUT). The setup separates origin of packets from what is done to them.
The INPUT chain grows slowly. If we for example add new source D that should have access to only S, then we add from D goto onlyS. Both A and D will be in zone “onlyS”.

Interestingly, routing has now two cases:

  • The in and out are within same zone. I’ve hadn’t studied that
  • The in and out are in different zones. This is where one has to add a policy if one wants to allow some traffic to be forwarded

The fail2ban does not quite follow the “zone logic”, since when we ban a client from service, it logically is in a different zone (for the duration of the ban).

FirewallD is a frontend. It reads its config and generates ruleset into nftables that is in kernel. FirewallD does not show the actual ruleset, only its own config.
nftables.service is a simple “oneshot” service that loads ruleset from file into kernel. The nft is an utility that shows/edits ruleset that is in kernel.
The nft is thus a debug tool for us when we are not quite sure what FirewallD (and other parties that tamper ruleset) has done (wrong).

5

Lets start with simpler example: client (browser) and server (this forum). Client forms connection to server. Firewall on server must allow incoming (HTTPS) connections.
Both machines have only one interface each and no VPN, etc, so obviously the communication is via interface and subnet(s) that connect them.

The situation is exactly same with client (wireguard) and server (wireguard) processes; they communicate via regular network.
What is different is that wireguard client does not render http-content to GUI – it does create a virtual interface that is linked to a virtual subnet.

When you send a packet to that subnet (say browser connects to webserver that listens on the virtual subnet), the wireguard client/server takes the packet, encrypts it, and sends result to wireguard server/client via the regular connection, where the wireguard server/client decrypts the payload and then a packet “arrives” to that machine via its virtual wireguard interface “from virtual subnet” (for the webserver).

6

I always thought iptables/nftables is a frontend of netfilter and firewalld/ufw is a frontend of iptables/nftables to make it easier to understand.

Ok. I’ll try to figure it out with my setup (without drop zone).

docker

To ban the IP 123.456.789.10 for the docker zone it needs actually the following for me. But docker made also some other rules before firewallds (table inet firewalld {) …

table ip filter {
        chain DOCKER {
            ...
        }

which I don’t pay attention for now. But maybe it has something to do why the IP doesn’t get blocked.

So I think it will probably be that way if a machine is trying to reach the docker zone:

        chain filter_INPUT {
                type filter hook input priority filter + 10; policy accept;
                ct state { established, related } accept
                ct status dnat accept
                iifname "lo" accept
                jump filter_INPUT_ZONES
                ct state invalid drop
                reject with icmpx admin-prohibited
        }

        chain filter_INPUT_ZONES {
                iifname "docker0" goto filter_IN_docker
                iifname "wg0" goto filter_IN_wireguard
                iifname "ens3" goto filter_IN_public
                goto filter_IN_public
        }

How I understand your explanation (“they communicate via regular network”) it would be public. So I’ll keep going that way to ban the IP (firewall-cmd --add-rich-rule='rule family=ipv4 source address=123.456.789.10 reject' --zone=public --permanent).

        chain filter_IN_public {
                jump filter_INPUT_POLICIES_pre
                jump filter_IN_public_pre
                jump filter_IN_public_log
                jump filter_IN_public_deny
                jump filter_IN_public_allow
                jump filter_IN_public_post
                jump filter_INPUT_POLICIES_post
                meta l4proto { icmp, ipv6-icmp } accept
                reject with icmpx admin-prohibited
        }

        chain filter_IN_public_deny {
                ip saddr 123.456.789.10 reject with icmp port-unreachable
        }

That should actually reject the way to the docker zone, right?

wireguard

This is still very tricky for me, even though I solved the internet problem. But I’m starting to understand a little :wink:

port

Open port (firewall-cmd --add-service=wireguard --zone=public --permanent) works and based on your explanation (“they communicate via regular network”) it also should be on public zone:
1.

        chain filter_INPUT {
                type filter hook input priority filter + 10; policy accept;
                ct state { established, related } accept
                ct status dnat accept
                iifname "lo" accept
                jump filter_INPUT_ZONES
                ct state invalid drop
                reject with icmpx admin-prohibited
        }

        chain filter_INPUT_ZONES {
                iifname "docker0" goto filter_IN_docker
                iifname "wg0" goto filter_IN_wireguard
                iifname "ens3" goto filter_IN_public
                goto filter_IN_public
        }

        chain filter_IN_public {
                jump filter_INPUT_POLICIES_pre
                jump filter_IN_public_pre
                jump filter_IN_public_log
                jump filter_IN_public_deny
                jump filter_IN_public_allow
                jump filter_IN_public_post
                jump filter_INPUT_POLICIES_post
                meta l4proto { icmp, ipv6-icmp } accept
                reject with icmpx admin-prohibited
        }

        chain filter_IN_public_allow {
                tcp dport 443 ct state { new, untracked } accept
                tcp dport 80 ct state { new, untracked } accept
                tcp dport 22 ct state { new, untracked } accept
                udp dport 51820 ct state { new, untracked } accept
        }

masquerade

I think this is actually a pretty straight forward image:

Source Linux Network Administrator's Guide, 2nd Edition: Chapter 11: IP Masquerade and Network Address Translation

So based on that it makes sense to me to enable masqurade on zone public becaue from server IP/interface (ens3 - public IP - 10.987.654.321) needs to “translate” to wireguard (wg0) network (10.0.0.0).
And for the other way around (client → internet) it needs a new policy to allow masquerade, zone (wireguard) to zone (public):

firewall-cmd --new-policy NAT_int_to_ext --permanent
firewall-cmd --policy NAT_int_to_ext --add-ingress-zone wireguard --permanent
firewall-cmd --policy NAT_int_to_ext --add-egress-zone public --permanent
firewall-cmd --policy NAT_int_to_ext --set-target ACCEPT --permanent
firewall-cmd --reload

Which is a actually really good to have the possibility to do it for zones separately.

But the client connects to the Wireguard server even without the masquerade enabled?! Shouldn’t that be impossible?

When you send a packet to that subnet (say browser connects to webserver that listens on the virtual subnet), the wireguard client/server takes the packet, encrypts it, and sends result to wireguard server/client via the regular connection, where the wireguard server/client decrypts the payload and then a packet “arrives” to that machine via its virtual wireguard interface “from virtual subnet” (for the webserver).

But the connection still runs over the public IP. Doesn’t it need to be translated to 10.0.0.0 via masquerade?

What is different is that wireguard client does not render http-content to GUI – it does create a virtual interface that is linked to a virtual subnet.

Do you mean, there is actually nothing “coming back” for example to “render http-content to GUI”? But how is the connection possible when public IP is still involved (without masquerade)?

7

Lets have two machines, A and B:

  • A has “public” IP 10.10.0.1/24
  • B has “public” IP 10.50.0.2/24
  • There are routers between 10.10.0.0/24 and 10.50.0.0/24 so they can talk “directly”

The A runs VPN server that listens at 10.10.0.1:X. The B connects to 10.10.0.1:X with VPN client.
The VPN client on B uses port Y, so there is connection between 10.10.0.1:X and 10.50.0.2:Y.

When VPN connection is on the A has second interface and address 192.168.0.1/24 on it.
Likewise the B has second interface and address 192.168.0.2/24 on it.

Lets say that A starts a httpd service. It does listen on port 80.
B starts browser and loads page from http://192.168.0.1/
In other words, B sends a packet from 192.168.0.2 to 192.168.0.1:80. Network stack on B sends packet through the vpn interface.
Magic happens. Described later
A receives packet from vpn interface to 192.168.0.1:80. Httpd handles the request and sends reply from 192.168.0.1:80 to 192.168.0.2. That leaves through vpn interface of A.
Magic happens.
B receives packet from its vpn interface. It is from 192.168.0.1:80 to 192.168.0.2. Browser gets reply and you see the page.

There were no NAT here.

What was the magic? At the same time the VPN client in B did send a packet from 10.50.0.2:Y to 10.10.0.1:X, to VPN server in A and the server did reply from 10.10.0.1:X to 10.50.0.2:Y. These did use the public networks. The “from 192.168.0.2 to 192.168.0.1:80” and the from 192.168.0.1:80 to 192.168.0.2 were never seen on the public network.

However, if you could decrypt the encrypted data that VPN client and server did exchange, you would see the plain HTTP GET and its reply, including the “from 192.168.0.2 to 192.168.0.1:80” headers.

So, the browser did send a packet, but the VPN client “ate it”. Right after VPN client did send “something else”. Same with replies. Traffic goes back and forth as long as you browse.

Now, lets add one more subnet and machine (C):

  • B has “LAN” IP 172.16.0.1/24
  • C has “LAN” IP 172.16.0.3/24
  • C has “default via 172.16.0.1”; the “gateway” of LAN is B

Now browser in C loads page from http://192.168.0.1/. Packet from 172.16.0.3 to 192.168.0.1:80 is sent to B, because C does not know where 192.168.0.0/24 is. B knows and is set to route. Just like before, the packet goes to vpn “tunnel” and the httpd in A receives it.

A sends a reply from 192.168.0.1:80 to 172.16.0.3. Where is 172.16.0.3? It is not in 192.168.0.0/24 nor in 10.10.0.0/24. For A, the “gateway” is probably 10.10.0.Z, so A should send the reply via 10.10.0.Z. Obviously, that will never succeed. Connection to server was not established.

There are two solutions to that:

  • Tell A a route “to 172.16.0.0/24 via 192.168.0.2”. Now A knows to send the reply to vpn tunnel and if B agrees to route it back to C, then everything is fine
  • Not tell A, but make B masquerade packets that come from 172.16.0.0/24 and leave to vpn with address 192.168.0.2. With that when packet from C arrives to A, it looks like it came from B via VPN tunnel and reply goes to tunnel. The reply is automatically “demasqueraded” at B without additional nftable rules

The B has to obviously has to masquerade packets that come from 172.16.0.0/24 and leave to public with address 10.50.0.2, or machines in public do not know to send replies to B.

8

I’ve to say, it was a bit tricky for me to understand your example. Took me a while to get my head around that example too but at the end it helps really well.

Actually, the whole thread is the solution. So I’ll mark this post as a solution. I’ll come back if I still have the Docker issue, but for now it seems to work as well.

Thank you so much for all your time and effort spent for me!