Networking issue

Rocky Linux Tech Help
1

Hi,
I’m supposed to know what I’m doing, yet I can’t explain what I’m missing or doing wrong.

I have this setup where Rocky Linux 8 is my file server, and has a VM running an old RHEL5 that serves all the other stuff I’ve been configuring for decades and don’t want to do again.
This machine has 2 ETH interfaces, and 2 virtual bridges. Each interface belongs to a bridge, and the same occur in the VM. One faces inwards, the other faces a fiber router with built-in firewalling.
The VM gets its IP from the fiber router (DHCP) and runs a masquerading firewall, so it NATs all traffic from the other interface. This includes my whole workplace and home, and the Rocky Linux host.

The outside facing bridge has had a dummy address so far, as it has not been in use.
Recently I requested a public IP address, and so had to assign a specific address to the outside facing bridge in Rocky Linux. The fiber router identifies my machine and forwards all incoming traffic to my MAC and IP, in fact I’m able to see this traffic with a sniffer. So far, so good. I can ping the router and get a response; traceroute shows one hop

Of course I want to do something more than sniff traffic, but unfortunately I’m unable to get anything out of the bridge.
Both inward and outward bridges are configured the same. My Rocky can be reached from other machines on the internal LAN. On the outside facing network, things are different.

So, the outside facing network has two physical machines: the fiber router and the Rocky. The Rocky ETH belongs to a bridge, and the VM virtual eth has been configured to be part of this bridge.
Firewalld is disabled, verified not running
iptables and nft show empty tables and default accept policies
selinux disabled at boot

Yet, if I open a socket listening on an UDP port where I see lots of messages on Wireshark, I get no traffic.
I can ping the fiber router and get a response
I can send myself traffic to the bridge address, and I see it back.

Again, both bridges have the same config and everything works OK on the internal bridge.

                    __________________                   ________________________
                   |     virbr0       |                 |    virbr1              |
                   |    192.168.69.3  |                 |  "DMZ", public address |
                   |                  |                 |                        |
Internal LAN ------| enp2s7                services                       enp2s6 |---------- Fiber router --- Internet
                   |                  |                 |                        |
                   |                  |                 |                        |
                   |    ------------------ VM -------------------------          |
                   |   |              |                 |              |         |
                   |   |  eth0          services      FW         eth1  |         |
                   |   | 192.168.69.2 |                 |        DHCP  |         |
                   |   | 192.168.69.1 |                 |              |         |
                   |    ------------------------------------------------         |
                   |__________________|                 |________________________|


Routing logical flow


Internal LAN ------ enp2s7 --- eth0   FW  eth1 --- enp2s6
                        |
Rocky services  virbr0 -
2

First, a bridge is a switch. The RHEL(eth1)---(vnet0)virbr1(enp2s6)---fiberRTR
is same as all physical RHEL(eth1)---(p1)switch(p0)---fiberRTR, with unmanaged switch.

On Rocky 8 bridged traffic does not visit filter. You can see comment in dmesg about it:

bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.

Two notes:

  • Since bridged traffic is not filtered (unless you explicitly do it), firewall of the host does not affect it
  • The iptables is a wrapper/translator to nftables. If you run it just to read, it will generate the (empty) “iptables default chains”. When you boot without running firewall or iptables, you will get:
# nft list ruleset
#

(Not that it matters.)

Unmanaged switches do not have IP address. They don’t need one. There was never a need for a dummy address on that bridge.

The only reason you have IP address on the virbr0 is for you to be able to ssh into the Rocky from home LAN.

This is where the plot thickens. Now you have a machine with two interfaces, each with IP address. The obvious question is, where does the traffic go? The routing.
Plain nmcli does include that info, but ip ro is more to the point.

There should be three rules. One link-local for each subnet (inside and outside), and the default route. The default has presumably been via 192.168.69.1
Should it now be via fiberRTR? You can access all of home LAN (192.168.69.0/24) with the link-local route.

Consider this: A packet arrives for Rocky on the virbr1 address from forums.rockylinx.org. A reply should be sent. If the default route is via 192.168.69.1, then the reply should leave from virbr0 to RHEL.
If that would happen, then forums.rockylinx.org would receive reply from public address of the RHEL.
More likely, no reply is sent at all, because the “from” in the reply has address of virbr1, and virbr0 should send with “from=192.168.69.3”.

If you have services that require the Rocky to talk via RHEL, then for those you would need to add static routes that make that happen.

Furthermore, if you want to send something from home LAN via 192.168.69.3 to internet, then the Rocky should route:

# cat /etc/sysctl.d/my.conf 
net.ipv4.ip_forward = 1

and NAT.

I’d recommend that you manage firewall in the Rocky with nftables.service. To configure it:

# cat /etc/sysconfig/nftables.conf
include "/etc/nftables/my.nft"

There are some examples in /etc/nftables/ for what to put into /etc/nftables/my.nft.

3

@jlehtone Jukka, “I’m supposed to know what I’m doing”. To be pedantic, a bridge is not a switch, an Ethernet switch is a bridge. Joke aside, thanks for your interest.

I haven’t configured routing yet, I’m aware of the loop, however, how come I can’t see incoming UDP traffic ? There is no return path, I see UDP datagrams with proper IP and proper MAC at the bridge interface with a sniffer, yet if I listen on that IP and port and print incoming traffic on the console, no datagrams are received. There is no reason for something to travel back to its origin if nothing can be received from the origin in the first place.
This thing behaves like there is a firewall or packet filter somewhere, yet I disabled those I know exist, and you just confirmed the bridge does not do frame filtering (seen that message, but “something” has to be blocking traffic somewhere).
I don’t need a firewall right now, I just need to see the traffic and then I’ll configure firewall and routing. And no, Rocky will not route from other interfaces, that’s the VM task and has been happily doing it for 20 years, starting as a physical machine and then being “assimilated” by first a Centos 7, now a Rocky 8.

For the record, the bridge has an IP because the physical interface can’t have it if both VM and Rocky need to be able to see each other without special support from the physical network (that is the reason for the bridge to exist in this setup).
I could replace the outside facing bridge, I don’t actually need both Rocky and VM to see each other there… I’ll try that if I can’t figure out why this is not working

Thanks.

4

My bad.

A rephrase, so I’m sure I’m on the same page:

  • Listen on enp2s7 or virbr0 and traffic in home LAN is visible
  • 192.168.69.3 receives traffic from home LAN
  • Listen on enp2s6 or virbr1 and traffic in fiber subnet is visible
  • IP on virbr1 does get ping replies, but no UDP

Can you log into the fiber router and test connect with something from it, or from “outside” (through the fiber router)?

If you shut down the VM, then the virbr1 should behave the same as now. Does it?

5

Thanks Jukka,

1 and 2: yes, internal stuff works like a charm, routing works too, Rocky updates (for example) go through the VM like any other machine in the home LAN. virbr0 also has some subinterfaces serving VLANs, everything works fine.
3 and 4:
The fiber router is closed, can’t toy with it.
I can ping the fiber router and get a reply (only if I assign my IP to the bridge ifc)
I can see incoming traffic with Wireshark on enp2s6 or virbr1. However, listening on the interface, I can’t get it.
UDP is easy because I have some machine somewhere logging to my pubic IP, so traffic is there and I don’t need to answer.
Here’s something I forgot to add. I can also nc -u to virbr1 IP, I did that to test the listener was actually listening there, I used an old program of mine and also nc -u -l ip port. It’s like there’s some well-intentioned entity blocking outside traffic…
The fiber router’s internal IP is a private one, of course, 192.168.1.1; incoming UDP traffic has public addresses, of course.

I didn’t think on shutting down the VM… I’ll try when I won’t disturb the usual activities.

I will try to connect another machine to the enp2s6 - fiber LAN (192.168.1.0) and see if I can get through to Rocky. The fiber router has some extra ports and looks like they are switched.

PS: what I was actually doing to see those UDP remote logs was just run tshark and pipe to a decoder, a bit of extra noise but I could do my job. A listener at the UDP port would have been great.

6

Update:

  • I can see hosts on the outside network.
  • Added a static route to the logging remote host through virbr1 and now I can get the logs…

This is weird…

So, I’m able to do what I need to do. Fine. However, I’d like to understand what is going on here. I can explain it if it was TCP, but UDP ???

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.