First, a little background. I’m moving my home network from CentOS 7 to Rocky 9 since CentOS 7 is EOL . I’m attempting to model what my network will look like with appropriate Rocky 9 VMs running on VMware on an old Sunfire 1U I acquired a long time ago. Glad I took this approach since I’m finding learning systemd and asynchronous device initialization to be a real learning experiences. I’m keeping my existing 192.168.0.0/24 network and trying to add a 10.0.0.0/8 network for the “internal” Rocky VM with the gateway VM NATing the external traffic from the 10 net VM. I did something similar to this a few years ago when I first got the Sunfire so, yes, two layers of NAT work.
The problem:
When I have just a single network interface on a VM, route -n looks like:
[root@rocky9mate system]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.0.1 0.0.0.0 UG 100 0 0 ens192
192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens192
When I configure a second network on what will become the gateway, route -n returns:
[root@rocky9 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.0.1 0.0.0.0 UG 100 0 0 ens192
0.0.0.0 192.168.0.1 0.0.0.0 UG 101 0 0 ens224
10.0.0.0 0.0.0.0 255.255.255.0 U 101 0 0 ens224
192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens192
192.168.0.1 0.0.0.0 255.255.255.255 UH 101 0 0 ens224
NM has added two incorrect routes that show up as the second line and last line of the route output. Line two is incorrect (wrong NIC) but never fires. OTH, the last line completely disables traffic since it puts my real gateway on a host route to the wrong NIC. Deleting that route entry gets everything working. How do I prevent NM from adding this route (and the other incorrect route, too)?
Thanks! The nmcli incantation let me know that I had incorrectly specified my 192.168.0.0 network gateway as the VM gateway. I corrected that to the IP for the gateway VM and the spurious routes are now gone. I even understand why NM thought it needed to add those routes :D. Old habits die hard so I had just pounded in my usual gateway without thinking.
I like the old route -n for posting since it puts everything in nice neat columns. Works better when I can specify a fixed width font though. I know CentOS 7 isn’t quite EOL but I’d like to be happily up on R9 before that happens.
I thought so. Seeing two default routes, one for each interface, smelled like there is a gateway set for each interface. However, the idea of default route is that it is the one default that is used if there is no more explicit route to destination. One per machine, not per interface.
Similarly, in the initscript config (that CentOS 7 still has, although it already defaults to NM), the gateway was not set in ifcfg-* but in /etc/sysconfig/network.
The NM handles “multiple defaults” with metrics. You really can have gateways on each link-local subnet, but only one is used. If that interface goes down, then the next (by metric) becomes the default route, rather than loss of route to “outside”.
Your VM is the gateway for other machines in the 10.0.0.0/24 and obviously does not have (other) gateway in that subnet.
If one does have multiple subnets and each of them do have routers to “outside”, then one needs policy routing (aka source-based routing) with rules (seen with ip ru) and additional routing tables in order to get the machine reply via interface that the replied for packet did come in from.
Up and working with the “gateway vm” natting traffic from the internal vm and passing it correctly to my real network gateway which nats the outbound stuff as before. Packets get un-natted on return and end up back at the internal vm. A post from iwalker on this thread Firewalld/NetworkManager Internet Routing Not working in Rocky Linux 9.x - #11 by l2g was how I got nat working.
Now, just need to get dhcp and dns working with this arrangement. Not sure vmware is isolating the internal network correctly since I keep getting 192.168.0.0/24 addresses on my “internal” vm. Both 10 net virtual nics are plugged into an isolated virtual switch so this looks like a vmware issue.
One machine, “Router”, is member of both. Other machines, “home” are member of LAN
Another machine on the WAN, “ISP”, acts as gateway to beyond WAN
ISP has DHCP server (for WAN) and hands Router its WAN-network config. ISP is the “gateway” for Router
Router has DHCP server (for LAN) and hands home their network config. Router is the “gateway” for home
Router does route traffic between LAN and WAN
The LAN IPv4 addresses are private, not to be seen outside of LAN
Router does masquerade/sNAT traffic from LAN to WAN
Router does filter traffic between LAN and WAN: LAN can go out, but only valid replies from WAN will get in
The support for policy objects (rules for traffic between zones) was added in very recent EL8/9 FirewallD. I have tried it once and failed. I rather wrote explicit ruleset for nftables.service than cope with FirewallD; a router does have rather static network config, so does not need “fancy dynamics”. The NAT-rules are also created by the firewall config.
Examples: Nftables/Examples - Gentoo Wiki
The the default name resolver (in glibc) has limitations. The NetworkManager.service can have dns=dnsmasq in its config. That makes NM start an instance of dnsmasq with the network and to write nameserver 127.0.0.1 into /etc/resolv.conf.
It is possible to feed additional config for that dnsmasq process. For example, make it listen on the LAN interface and act as DHCP and DNS server for the “home” machines. (Optionally as TFTP server too, to allow PXEboot.) I have that on some subnets.
The NAT set up I mentioned in the other thread was a piece of cake to set up. It does involve setting up policy rules between zones but the rules are very straight forward and (what I like) sort of self documenting.
Reminder: my set up is intentionally overly complicated since I want to model what my existing 192.168 network (CentOS 7 with miscellaneous other boxes and gadgets) will look like with a Rocky 9 network running on vms inside of the 192.168 network. I went with a 10 network to make sure that I wasn’t getting any “leakage” from the working 192.168 net onto my model future net (e.g., like I may be seeing for dhcp). So far I’ve tripped over how to convert my old xinetd services to systemd, learning how to write udev rules so I get consistent device names for my VTL and the above new way to do NAT. Much better than my previous method of just blasting in the new OS and fixing whatever broke from the update .
.