First encountered on EL8.10, since then found on 9.6. I’m unable to verify on 10.0 at this time due to Customer contraints. There is no sign of delay, renegotiate, drop, frame, hist, etc, issues with network circuits. All switches upstream of failure cases are managed and report all circuits linking and activating successfully with no reneg or retran.
Hosts are all multihomed servers running 8.10 in the process of migrating to 9. All servers have at least one eth and one ib NM conns, ipv4.method=auto/dhcp, ipv6.method=ignore, with a fixed dns addr, these all work as expected. Some servers have a secondary eth with the same conn elements except ipv4 addr and gate are static. All statically addressed eth are secondary interrfaces, all primaries are the first eth and all hostnames map to their primary conn addr’s resolved name per upstream dhcpd. All primary eth and ib place in a custom non-default firewalld zone “internal”. All secondary eth places in a custom non-default firewalld zone “external”. When conn.autoconnect=true is set on a profile with ipv4.method=manual NM updates autoconnect to false. The same does not happen on dhcp addressed profiles.
On first boot after imaging, when NM has no saved prior state in /run/NetworkManager/…, all of the statically addr’d secondary eth conns succeed through activation but none end with the interface in UP state with addrs attached. NM logs show the same successful transitions through activation with only the expected exception: static conns on first up with no cached state do not match/guess an associated /etc/NetworkManger/… profile, dhcp conns do and thus get an additional “guessing” line in the logs (also because at first boot the system image has just been constructed and there are no conn.uid=).
Manual ‘nmcli con up’ raises the interface as expected and with no unusual logs. Once up for the first time, NM can be stopped and the forced conn profile write into /run/NetworkManager/… is byte-identical to the profile in /etc/NetworkManager/… Subseqent NM starts/restarts/etc pick up the /run/… copy and all behaves as expected. Before manually up’ing the connection firewalld reports as expected: primary eth and ib interfaces are in the “internal” zone and the “external” zone has no associated interfaces. All secondary eth with static addrs, after bein manually up’d, are correctly placed in the “external” zone. All transitions (other than reboot, ie, link/port/act/etc changes forced on the associated circuit) result in normal NM response and recovery, this all works fine once the static profile is manually up’d the first time.
Things I’ve tried:
- adding/removing conn.autoconnect=true
- changing NM conf plugins from none to keyfile
- full NM logging (trace/all)
- added/removed trailing ‘;’ to dns-associated fields (because ‘nmcli con mod’ does this, other times it doesn’t)
- switching ipv6 between method=ignore and addr-gen-mode=stable-privacy, method=auto
- swapping circuits and config between primary and secondary PHYs (issue moves to whichever interface gets the static addr)
- adding uuids to the provisioning (sysimage construction with ‘uuidgen -s -n {ns} -N {hostname-with-nic}’ for predictability across system image builds)
I’ve also verified that Dracut NM is not leaking config again into the main boot. I’ve tried force clearing NM /run cache both after full boot and during dracut::pre-pivot per https://bugzilla.redhat.com/show_bug.cgi?id=2093050; we’ve been bitten multiple times by this perpetually-shave-resistent yak. While the “leaking” issue is present in 8.9 and 9.2 (stock, before updates) it appears to be fixed otherwise, and anyway it’s mitigation does not impact this issue in any way.
NetworkManager.conf:
[main]
#plugins=ifcfg-rh
[logging]
level=TRACE
domains=ALL
eth0.nmconnection (compute104 sample):
[connection]
id=eth0
type=ethernet
interface-name=eth0
zone=internal[ethernet]
[ipv4]
dns=10.1.0.1
ignore-auto-dns=true
may-fail=false
method=auto[ipv6]
addr-gen-mode=stable-privacy
method=auto[proxy]
ib0.nmconnection (compute104 sample):
[connection]
id=ib0
type=infiniband
interface-name=ib0
zone=internal[infiniband]
transport-mode=datagram[ipv4]
dns=10.2.0.1
ignore-auto-dns=true
may-fail=false
method=auto[ipv6]
addr-gen-mode=stable-privacy
method=auto[proxy]
eth1.nmconnection (sample):
[connection]
id=eth1
type=ethernet
interface-name=eth1
zone=external[ethernet]
[ipv4]
addresses=REDACTED
gateway=REDACTED
dns=REDACTED;REDACTED
dns-search=REDACTED
may-fail=false
method=manual[ipv6]
method=ignore[proxy]
NM startup logging:
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.5649] NetworkManager (version 1.40.16-19.el8_10) is starting… (after a restart, boot:59ca1e35-3ede-4f98-bb6f-704c1876eb41)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.5650] Read config: /etc/NetworkManager/NetworkManager.conf
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.5649] NetworkManager (version 1.40.16-19.el8_10) is starting… (after a restart, boot:59ca1e35-3ede-4f98-bb6f-704c1876eb41)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.5650] Read config: /etc/NetworkManager/NetworkManager.conf
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.5668] bus-manager: acquired D-Bus service “org.freedesktop.NetworkManager”
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.5703] manager[0x55d296a4a020]: monitoring kernel firmware directory ‘/lib/firmware’.
Jun 24 22:35:57 redacted dbus-daemon[1596]: [system] Activating via systemd: service name=‘org.freedesktop.hostname1’ unit=‘dbus-org.freedesktop.hostname1.service’ requested by ‘:1.20’ (uid=0 pid=2236 comm=“/usr/sbin/NetworkManager --no-daemon " label=“system_u:system_r:kernel_t:s0”)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.5668] bus-manager: acquired D-Bus service “org.freedesktop.NetworkManager”
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.5703] manager[0x55d296a4a020]: monitoring kernel firmware directory ‘/lib/firmware’.
Jun 24 22:35:57 redacted-ib0 dbus-daemon[1596]: [system] Activating via systemd: service name=‘org.freedesktop.hostname1’ unit=‘dbus-org.freedesktop.hostname1.service’ requested by ‘:1.20’ (uid=0 pid=2236 comm=”/usr/sbin/NetworkManager --no-daemon " label=“system_u:system_r:kernel_t:s0”)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7790] hostname: hostname: using hostnamed
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7791] hostname: static hostname changed from (none) to “redacted”
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7799] dns-mgr: init: dns=default,systemd-resolved rc-manager=symlink
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7823] manager: rfkill: Wi-Fi enabled by radio killswitch; enabled by state file
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7790] hostname: hostname: using hostnamed
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7791] hostname: static hostname changed from (none) to “redacted”
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7799] dns-mgr: init: dns=default,systemd-resolved rc-manager=symlink
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7823] manager: rfkill: Wi-Fi enabled by radio killswitch; enabled by state file
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7824] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7825] manager: Networking is enabled by state file
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7840] settings: Loaded settings plugin: ifcfg-rh (“/usr/lib64/NetworkManager/1.40.16-19.el8_10/libnm-settings-plugin-ifcfg-rh.so”)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7841] settings: Loaded settings plugin: keyfile (internal)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7824] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7825] manager: Networking is enabled by state file
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7840] settings: Loaded settings plugin: ifcfg-rh (“/usr/lib64/NetworkManager/1.40.16-19.el8_10/libnm-settings-plugin-ifcfg-rh.so”)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7841] settings: Loaded settings plugin: keyfile (internal)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7873] dhcp: init: Using DHCP client ‘internal’
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7874] device (lo): carrier: link connected
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7875] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7884] device (eth0): carrier: link connected
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7887] manager: (eth0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7891] manager: (eth0): assume: will attempt to assume matching connection ‘eth0’ (7ba00b1d-8cdd-30da-91ad-bb83ed4f7474) (indicated)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7892] device (eth0): state change: unmanaged → unavailable (reason ‘connection-assumed’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7896] device (eth0): state change: unavailable → disconnected (reason ‘connection-assumed’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7903] device (eth0): Activation: starting connection ‘eth0’ (7ba00b1d-8cdd-30da-91ad-bb83ed4f7474)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7907] device (eth1): carrier: link connected
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7911] manager: (eth1): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7915] device (eth1): state change: unmanaged → unavailable (reason ‘connection-assumed’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7918] device (eth1): state change: unavailable → disconnected (reason ‘connection-assumed’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7923] device (eth1): Activation: starting connection ‘eth1’ (0c33a1d2-99bc-439e-80d4-4a84f2a43659)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7928] manager: (ib0): new InfiniBand device (/org/freedesktop/NetworkManager/Devices/4)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.7931] device (ib0): state change: unmanaged → unavailable (reason ‘managed’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8003] device (eth0): state change: disconnected → prepare (reason ‘none’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8006] device (eth0): state change: prepare → config (reason ‘none’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8008] device (eth1): state change: disconnected → prepare (reason ‘none’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8011] device (eth1): state change: prepare → config (reason ‘none’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8012] device (eth1): state change: config → ip-config (reason ‘none’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8015] device (eth1): state change: ip-config → ip-check (reason ‘none’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8040] device (eth1): state change: ip-check → secondaries (reason ‘none’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8042] device (eth1): state change: secondaries → activated (reason ‘none’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8044] manager: NetworkManager state is now CONNECTED_LOCAL
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8047] device (eth1): Activation: successful, device activated.
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8052] device (eth0): state change: config → ip-config (reason ‘none’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8055] dhcp4 (eth0): activation: beginning transaction (timeout in 45 seconds)
Jun 24 22:35:57 redacted dbus-daemon[1596]: [system] Activating via systemd: service name=‘org.freedesktop.resolve1’ unit=‘dbus-org.freedesktop.resolve1.service’ requested by ‘:1.20’ (uid=0 pid=2236 comm=“/usr/sbin/NetworkManager --no-daemon " label=“system_u:system_r:kernel_t:s0”)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8092] dhcp4 (eth0): state changed new lease, address=10.1.0.5
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8094] policy: set ‘eth0’ (eth0) as default for IPv4 routing and DNS
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8153] device (eth0): state change: ip-config → ip-check (reason ‘none’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8184] device (eth0): state change: ip-check → secondaries (reason ‘none’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8186] device (eth0): state change: secondaries → activated (reason ‘none’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8188] manager: NetworkManager state is now CONNECTED_SITE
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8191] device (eth0): Activation: successful, device activated.
Jun 24 22:35:57 redacted NetworkManager[2236]: [1750818957.8193] manager: NetworkManager state is now CONNECTED_GLOBAL
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7873] dhcp: init: Using DHCP client ‘internal’
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7874] device (lo): carrier: link connected
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7875] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7884] device (eth0): carrier: link connected
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7887] manager: (eth0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7891] manager: (eth0): assume: will attempt to assume matching connection ‘eth0’ (7ba00b1d-8cdd-30da-91ad-bb83ed4f7474) (indicated)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7892] device (eth0): state change: unmanaged → unavailable (reason ‘connection-assumed’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7896] device (eth0): state change: unavailable → disconnected (reason ‘connection-assumed’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7903] device (eth0): Activation: starting connection ‘eth0’ (7ba00b1d-8cdd-30da-91ad-bb83ed4f7474)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7907] device (eth1): carrier: link connected
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7911] manager: (eth1): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7915] device (eth1): state change: unmanaged → unavailable (reason ‘connection-assumed’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7918] device (eth1): state change: unavailable → disconnected (reason ‘connection-assumed’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7923] device (eth1): Activation: starting connection ‘eth1’ (0c33a1d2-99bc-439e-80d4-4a84f2a43659)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7928] manager: (ib0): new InfiniBand device (/org/freedesktop/NetworkManager/Devices/4)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.7931] device (ib0): state change: unmanaged → unavailable (reason ‘managed’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8003] device (eth0): state change: disconnected → prepare (reason ‘none’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8006] device (eth0): state change: prepare → config (reason ‘none’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8008] device (eth1): state change: disconnected → prepare (reason ‘none’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8011] device (eth1): state change: prepare → config (reason ‘none’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8012] device (eth1): state change: config → ip-config (reason ‘none’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8015] device (eth1): state change: ip-config → ip-check (reason ‘none’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8040] device (eth1): state change: ip-check → secondaries (reason ‘none’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8042] device (eth1): state change: secondaries → activated (reason ‘none’, sys-iface-state: ‘external’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8044] manager: NetworkManager state is now CONNECTED_LOCAL
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8047] device (eth1): Activation: successful, device activated.
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8052] device (eth0): state change: config → ip-config (reason ‘none’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8055] dhcp4 (eth0): activation: beginning transaction (timeout in 45 seconds)
Jun 24 22:35:57 redacted-ib0 dbus-daemon[1596]: [system] Activating via systemd: service name=‘org.freedesktop.resolve1’ unit=‘dbus-org.freedesktop.resolve1.service’ requested by ‘:1.20’ (uid=0 pid=2236 comm=”/usr/sbin/NetworkManager --no-daemon " label=“system_u:system_r:kernel_t:s0”)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8092] dhcp4 (eth0): state changed new lease, address=10.1.0.5
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8094] policy: set ‘eth0’ (eth0) as default for IPv4 routing and DNS
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8153] device (eth0): state change: ip-config → ip-check (reason ‘none’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8184] device (eth0): state change: ip-check → secondaries (reason ‘none’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8186] device (eth0): state change: secondaries → activated (reason ‘none’, sys-iface-state: ‘assume’)
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8188] manager: NetworkManager state is now CONNECTED_SITE
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8191] device (eth0): Activation: successful, device activated.
Jun 24 22:35:57 redacted-ib0 NetworkManager[2236]: [1750818957.8193] manager: NetworkManager state is now CONNECTED_GLOBAL
Jun 24 22:36:03 redacted-ib0 NetworkManager[2236]: [1750818963.7997] manager: startup complete