Intel 225 problems in Rocky 9.2 - ASUS B550

curto · November 1, 2023, 2:51am

Guys,

Have recently upgraded my motherboard from an ASUS 450 to an ASUS 550.

Took the cards i had in the old box - Intel Dual Port PCIe Gigabit ethernet and Intel Dual Port 10GBe and moved them to the new board.

These cards had both been working for the last 2 years under firstly Centos 8, then for the last 3 months under Rocky 9.2

When i boot now - the 1GB card is not recognized - no listing in lspci and nothing in dmesg.

I thought i would therefore try the onboard 2.5GB port(s) which are Intel 225 Rev 3 - but see the problems that are listed all over the internet with frequent random dropouts etc

It appears that when the igc driver is loading for the 225 ports it is conflicting as i can not activate the other Intel cards. The 10GB is recognised but i can not get it to load its VLAN config and attach to my switch at all.

I would like to stop the 225 drivers loading and do some experimentation with the 1GB dual port and the 10GB card

Can anyone advise is there a way to do this in Rocky ?

Craig

jlehtone · November 1, 2023, 7:39am

Reseat the card?
Are some PCIe-slots and M.2 connectors mutually exclusive on that board? If there is NVMe “stealing” shared lanes …

You probably can disable the NIC in BIOS. No device listed → nothing to load drivers for.

curto · November 1, 2023, 7:48am

I have tried a different slot and a different card just in case that was the issue (sorry should have mentioned that in the original Post)

I did not even think about the BIOS option - will try that tomorrow and report back

Yes i believe the NVME share lanes but i do not think they take them over completely - not using them so will look to see if i can disable them also

Craig

curto · November 1, 2023, 11:57pm

Yep did some looking last night - the online manual says that various modes of M.2 slots and population of different x16 slots share the bandwidth with the Thunderbolt and PCIex1 slots.

What they really mean is that they disable certain slots and features !!

So i will wait for this big copy i am doing to finish and will then shutdown the machine and delve into the BIOS - will report back once done

Craig

curto · November 2, 2023, 7:32am

OK guys this now gets weirder.

I managed to disable the I225 ethernet ports in the BIOS and then installed a brand new Intel Dual port card in one of the PCieX1 slots and the board is recognized.

However i now see some more behaviour that i now realise was hanging around previously.

What i am seeing is documented in this post

github.com/systemd/systemd

[systemd >=242] LACP no longer works on NetworkManager bond interfaces

opened 02:40AM - 24 Mar 20 UTC

chenxiaolong

udev

**systemd version the issue has been seen with** / **Used distribution** * `s…ystemd-245~rc1-4.fc32.x86_64` (Fedora 32 Beta) * `systemd-243.7-1.fc31.x86_64` (Fedora 31) * Self-built systemd 242, 243, 245 **Expected behaviour you didn't see** Ever since systemd 242 (possibly around the `MACAddressPolicy` changes), LACP (802.3ad) seems to be broken when the interfaces are configured with NetworkManager. This happened somewhere between version 241 and 242, but I was unable to find the exact commit with git bisect due to some commits not building or not being bootable on my system. Setup: * Completely clean Fedora Server installation (with versions above) and no config changes beyond NetworkManager * Network Manager configuration: https://gist.github.com/chenxiaolong/8236466ddc7e75ab3fc3b2d03232ef1d * All settings are defaults created by the Fedora installer aside from picking the bond type to be LACP * `bond0` (LACP) containing physical interfaces `ens7` and `ens8` * `bridge0` has `bond0` attached * DHCP on `bridge0` * NIC: Intel X540-AT2 (ixgbe driver) w/10Gbps RJ45 * Switch: Arista DCS-7050TX-64-R with default settings + LACP configuration for two ports: https://gist.github.com/chenxiaolong/263f5bc576ec85a15d43514aef719ec5 When bringing up the interface with `nmcli con up 'Bridge connection 1'`, LACP never converges and the `bond0` link never comes up. Doing a `tcpdump` on the physical interfaces, `ens7` and `ens8`, when the system is in this state shows `packets dropped by interface` increasing by 1 every 30 seconds. I suspect LACPDUs are being dropped. On the exact same F31 or F32 beta system with no configuration files changed, if I compile and install systemd 241, everything works fine. It also does not happen if I use either of the following workarounds: 1. Create `/etc/systemd/network/98-bond-mac.link` with: ```ini [Match] Type=bond [Link] MACAddressPolicy=none ``` 2. ~Set any MAC address for `bond0` in `/etc/sysconfig/network-scripts/ifcfg-bridge0_slave_1`:~ EDIT: This works for ixgbe, but not mlx5_core (Mellanox). ```bash # This value literally works--seems to be ignored HWADDR=aa:bb:cc:dd:ee:ff ``` I have no idea why this would be impacting LACP. With or without the workarounds, `ens7`, `ens8`, and `bond0` have the same MAC address. Also, what's interesting is that once the interface comes up with one of these workarounds, they continue working even after I get rid of the workaround. If I reload the `bonding` kernel module or reboot, then it breaks again. Are my workarounds with the MAC address stuff triggering some sort of side effect in systemd or udev that's "unbreaking" LACP? **Unexpected behaviour you saw** Something related to the MAC address policy seems to cause LACPDUs to get dropped and the bond link to never come up. **Steps to reproduce the problem** 1. Set up switch for LACP on two ports 2. On the Linux side, use NetworkManager 3. Create a bond interface consisting of two physical interfaces 4. Create a bridge interface with the bond interface attached 5. (Optionally) set up DHCP on the bridge interface 6. Check `/proc/net/bonding/bond0` and note that LACP doesn't converge --- Side note: I also tested on Ubuntu 20.04 daily builds (systemd 245.2-1ubuntu1). It does _not_ have this problem, which is suspect is due to their patch: https://git.launchpad.net/ubuntu/+source/systemd/tree/debian/patches/Skip-falling-back-to-device-name-when-net_get_name-device.patch?h=applied/245.2-1ubuntu1

Essentially i am trying to setup a bond for the two x 1GB ports to a Brocade 6610 switch (i have also tried it with 2 x 10GB ports) and can not get the bond to come up and forward packets - the switch
thinks the ports are OK but i can not get anyting through them.

Essentially it appears that the Bond does some weird stuff with Mac addresses when trying to bring the links up.

I am going to try the workaround listed in the above post and see if that gets me any further.

Will report back

Craig