Rocky 9.5 slow network "bad gso"

Rocky Linux Tech Help
1

Hi - I just patched a couple of servers and noticed quite quickly very slow throughput on the network and these kernel messages
Nov 22 09:03:23 vuh-lb-webdev04 kernel: ens192: bad gso: type: 1, size: 1448
The network is virtio on nutanix

The kernel was 5.14.0-503.14.1.el9_5
I have put the kernel down to a previous version and its ok now.
grubby --set-default /boot/vmlinuz-5.14.0-427.24.1.el9_4.x86_64

Regards
Mark

2

You can try this command on the latest kernel we provide.

ethtool --offload ens192 rx-gro-hw off

Otherwise you’ll need to wait for upstream fixes to land in CentOS Stream 9 / RHEL 9 before they come to us.

4 Likes
3

Hello there.

Thanks for the temporary patch, it works for me too.

We detected same network issues when pulling a docker image from a local repository, but not when pulling from the docker hub.

Do you know when the official patch will be released ?

Lucas

4

I’ve had the same problem with Nutanix VMs and 9.5

The workaround with turning ‘rx-gro-hw’ off is fine for installed VMs (upgraded from 9.4) - but doesn’t help when network installing 9.5 … I’m trying to find if there is a workaround for this ?

The Redhat knowledge base article about this issue: Poor virtio-net performance and "bad gso: type: 1, size:" logged in RHEL 9.5 - Red Hat Customer Portal (requires a login) just has ‘TBA’ for when the issue will be fixed and the referenced Redhat bugs are private (probably because they are kernel bugs?)

There is no mention if the 9.5 installer will also fixed …

5

I’ve managed to find a hacky workaround to this problem when PXE installing 9.5 using the virtio_net driver - which involves adding the ethtool workaround into the PXE install initrd.img - if anyone is interested, the following script can be run to create a new/hacked PXE initrd.img from the original:

#!/bin/sh -x

initrd="$1"
[ -z "$initrd" ] && exit 1

dir="$(mktemp -d)"

cd "$dir" || exit 1

lsinitrd --unpack "$initrd"

mkdir usr/lib/systemd/system/nm-wait-online-initrd.service.d
cat << EOF > usr/lib/systemd/system/nm-wait-online-initrd.service.d/hack.conf
[Service]
ExecStartPost=/usr/bin/rx-gro-hw-off.sh
EOF

cat << "EOF" > usr/bin/rx-gro-hw-off.sh
#!/usr/bin/sh
dev="$(ip route | awk '/^default/ {print $5}')"
if ethtool -i "$dev" | grep -q virtio_net; then
   ethtool -K "$dev" rx-gro-hw off
fi
exit 0
EOF
chmod +x usr/bin/rx-gro-hw-off.sh

cp -p /usr/sbin/ethtool usr/sbin/ethtool

find .buildstamp * | cpio -c -o | xz --check=crc32 --lzma2=dict=1MiB -T0 > /tmp/new-initrd.img

cd -
# rm -rf "$dir"
exit 0

You will need to run the above as root on a 9.5 install and have the ethtool and dracut RPMS installed

Once created, replace your PXE install initrd.img with the file /tmp/new-initrd.img

1 Like
6

If you are using NetworkManager, you can do this command:

nmcli c modify enp1s0 ethtool.feature-rx-gro-hw off

This will enable workaround after reboots.

1 Like
7

CentOS Stream kernel 5.14.0-533.el9 has the fix applied - so hopefully Redhat will release an updated RHEL 9.5 kernel in the near future ?

2 Likes
8

Any update on this? I see 9.5 came with kernel 5.14.0-503.14.1 and now 5.14.0-503.15.1 is available, but the release notes don’t seem to specifically address this particular issue. If I need to wait for 5.14.0-533, is there any ETA?

9

The required changes are not in the 5.14.0-503.15.1 kernel source (from the CentOS 9 Stream fix)

The page Poor virtio-net performance and "bad gso: type: 1, size:" logged in RHEL 9.5 - Red Hat Customer Portal still has ‘TBA’ for when a fix will be available

5.14.0-533 won’t be available for RHEL/Rocky 9 - but they will (hopefully?) backport the fix to a 5.14.0-503 series kernel

1 Like
10

That page is now reporting a fix for 9.5 as: ETA - Feb 4th 2025

1 Like
11

… and today it is now reporting it is fixed in kernel-5.14.0-503.19.1.el9_5 - which is available for RHEL 9.5 - so I guess it should be available soon for Rocky 9.5

1 Like
12

Great News! Thank you for updating this article james-p!