There is a version mismatch in kmod-kvdo module and it does not load in Rocky 9.6.
Kernel RPM version is kernel-5.14.0-570.25.1.el9_6.x86_64
kmod-kvdo RPM version is kmod-kvdo-8.2.5.10-161.el9.x86_64
Anyone having this problem?
I have a machine which is still not upgraded, (Rocky 9.5+) and with these versions, everything is fine:
kernel-5.14.0-503.40.1.el9_5.x86_64
kmod-kvdo-8.2.4.15-141.el9_5.x86_64
So I guess it’s just a matter of recompiling kmod-kvdo RPM…?
Same in RHEL9 as well. Sounds like a bug to be fixed upstream.
That said VDO in that form has been deprecated anyway, and is integrated with LVM.
From the 9.6 release notes: 9.6 Release Notes | Red Hat Enterprise Linux | 9 | Red Hat Documentation
and this RH article: How to configure a storage device to use VDO deduplication and compression in RHEL 9? - Red Hat Customer Portal
bearing that in mind, you probably want to migrate away from the way you are currently using it.
I already read those docs and I have configured my VDO with LVM and not VDO tools. But the failing RPM is the one that provides the KVDO kernel module, which (I believe) is still needed anyway.
Here is the line I used for creating my VDO LV:
lvcreate --type vdo --name lv_vm -l 100%FREE --virtualsize 5T --compression y --deduplication y vg_vdo
(the vg_vdo is a volume group dedicated to VDO volumes)
Running that command in a stock Rocky 9.6 installation fails saying that the KVDO kernel module cannot be located. And if it is installed through the kmod-kvdo RPM, it still fails because the module contained in the RPM fails the checks done against the kernel symbol versions.
if I try to load the KVDO module after installing kmod-kvdo RPM, I get this error:
[root@leovm01 ~]# modprobe kvdo
modprobe: FATAL: Module kvdo not found in directory /lib/modules/5.14.0-570.25.1.el9_6.x86_64
And indeed, the kvdo module cannot be found under the modules directory for that kernel.
Following the official documentation,
I get the same error:
[root@leovm01 virt-stack]# lvcreate --name VDOPoolName --size 425G vg_vdo
WARNING: xfs signature detected on /dev/vg_vdo/VDOPoolName at offset 0. Wipe it? [y/n]: y
Wiping xfs signature on /dev/vg_vdo/VDOPoolName.
Logical volume "VDOPoolName" created.
[root@leovm01 virt-stack]# vgs
VG #PV #LV #SN Attr VSize VFree
vg0 1 2 0 wz--n- 49.96g 0
vg_vdo 1 1 0 wz--n- 425.54g 556.00m
[root@leovm01 virt-stack]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root vg0 -wi-ao---- 40.00g
swap vg0 -wi-ao---- 9.96g
VDOPoolName vg_vdo -wi-a----- 425.00g
[root@leovm01 virt-stack]# lvconvert --type vdo-pool --name VDOVolumeName --virtualsize 5T vg_vdo/VDOPoolName
WARNING: Converting logical volume vg_vdo/VDOPoolName to VDO pool volume with formatting.
THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert vg_vdo/VDOPoolName? [y/n]: y
modprobe: FATAL: Module kvdo not found in directory /lib/modules/5.14.0-570.25.1.el9_6.x86_64
/usr/sbin/modprobe failed: 1
vdo: Required device-mapper target(s) not detected in your kernel.
[root@leovm01 virt-stack]#
I ran rpm ql kmod-kvdo and got this output:
$ rpm -ql kmod-kvdo
/etc/depmod.d/kvdo.conf
/lib/modules/5.14.0-570.el9.x86_64
/lib/modules/5.14.0-570.el9.x86_64/extra
/lib/modules/5.14.0-570.el9.x86_64/extra/kmod-kvdo
/lib/modules/5.14.0-570.el9.x86_64/extra/kmod-kvdo/vdo
/lib/modules/5.14.0-570.el9.x86_64/extra/kmod-kvdo/vdo/kvdo.ko
/usr/lib/.build-id
/usr/lib/.build-id/7b
/usr/lib/.build-id/7b/1b3a002a0362b39d3072606592ef94105b357a
/usr/share/doc/kmod-kvdo/greylist.txt
And then browsing /lib/modules I found this:
/lib/modules/5.14.0-570.21.1.el9_6.x86_64/weak-updates/kmod-kvdo/vdo:
total used in directory 12 available 23.9 GiB
drwxr-xr-x 2 root root 4096 Jun 15 06:44 .
drwxr-xr-x 3 root root 4096 Jun 15 06:44 ..
lrwxrwxrwx 1 root root 62 Jun 15 06:44 kvdo.ko -> /lib/modules/5.14.0-570.el9.x86_64/extra/kmod-kvdo/vdo/kvdo.ko
So it appears the kmod-kvdo driver is being installed in a generic folder under lib/modules and then being linked. It appears then that the link is nolonger being created in the 570.25.1 kernel modules directory so you may have to create the linke by hand.
I already tried that, there is an error with the kernel module, it does not load even making the links manually:
[root@leovm01 modules]# uname -a
Linux leovm01 5.14.0-570.25.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jul 7 18:09:10 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
[root@leovm01 modules]# cd 5.14.0-570.25.1.el9_6.x86_64
[root@leovm01 5.14.0-570.25.1.el9_6.x86_64]# mkdir -p extra/kmod-kvdo/vdo
[root@leovm01 5.14.0-570.25.1.el9_6.x86_64]# cd extra/kmod-kvdo/vdo
[root@leovm01 vdo]# ln -s /lib/modules/5.14.0-570.el9.x86_64/extra/kmod-kvdo/vdo/kvdo.ko .
[root@leovm01 vdo]# ls
kvdo.ko
[root@leovm01 vdo]# ls -l
total 0
lrwxrwxrwx. 1 root root 62 Jul 12 17:02 kvdo.ko -> /lib/modules/5.14.0-570.el9.x86_64/extra/kmod-kvdo/vdo/kvdo.ko
[root@leovm01 vdo]# depmod -a
[root@leovm01 vdo]# modprobe kvdo
modprobe: ERROR: could not insert 'kvdo': Invalid argument
[root@leovm01 vdo]# dmesg|tail
[10028.414835] kvdo: disagrees about version of symbol dm_disk
[10028.414836] kvdo: Unknown symbol dm_disk (err -22)
[10028.414843] kvdo: disagrees about version of symbol dm_register_target
[10028.414844] kvdo: Unknown symbol dm_register_target (err -22)
[10028.414857] kvdo: disagrees about version of symbol dm_table_get_mode
[10028.414858] kvdo: Unknown symbol dm_table_get_mode (err -22)
[10028.414863] kvdo: disagrees about version of symbol dm_table_get_md
[10028.414864] kvdo: Unknown symbol dm_table_get_md (err -22)
[10028.414870] kvdo: disagrees about version of symbol dm_get_device
[10028.414871] kvdo: Unknown symbol dm_get_device (err -22)
[root@leovm01 vdo]#
As it can be seen in the dmesg output, there are symbol version mismatches between the kvdo module and the running kernel.
The /etc/depmod.d/kvdo.conf file (whcih is also installed by kmod-kvdo RPM) contains the following:
override kvdo 5.14.0-570.el9.* weak-updates/kmod-kvdo/vdo
…which appears to say that every kernel must do the weak-updates links under its own modules directory, pointing to the generic ones you mentioned. So yes, it should be behaving like you said, but it isn’t.
Most probably there was some minor ABI deviation in the new kernel, and the old kmod-kvdo module is not compqtible with it.
I believe this should be reported upstream. As of now, VDO is not working at all in ROCKY/RHEL 9.6 with the current kernel.
Yep, I got that as well, even on RHEL9, that’s why it’s an upstream bug, since Rocky 9 is based on RHEL9. So it would require fixing in RHEL9 first, upstream as mentioned in my previous post.
Is there any way this can be reported upstream? It is a breaking change, and for a basic storage configuration, no less…
Usually on https://issues.redhat.com would be the place for it to be opened upstream.
Luckily Rocky 10 doesn’t have this problem. A potential option to migrate to and use vdo.
Good point, but Rocky 9.5+ did not have it either and now it does. And VDO seems to work exactly the same way in Rocky 10. So the same could happen there I guess. This is a critical bug in upstream hopefully the solve it soon.
Oh, and of course VDO is supported in either of them, according to docs.
No it won’t happen in RHEL10 or Rocky10 because there is no kmod-kvdo. The implementation is different and the kernel already has the vdo module in EL10 by default.
But yes, it should be fixed in RHEL9 whilst it is still using the old method with the kmod, even if it has been marked as deprecated (due to the implementation changing in EL10).
OK, I checked some references, and you are right, sorry. KVDO module is integrated in dm-vdo in RHEL10 and does need kmod-kvdo. This is indeed a good reason to migrate to Rocky10; pity that Hetzner still does not support Rocky10 with its image install script.
I’ll ask for it 
Sorry again for my mistake 
Oh strange, when I go to create a new VPS, they have Rocky 10 as an option. Not sure what the image install script is, not something I used.
For dedicated servers, not VPSs…
Ah yeah, not used dedicate there yet, but been tempted to get one
Amazing beasts, and dirt cheap…
I’m just installing one today (EX44) with 20 cores, 64 GB RAM, 512 GB dual NVMe (at 4 GB/sec transfer rate!!) for just 39 EUR/mo
Also in my experience, tech support is top notch.
Been with Hetzner for 15 years now.
I rebuild kmod-kvdo.rpm and with it I manage to mount all my devices.
$ dnf download --source kmod-kvdo
$ rpm -ivh kmod-kvdo-8.2.5.10-161.el9.src.rpm
$ sudo dnf install rpm-build
$ sudo dnf install libuuid-devel
$ cd rpmbuild/SPECS
$ rpmbuild -ba kmod-kvdo.spec
$ cd ../RPMS/x86_64/
$ sudo dnf reinstall ./kmod-kvdo-8.2.5.10-161.el9.x86_64.rpm
to mount the broken vg without reboot you have to activate the vg group