RL 9.3 kernel-devel rpm incomplete?

Rocky Linux Tech Help
1

Hi guys,

After upgrading from RL 9.2 to RL 9.3, I can’t build any of my (proprietary) kernel modules anymore.
The code itself is not the problem. It builds fine, BUT in the usual modpost phase, there is a tool named
resolve_btfids which normally gets build and then used to resolve exported symbols. It fails to
build due to missing files in the source tree which should be provided by the kernel-devel package.

Here is the output of the relevant tree on RL 9.3:

rpm -ql kernel-devel | grep /tools/bpf
/usr/src/kernels/5.14.0-362.8.1.el9_3.x86_64/tools/bpf
/usr/src/kernels/5.14.0-362.8.1.el9_3.x86_64/tools/bpf/Makefile
/usr/src/kernels/5.14.0-362.8.1.el9_3.x86_64/tools/bpf/bpftool
/usr/src/kernels/5.14.0-362.8.1.el9_3.x86_64/tools/bpf/bpftool/Documentation
/usr/src/kernels/5.14.0-362.8.1.el9_3.x86_64/tools/bpf/bpftool/Documentation/Makefile
/usr/src/kernels/5.14.0-362.8.1.el9_3.x86_64/tools/bpf/bpftool/Makefile
/usr/src/kernels/5.14.0-362.8.1.el9_3.x86_64/tools/bpf/resolve_btfids
/usr/src/kernels/5.14.0-362.8.1.el9_3.x86_64/tools/bpf/resolve_btfids/Build
/usr/src/kernels/5.14.0-362.8.1.el9_3.x86_64/tools/bpf/resolve_btfids/Makefile
/usr/src/kernels/5.14.0-362.8.1.el9_3.x86_64/tools/bpf/resolve_btfids/main.c
/usr/src/kernels/5.14.0-362.8.1.el9_3.x86_64/tools/bpf/runqslower
/usr/src/kernels/5.14.0-362.8.1.el9_3.x86_64/tools/bpf/runqslower/Makefile

For comparison, here is the output of the relevant tree on RL 9.2 (of course from a different kernel version):


rpm -ql kernel-devel | grep /tools/bpf
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/Makefile
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/bpftool
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/bpftool/Documentation
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/bpftool/Documentation/Makefile
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/bpftool/Makefile
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/.fixdep.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/Build
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/Makefile
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/fixdep
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/bpf_helper_defs.h
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include/bpf
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include/bpf/bpf.h
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include/bpf/bpf_core_read.h
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include/bpf/bpf_endian.h
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include/bpf/bpf_helper_defs.h
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include/bpf/bpf_helpers.h
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include/bpf/bpf_tracing.h
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include/bpf/btf.h
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include/bpf/libbpf.h
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include/bpf/libbpf_common.h
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include/bpf/libbpf_legacy.h
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include/bpf/libbpf_version.h
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include/bpf/skel_internal.h
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/include/bpf/usdt.bpf.h
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/libbpf.a
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.bpf.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.bpf_prog_linfo.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.btf.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.btf_dump.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.gen_loader.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.hashmap.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.libbpf.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.libbpf_errno.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.libbpf_probes.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.linker.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.netlink.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.nlattr.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.relo_core.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.ringbuf.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.str_error.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.strset.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libbpf/staticobjs/.usdt.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libsubcmd
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libsubcmd/.exec-cmd.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libsubcmd/.help.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libsubcmd/.pager.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libsubcmd/.parse-options.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libsubcmd/.run-command.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libsubcmd/.sigchain.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libsubcmd/.subcmd-config.o.d
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/libsubcmd/libsubcmd.a
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/main.c
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/resolve_btfids/resolve_btfids
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/runqslower
/usr/src/kernels/5.14.0-284.30.1.el9_2.x86_64/tools/bpf/runqslower/Makefile

This indicates to me, that there are a lot of files missing in kernel-devel-5.14.0-362.8.1.el9_3.x86_64

Or am I missing something?

Thanks
-Fritz

2

Examining the kernel.spec file, I could find:

* Wed Jun 28 2023 Jan Stancek <jstancek@redhat.com> [5.14.0-333.el9]
...
- spec: remove resolve_btfids from kernel-devel (Viktor Malik) [2178930]
- tools/resolve_btfids: Fix setting HOSTCFLAGS (Viktor Malik) [2178930]
- tools/resolve_btfids: Pass HOSTCFLAGS as EXTRA_CFLAGS to prepare targets (Viktor Malik) [2178930]
- tools/resolve_btfids: Tidy HOST_OVERRIDES (Viktor Malik) [2178930]
- tools/resolve_btfids: Compile resolve_btfids as host program (Viktor Malik) [2178930]
- tools/resolve_btfids: Alter how HOSTCC is forced (Viktor Malik) [2178930]
- tools/resolve_btfids: Install subcmd headers (Viktor Malik) [2178930]

Assuming, that the number in square brackets is a bugzilla ID, I went to bugzilla.redhat.com and
looked for that, but I don’t have permission to view that bug :frowning:

This begs the question about the rationale for this change. So:

Anyone with more permissions on bugzilla can explain WHY they decided to remove that?

3

Aren’t all kernel-bugs non-public?

4

@felfert you could open a bug report in rh’s bugzilla and explain your use case as generic issue and the necessity of the mentioned files …

5

What it sounds like is you need to run make prepare. I would look at these links:

6

Thanks a lot for those links. That explains it.
However I disagree with the reasoning given there:

  • After all, the kernel-devel package is not a noarch package and therefore any tools can stay in there.
  • Running make prepare in the build tree rebuilds all tools and - as a side effct - recreates the kernel’s config which is a no-go if one wants to make shure a thirdparty module is built against an unmodified kernel.
  • It also happens, that make prepare currently fails earlier in building objtool.

=> So they either should revert that patch or add the tools into another package. Removing them completely is not a valid option.
=> Will file a bug on bugzilla…

7

As an interim solution, I have disabled the BTF step in the modpost phase.
This can be easily done by NOT providing an uncompressed vmlinux in the toplevel build tree.
The build then prints an additional hint:
Skipping BTF generation for <my module source path>/<modulename>.ko due to unavailability of vmlinux

8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.