Crowdstrike - freezing RockyLinux After 9.4 upgrade

[antoine]

Crowdstrike users may see their servers freezing after upgrading to RockyLinux 9.4.
Crowdstrike support is aware of that problem (due to a kernel bug same as https://supportportal.crowdstrike.com/s/article/Linux-Sensor-in-user-mode-may-fail-on-specific-6-x-kernel-versions).

[helkin]

Thanks for the info, that might explain the issue with my Rocky VM. What would you look for in the logs to make sure this is whats happening?

For me if I try to boot with the new kernel it just freezes after this:

systemd[1]: Starting Hostname Service…
kernel: Warning: Unmaintained driver is detected: ip_set
And I never get to the login screen.

But booting from old kernel works fine. But I don’t see any warnings or errors or anything in Journalctl so not sure what I need to be looking for.

I think this crowdstrike KB article might be more specific to Rocky: https://supportportal.crowdstrike.com/s/article/Tech-Alert-Kernel-crash-may-occur-for-Linux-sensor-running-in-user-mode-on-hosts-running-RHEL-9-4

[antoine]

Hello Helkin,

In fact, I couldn’t find any log concerning this kernel panic. I had to uninstall crowdstrike to confirm that it was indeed causing the freeze.

As for ip_set, it’s not the cause. You can blacklist it in the kernel to prevent it from being loaded, but that won’t solve your problem (I’ve tested this too). In my case, it’s loaded by docker, which still uses nft.

Crowdstrike won’t be supporting rocky 9.4 for a while yet.

To uninstall crowdstrike, a dnf won’t work. You’ll have to use rpm -e (at your own risk).

[iwalker]

Someone on Rocky mattermost, said this helped them:

sudo /opt/CrowdStrike/falconctl -s --backend=kernel

apparently this is documented in a RH KB article.

[antoine]

This KB: Kernel panic observed after booting 5.14.0-427.13.1.el9_4.x86_64 by falcon-sensor process. - Red Hat Customer Portal
Thx Ian reporting this here.

I tried the solution.
First I rebooted with the previous kernel (9.3) and then I switched to the kernel backend:

/opt/CrowdStrike/falconctl -s --backend=kernel

I can confirm it works after rebooting to the latest kernel (and it’s a better solution than to uninstall).

[iwalker]

Thx @antoine I marked your post as the solution as it was more verbose than mine, and it also confirms it works

[helkin]

Thanks both. I have also applied this setting and can confirm that my server is working and Falcon can see the host online (although only in reduced functionality mode).
It’s also possible to apply the setting using Ansible by adding this to your playbook:

  - role: crowdstrike.falcon.falcon_configure
    vars:
      falcon_cid: "{{ cs_cid }}"
      falcon_backend: kernel
    when: ansible_pkg_mgr == 'dnf'

[antoine]

A new kernel will be available soon that correct the ‘bug’.
See https://access.redhat.com/errata/RHSA-2024:3306