"watchdog: BUG: soft lockup - CPU stuck" errors with USB devices

[timppu]

Rocky Linux 9.1 (tried both with the default Gnome and XFCE editions)

5.14.0-162.23.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 11 19:09:37 UTC 2023

I recently installed Rocky Linux 9.1 on my Dell laptop (dual-boot with Windows 11), and if I have any devices connected to the USB-C or USB-A ports, I frequently get e.g. the soft lockup errors. These seem to come either while booting up, or shutting down. On the USB-A port(s) I normally connect my powered USB hub to which I’ve connected the keyboard, mouse and USB audio speakers, and to USB-C I’ve connected an USB-C → DisplayPort cable for my monitor.

I’ve tested it with only the USB-C (monitor) connected, and only the USB hub (keyboard, mouse etc.) connected, and in both cases errors occur.

When the error hits, it seems to get stuck for good, ie. it just keeps repeating the same errors to the screen, never really advancing from them. I have to cold boot the laptop.

If I have no USB devices connected, like right now, no problems detected. Neither do I see anything abnormal on the Windows side, all the USB devices/hubs seem to be working just fine. I booted up also with a Linux Mint 21.1 live-USB, and didn’t see any problems there in booting up or shutting down.

During the boot up, I may see e.g. these kinds of messages on the screen:

irq 147: nobody cared (try booting with the "irqpoll" option)
Disabling IRQ #147
watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [Xorg:1754]

In the /var/log/messages, I see e.g. these kinds of errors (I presume these are during the shutdown?):

Apr 23 07:26:35 dell systemd[1]: dbus-:1.1-org.fedoraproject.SetroubleshootPrivileged@0.service: Main process exited, code=killed, status=14/ALRM
Apr 23 07:26:35 dell systemd[1]: dbus-:1.1-org.fedoraproject.SetroubleshootPrivileged@0.service: Failed with result 'signal'.
Apr 23 07:26:37 dell systemd[1]: dbus-:1.1-org.fedoraproject.Setroubleshootd@0.service: Main process exited, code=killed, status=14/ALRM
Apr 23 07:26:37 dell systemd[1]: dbus-:1.1-org.fedoraproject.Setroubleshootd@0.service: Failed with result 'signal'.
Apr 23 07:26:37 dell systemd[1]: dbus-:1.1-org.fedoraproject.Setroubleshootd@0.service: Consumed 1.996s CPU time.
Apr 23 07:26:41 dell systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
Apr 23 07:26:54 dell kernel: watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [kworker/0:4:152]
Apr 23 07:26:54 dell kernel: Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink qrtr bnep vfat fat joydev hid_multitouch snd_sof_pci_intel_cnl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_hda_codec_hdmi snd_sof_xtensa_dsp snd_sof snd_sof_utils soundwire_bus snd_ctl_led snd_soc_skl iwlmvm snd_soc_hdac_hda snd_hda_ext_core snd_soc_sst_ipc snd_hda_codec_realtek dell_rbtn snd_soc_sst_dsp snd_soc_acpi_intel_match snd_hda_codec_generic snd_soc_acpi snd_soc_core x86_pkg_temp_thermal mac80211 snd_compress intel_powerclamp snd_hda_intel iTCO_wdt snd_intel_dspcfg i2c_designware_platform mei_wdt iTCO_vendor_support dell_laptop snd_intel_sdw_acpi i2c_designware_core intel_rapl_msr ledtrig_audio coretemp snd_hda_codec dcdbas libarc4 kvm_intel snd_hda_core dell_smm_hwmon
Apr 23 07:26:54 dell kernel: snd_hwdep kvm btusb snd_seq btrtl snd_seq_device iwlwifi btbcm snd_pcm uvcvideo irqbypass btintel rapl btmtk videobuf2_vmalloc intel_cstate snd_timer videobuf2_memops dell_wmi intel_uncore videobuf2_v4l2 pcspkr wmi_bmof videobuf2_common cfg80211 snd bluetooth dell_smbios mei_me dell_wmi_sysman i2c_i801 intel_wmi_thunderbolt firmware_attributes_class processor_thermal_device_pci_legacy soundcore dell_wmi_descriptor i2c_smbus videodev mei intel_soc_dts_iosf processor_thermal_device mc processor_thermal_rfim intel_lpss_pci processor_thermal_mbox intel_lpss processor_thermal_rapl rfkill intel_rapl_common idma64 intel_pch_thermal int3403_thermal int340x_thermal_zone dell_smo8800 intel_pmc_core int3400_thermal acpi_thermal_rel intel_hid sparse_keymap acpi_pad xfs libcrc32c dm_crypt typec_displayport i915 nouveau intel_gtt drm_ttm_helper drm_buddy i2c_algo_bit drm_dp_helper drm_kms_helper rtsx_pci_sdmmc mmc_core syscopyarea sysfillrect sysimgblt fb_sys_fops cec mxm_wmi nvme ahci
Apr 23 07:26:54 dell kernel: ttm crct10dif_pclmul libahci crc32_pclmul crc32c_intel e1000e drm nvme_core ghash_clmulni_intel libata serio_raw t10_pi rtsx_pci i2c_hid_acpi i2c_hid wmi video pinctrl_cannonlake dm_mirror dm_region_hash dm_log dm_mod fuse
Apr 23 07:26:54 dell kernel: CPU: 0 PID: 152 Comm: kworker/0:4 Not tainted 5.14.0-162.23.1.el9_1.x86_64 #1
Apr 23 07:26:54 dell kernel: Hardware name: Dell Inc. Precision 3530/04MPXC, BIOS 1.25.0 11/21/2022
Apr 23 07:26:54 dell kernel: Workqueue: pm pm_runtime_work
Apr 23 07:26:54 dell kernel: RIP: 0010:__do_softirq+0x78/0x2ac
Apr 23 07:26:54 dell kernel: Code: 14 81 67 2c ff f7 ff ff be 00 01 00 00 e8 10 fa 34 ff c7 44 24 10 0a 00 00 00 65 66 c7 05 0e a6 22 70 00 00 fb b8 ff ff ff ff <49> c7 c3 c0 60 a0 90 41 0f bc c7 89 c5 83 c5 01 74 76 48 98 4d 8d
Apr 23 07:26:54 dell kernel: RSP: 0018:ffffaec780003f98 EFLAGS: 00000246
Apr 23 07:26:54 dell kernel: RAX: 00000000ffffffff RBX: 0000000000000000 RCX: 000000000000001f
Apr 23 07:26:54 dell kernel: RDX: 000000000000002a RSI: 000000003161faed RDI: fffffffa3400f4e9
Apr 23 07:26:54 dell kernel: RBP: ffffaec781697ab8 R08: 000000061b2275ef R09: 7fffffffffffffff
Apr 23 07:26:54 dell kernel: R10: 000000061abc0200 R11: 000000061abc493e R12: 0000000000000000
Apr 23 07:26:54 dell kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000200
Apr 23 07:26:54 dell kernel: FS:  0000000000000000(0000) GS:ffff9d53ac200000(0000) knlGS:0000000000000000
Apr 23 07:26:54 dell kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 23 07:26:54 dell kernel: CR2: 00007f0d739ac454 CR3: 000000032b810005 CR4: 00000000003706f0
Apr 23 07:26:54 dell kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 23 07:26:54 dell kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Apr 23 07:26:54 dell kernel: Call Trace:
Apr 23 07:26:54 dell kernel: <IRQ>
Apr 23 07:26:54 dell kernel: __irq_exit_rcu+0xb5/0xe0

[jbkt23]

@timppu

nobody cared (try booting with the "irqpoll" option)

Searching that term does not bring up a specific solution. The work around is to add the suggested “irgpoll” statement to your kernel command line arguments.
To do this one time, when you boot highlight the default kernel and enter the letter “e”. Then, scroll down to the line beginning with “linux”, go to the end, delete “rhgb quiet” and add “irqpoll”. Then enter Ctrl+x to boot.
Let us know what happens.

[timppu]

Using the “irqpoll” option didn’t really get me any further. I saw some extra messages and the mention of “nouveau” (the open source NVidia drivers?) in some of them caught my interest, whether this had anything to do with graphics drivers etc.

Anyway, as the issue seemed to be somehow linked to connected USB devices (the laptop works fine when no USB devices are connected), I finally replaced my USB hub with a newer one, and the issues seem to have vanished, at least for now.

So I don’t know what it is, was that older USB hub breaking down and causing those CPU soft lockups (it doesn’t show any issues on the same laptop running Windows 10 or 11 though), or did it use a different driver than the newer USB hub and the driver had issues, I don’t know.

Googling for it earlier and trying various fixes which didn’t seem to work, somehow this discussion to me sounds like a possible explanantion what was happening to me:

https://patchwork.kernel.org/project/linux-arm-kernel/patch/2127594073.298820.1493143869792@email.1und1.de/

Especially this part in the discussion piqued my interest:

If I read that right, the usbmon output shows that the interrupt
endpoint is stalled (keeps returning -EPIPE). A STALL is a special
device-side USB condition that tells the host something is wrong and
will persist until cleared manually. It seems that the driver isn’t
prepared for this (see
drivers/usb/serial/pl2303.c#pl2303_read_int_callback) and just keeps
resubmitting the URB, so it will stall again as fast as the endpoint
allows it to. This may be the reason why you get so many transfers
that it overwhelms the CPU.

A fix would be to catch -EPIPE in that function and handle it
explicitly (with either a CLEAR_STALL to the endpoint or a full USB
reset… would have to look at the documentation for PL2303 to see
what the stall actually means and how you’re supposed to treat it).

Either way, while I didn’t find an actual fix to my problem, I found a solution, replacing the USB hub to which e.g. my keyboard, mouse and USB audio speakers are connected.

[timppu]

…aaaand no. The issue is back. It seemed as if I didn’t see these errors anymore, earlier I couldn’t normally even boot up the system normally before the whole system would become stuck, but now it seemed to work just fine…

…until I was away from the system for awhile so that the screen became locked, and when I clicked the mouse or pressed a key in order to get to the login screen, the screen would get stuck. I could only proceed by disconnecting my newer USB hub and the USB-C → DisplayPort cable, but even then the system would get stuck when trying to shut it down. And yeah there were again lots of those error messages in /var/log/messages.

Still unsolved.

[timppu]

Now it appears to me the problems don’t come from connecting USB devices, but connecting AC power. I don’t see the CPU soft lockups with my USB-C → DisplayPort monitor and USB hub connected, as long as I don’t have AC power connected.

I became confused because whenever I connected the USB devices, I would connect the AC power as well.

After some googling I decided to disable hibernate (and sleep) modes using these instructions:

How to Disable Suspend and Hibernation Modes In Linux

Another thing, I forgot that I had decided not to create a separate swap partition, but didn’t remember to create a swap file, so I created a 4GB swap file as well, just in case.

At least for now the system seems stable also with AC power (and all USB hubs/devices connected) but I will keep using this for some time before marking this as a solution, as I already earlier thought I had solved the problem, but then the issues came back.

Not sure if it affects this as well that I have Secure Boot enabled, and also I enabled luks2 encryption for the whole root partition.

It seems to me these may cause issues in some cases (e.g. I read some discussion where encryption caused problems for the swap (partition)). So maybe I wouldn’t have run any issues if I had not encrypted the Linux partition and/or was not using SecureBoot, but I rather use encryption as this is my work laptop, and SecureBoot is required for Windows 11 which is on this dualboot laptop as well (it is encrypted too, with BitLocker).