KVM Switch Causes Remote Reboots To Stall

Rocky Linux General
1

I use Cockpit to manage my RL9 server remotely. When there are kernel updates I reboot the system via the Cockpit interface. Each time I’ve done this, at least twice, the process stops till I go and turn on the monitor attached to the server.
So I did another test just now logging in via ssh and rebooting and the same thing happened, it stalled, but I added a subtle difference described as follows.
My server is connected to a monitor via a KVM switch, which in this case is enabled for another machine at the time of reboot. With the monitor off, to continue the reboot all I had to do was switch the KVM to the rocky machine.
How do I prevent this from happening?
This did not happen when I was running RL8 with same setup.
There is a difference, I upgraded the KVM switch from VGA based to HDMI based since my other machines don’t have VGA connections, done soon after installing RL9.
There are two parts to a reboot, Shutting down and then efi firmware boot process.
Is the HDMI interface causing the machine not to shutdown completely, I can check logs for that or is the firmware hanging on boot?

Journalctl -b -1 OUtput:

Apr 16 10:19:32 bagend.kellyrand systemd[1]: Reached target System Shutdown.
Apr 16 10:19:32 bagend.kellyrand systemd[1]: Reached target Late Shutdown Services.
Apr 16 10:19:32 bagend.kellyrand systemd[1]: systemd-reboot.service: Deactivated successfully.
Apr 16 10:19:32 bagend.kellyrand systemd[1]: Finished System Reboot.
Apr 16 10:19:32 bagend.kellyrand systemd[1]: Reached target System Reboot.
Apr 16 10:19:32 bagend.kellyrand kernel: audit: type=1130 audit(1681654772.253:194): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=system>
Apr 16 10:19:32 bagend.kellyrand systemd[1]: Shutting down.
Apr 16 10:19:32 bagend.kellyrand systemd[1]: Using hardware watchdog 'SP5100 TCO timer', version 0, device /dev/watchdog0
Apr 16 10:19:32 bagend.kellyrand systemd[1]: Watchdog running with a timeout of 10min.
Apr 16 10:19:32 bagend.kellyrand kernel: watchdog: watchdog0: watchdog did not stop!
Apr 16 10:19:32 bagend.kellyrand systemd-shutdown[1]: Using hardware watchdog 'SP5100 TCO timer', version 0, device /dev/watchdog0
Apr 16 10:19:32 bagend.kellyrand systemd-shutdown[1]: Watchdog running with a timeout of 10min.
Apr 16 10:19:32 bagend.kellyrand systemd-shutdown[1]: Syncing filesystems and block devices.
Apr 16 10:19:32 bagend.kellyrand systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Apr 16 10:19:32 bagend.kellyrand systemd-journald[626]: Journal stopped

The only thing I see is the watchdog timer not shutting down. It says it has a 10 min. wait.

2

I’m still experiencing this issue on reboots. The “Watchdog” is not letting the system shutdown completely until I switch the KVM focus to the server enabling keyboard input and video if the monitor was on, it is not, and that status has no effect.
I’ve searched for a “watchdog” service, socket and timer and have not found anything under systemd.
Where do I hunt this down? I can’t imagine that all servers are connected to a keyboard and monitor continuously.

So these are the “watchdogs” configured in the kernel:

CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ACPI_WATCHDOG=y
CONFIG_IPMI_WATCHDOG=m
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
CONFIG_WATCHDOG_HANDLE_BOOT_ENABLED=y
CONFIG_WATCHDOG_OPEN_TIMEOUT=0
CONFIG_WATCHDOG_SYSFS=y
CONFIG_SOFT_WATCHDOG=m
CONFIG_HP_WATCHDOG=m

Reading the kconfig file found here on my fedora install:
/usr/src/kernels/6.4.6-100.fc37.x86_64/drivers/watchdog/
I might be able to blacklist a module or add a kernel parameter to change the behavior.

3

After a number of test reboots I’m beginning to think this is not a kernel watchdog issue. It is a firmware / hardware / KVM switch issue.