Hello everyone,
I am currently investigating a critical server hang incident on a single standalone server running Rocky Linux 8.5.
This server operates as a dedicated logging service that receives mirrored traffic over a 10GbE network from 6 production database servers, filtering and logging DB-related packets in real time.
Recently, the system suddenly became completely unresponsive (Hang state) and required a hard reset. Notably, we were continuously monitoring the server’s resource utilization, and at the time of the hang, both CPU and memory usage were extremely stable at around 20%.
Upon analyzing the system logs, the only suspicious error found was the following message, which is still continuously flooding my logs every single minute even after the reboot.
Affinity broken due to vector space exhaustion
Through my initial investigation, I confirmed that this correlates with Red Hat Bugzilla BZ#2084646 and Rocky Linux Errata RLSA-2022:6460.
While I now know which bug it is and that a kernel upgrade is required, I am struggling to find clear technical references that explain the exact causal relationship (correlation) between this specific error log and a complete system hang.
If anyone has experienced this exact issue or can share insights into the technical chain of events that leads from vector exhaustion to a total system freeze, I would be incredibly grateful. Specifically, I would highly appreciate your expertise on the following 4 points.
Environment:
-
OS: Rocky Linux 8.5
-
Kernel Version:
4.18.0-348.el8.0.2.x86_64 -
irqbalance Version:
1.9.2-1.el8.x86_64 -
Hardware Specs :
Server Model: Dell PowerEdge R550
CPU: Intel Xeon Silver 4309Y (8C/16T)
RAM: 128GB (DDR4-2666 16GB RDIMM x 8)
Storage: 2.4TB 10K RPM SAS 12Gbps (Hot-plug)
Network Card: Intel X710 Dual Port 10GbE SFP+ Adapter (PCIe)
1. Detailed Mechanism of Vector Space Exhaustion
Could someone explain the exact technical root cause of how the irqbalance utility and the kernel’s APIC vector management system interact to trigger this? Since the Affinity broken logs were continuously printed over a long period before the hang, does this bug cause a “gradual leak” of interrupt vectors over time during routine IRQ balancing? How does the limit of ~202 user-space vectors per CPU actually get depleted?
2. Correlation between Continuous Logs and the Sudden Server Hang
How does this gradual vector exhaustion eventually culminate in a complete, sudden server freeze? Does the system finally reach a breaking point where critical storage (SAS/NVMe) or network driver interrupts go completely unserviced, resulting in fatal I/O blockage, driver timeouts, or kernel deadlocks?
3. Log & Diagnostic Validation
Based on a scenario where these error messages continuously repeat and eventually end with an abrupt system hang, is it safe to definitively conclude that the hang is a direct consequence of the bug described in BZ#2084646?
4. Immediate Temporary Workarounds
Apart from upgrading the kernel (which requires a strict scheduled maintenance window), what are the best temporary workarounds to stabilize the system in its current state? (e.g., Disabling the irqbalance daemon entirely, specific configuration tweaks, or banning certain high-traffic IRQs from being balanced?)
Any detailed technical insights, design docs, or architectural explanations would be immensely helpful for our report.
Thank you in advance for your time and expertise!