High I/O load causing server to crash

Rocky Linux Help & Support
1

We have few instances of servers crashing with high I/O load running on rocky 9.x wherein the application is using NFS share and the CPU load builds up over time ( about 20-30 mins ) causing the server to go unresponsive and ultimately causing outage(s)
Hardware: HPE Gen 8 servers // 256G RAM // 48 Core CPU // swap disabled
Role: Application servers running containerd with K8S
Behaviour:

We are seeing the application getting hung due to heavy I/O causing CPU load to go up 600+ and server get unresponsive.
This seems to be happening under high load from application and same application setup works fine on centos servers ( we are in transition phase )

atop suggests the load building up due to high read / write on NFS mount and also causing local disk to have high utilization, we are trying to get some help if this may be related to tuning / config issue

10 Min before outage

image
image1036×481 265 KB

2

20 min ago atop ( i was not able to add two screenshot in description )

image
image1632×913 134 KB

3

Define what it means by “the application”.

4

@gerry666uk We run a java based web application which runs on containerd + kuberenetes

The server gets hung and only solution is to force reboot

5

OK, so java web app would normally use threading for multiple requests, but your output shows multiple java processes? Are they all “the application”, or are there actually multiple applications? If you stop “the application” (or one of them), does the i/o problem go away? I’m guessing they all have the same issue and they’re cumulative?

6

They appear to be cumulative. NFS seems to be written with too much data , we are not sure whether the application is pushing too much data or some sort of buffer is getting full and causing outage ultimately