Performance drop compared Centos 7.9

[sbsea]

we are noticing performance drop compared to Centos 7.9 -

few differences - spike in IRQ % and high buffer use - has anyone encountered this issue?
any thoughts on why high use of IRQ?

[Ritov]

What exactly are your systems doing?

[sbsea]

I am running cassandra load tests against 3-node linux cluster.

test results for 100000000 writes and 50 threads
Write Op rate 118,082 op/s (Rocky) 126,771 op/s (Centos)

Centos

tuna --show_irqs | grep nvme
24 nvme0q0 0
29 nvme0q1 0
30 nvme0q2 0
31 nvme0q3 0
32 nvme0q4 0
33 nvme0q5 0
34 nvme0q6 0
35 nvme0q7 0
36 nvme0q8 0
37 nvme0q9 0
38 nvme0q10 0
39 nvme0q11 0
40 nvme0q12 0
41 nvme0q13 0
42 nvme0q14 0
43 nvme0q15 0

Rocky

tuna --show_irqs | grep nvme
26 nvme0q0 0
27 nvme0q1 0,8
28 nvme0q2 1
29 nvme0q3 2
30 nvme0q4 3
31 nvme0q5 4
32 nvme0q6 5
33 nvme0q7 6
34 nvme0q8 7
35 nvme0q9 9
36 nvme0q10 10
37 nvme0q11 11
38 nvme0q12 12
39 nvme0q13 13
40 nvme0q14 14
41 nvme0q15 15

Disk I/O rate is comparable using ‘dd’, ‘fio’ tools

[Ritov]

Sorry if I miss the major Rocky version but either one or other (8/9) both use very different kernels compared to C7. Did you check the new and different schedulers? They will for sure give you different results. A good starting point as abstraction layer is the tuneD service, check this first and try different profiles …

[sbsea]

we are using Rocky Linux release 8.9 (Green Obsidian) (core-4.1-amd64:core-4.1-noarch)

[Xino]

My 2 cents
I am interested and trying to clarify the situation and if I can help.
Could you share with us are you testing on the same motherboard?
Motherboard - brand, model, firmware?
Processor
Memory - volume
Disk/s - number, single and/or in RAID
Setting in BIOS to optimize processes
Swap - enabled, disabled
As they wrote before me, is the optimized operating mode of the OS set
Is the primary installation but the OS minimal?
Kernel optimization, settings
Network optimization of the 3-node cluster in question
I do not wish to underestimate the effort and time allocated by you!