fkaluza
October 18, 2024, 10:21am
1
I am struggling with large memory usage (fresh rebooted server) eats 6 GB of RAM memory
total used free shared buff/cache available
Mem: 31776 6363 23424 1 1988 25028
after shutting down services and killing processes (sudo sync; echo 3 > /proc/sys/vm/drop_caches; free -m showed still 6 GB in use) I found
Area Used Cache Noncache
kernel dynamic memory 5.9G 47.1M 5.9G
Kernel dynamic memory loccks 5.9 GB in noncache memory.
This is baremetal server in OVH with AMD EPYC 4244P 6-Core Processor
Linux 4.18.0-553.22.1.el8_10.x86_64 #1 SMP Wed Sep 25 09:20:43 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Could you give some hint to look what is consuming memory
anthyve
October 18, 2024, 11:02am
2
sudo slabtop should provide some insight into the kernel allocator.
6Gb sounds a lot, but it looks like you have 32Gb. I have 16Gb and 3Gb being used, so it’s a similar ratio.
fkaluza
October 21, 2024, 10:25am
4
on other machines I don’t have problems with nonchache memory only with two with the same hardware, I still looking whi lernel uses noncache
Can you post output of sudo slabtop?
at this moment I have some jaa software running on machine
smem -tkw
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 15.1G 9.0G 6.1G
userspace memory 9.7G 485.8M 9.2G
free memory 6.3G 6.3G 0
----------------------------------------------------------
31.0G 15.7G 15.
3
slabtop -o
Active / Total Objects (% used) : 3109524 / 3138039 (99.1%)
Active / Total Slabs (% used) : 89130 / 89130 (100.0%)
Active / Total Caches (% used) : 172 / 274 (62.8%)
Active / Total Size (% used) : 824303.12K / 839000.36K (98.2%)
Minimum / Average / Maximum Object : 0.01K / 0.27K / 11.62K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
681975 681215 99% 0.19K 32475 21 129900K dentry
385024 384406 99% 0.06K 6016 64 24064K lsm_inode_cache
219072 218954 99% 0.09K 5216 42 20864K kmalloc-96
210221 210221 100% 1.07K 7249 29 231968K nfs_inode_cache
173656 173596 99% 0.57K 6202 28 99232K radix_tree_node
161670 161230 99% 0.02K 951 170 3804K avtab_node
143104 142139 99% 0.06K 2236 64 8944K kmalloc-64
134400 134400 100% 0.02K 525 256 2100K kmalloc-16
117810 117810 100% 0.04K 1155 102 4620K xfs_ifork
110220 109930 99% 1.06K 3674 30 117568K xfs_inode
109725 109725 100% 0.19K 5225 21 20900K xfs_ili
65024 63024 96% 0.03K 508 128 2032K kmalloc-32
54784 54784 100% 0.01K 107 512 428K kmalloc-8
52008 51557 99% 0.64K 2167 24 34672K inode_cache
49472 49463 99% 0.12K 1546 32 6184K kernfs_node_cache
49216 49057 99% 0.06K 769 64 3076K ebitmap_node
39354 39295 99% 0.19K 1874 21 7496K kmalloc-192
26690 26140 97% 0.02K 157 170 628K hashtab_node
25600 25600 100% 0.02K 100 256 400K lsm_file_cache
21088 20987 99% 0.25K 659 32 5272K kmalloc-256
20930 20034 95% 0.23K 598 35 4784K vm_area_struct
18816 9761 51% 0.50K 588 32 9408K kmalloc-512
18688 11402 61% 1.00K 584 32 18688K kmalloc-1k
16704 16378 98% 0.06K 261 64 1044K anon_vma_chain
16640 16640 100% 0.06K 260 64 1040K kmalloc-rcl-64
12122 12122 100% 0.71K 551 22 8816K proc_inode_cache
10122 9927 98% 0.09K 241 42 964K anon_vma
9920 9795 98% 0.06K 155 64 620K vmap_area
9296 9296 100% 0.07K 166 56 664K Acpi-Operand
8220 8220 100% 0.53K 274 30 4384K drbd_req
8211 8211 100% 0.19K 391 21 1564K drbd_ee
8008 8008 100% 0.07K 143 56 572K avc_node
6804 6804 100% 0.44K 189 36 3024K xfs_buf
6752 4963 73% 0.25K 211 32 1688K filp
6144 6144 100% 0.01K 12 512 48K kmalloc-cg-8
6035 6035 100% 0.05K 71 85 284K ftrace_event_field
6016 5428 90% 0.12K 188 32 752K kmalloc-128
5760 4960 86% 0.12K 180 32 720K pid
5508 5508 100% 0.08K 108 51 432K task_delay_info
4896 4896 100% 0.12K 153 32 612K nfs_page
4845 4845 100% 0.05K 57 85 228K Acpi-Namespace
4746 4746 100% 0.09K 113 42 452K kmalloc-rcl-96
4116 4116 100% 0.19K 196 21 784K cred_jar
4096 4068 99% 0.25K 128 32 1024K bio-224
3936 3936 100% 0.12K 123 32 492K eventpoll_epi
3072 3072 100% 0.02K 12 256 48K kmalloc-cg-16
2775 2675 96% 0.31K 111 25 888K bio-280
2760 2760 100% 0.09K 60 46 240K trace_event_file
2656 2656 100% 0.25K 83 32 664K skbuff_head_cache
2479 2442 98% 0.42K 67 37 1072K xfs_efd_item
2464 2464 100% 0.50K 77 32 1232K kmalloc-cg-512
2356 2318 98% 0.41K 62 38 992K xfs_efi_item
2176 2176 100% 0.25K 68 32 544K proc_dir_entry
2121 2121 100% 0.38K 101 21 808K mnt_cache
2040 2040 100% 0.02K 12 170 48K fsnotify_mark_connector
2040 2040 100% 0.02K 12 170 48K audit_buffer
1911 1911 100% 0.76K 91 21 1456K shmem_inode_cache
1833 1833 100% 0.10K 47 39 188K buffer_head
1829 1674 91% 0.26K 59 31 472K xfs_buf_item
1734 1734 100% 0.08K 34 51 136K kernfs_iattrs_cache
1730 1724 99% 11.62K 1730 1 27680K task_struct
1712 1662 97% 2.00K 107 16 3424K kmalloc-2k
1536 1536 100% 0.03K 12 128 48K kmalloc-cg-32
1496 1488 99% 4.00K 187 8 5984K kmalloc-4k
1275 1275 100% 0.05K 15 85 60K drbd_al
1250 1250 100% 1.25K 50 25 1600K signal_cache
1224 1224 100% 0.04K 12 102 48K pde_opener
1168 1168 100% 0.05K 16 73 64K file_lock_ctx
1056 1056 100% 0.12K 33 32 132K kmalloc-rcl-128
1035 965 93% 0.69K 45 23 720K sock_inode_cache
1024 859 83% 1.00K 32 32 1024K kmalloc-cg-1k
882 882 100% 0.19K 42 21 168K inet_peer_cache
876 876 100% 0.05K 12 73 48K Acpi-Parse
840 840 100% 0.07K 15 56 60K eventpoll_pwq
768 768 100% 0.06K 12 64 48K kmalloc-cg-64
768 768 100% 0.06K 12 64 48K fs_cache
768 768 100% 0.06K 12 64 48K xfs_extfree_intent
765 685 89% 2.06K 51 15 1632K sighand_cache
704 704 100% 0.06K 11 64 44K kmem_cache_node
672 672 100% 0.07K 12 56 48K nsproxy
672 672 100% 1.12K 24 28 768K UNIX
663 663 100% 0.08K 13 51 52K Acpi-State
625 561 89% 0.31K 25 25 200K nf_conntrack
621 621 100% 0.17K 27 23 108K xfs_icr
612 612 100% 0.08K 12 51 48K Acpi-ParseExt
612 612 100% 0.08K 12 51 48K sigqueue
612 612 100% 0.08K 12 51 48K inotify_inode_mark
567 567 100% 0.19K 27 21 108K bio-160
532 532 100% 0.56K 19 28 304K skbuff_fclone_cache
504 504 100% 0.44K 14 36 224K kmem_cache
504 504 100% 0.09K 12 42 48K kmalloc-cg-96
448 448 100% 0.12K 14 32 56K tcp_bind_bucket
444 444 100% 0.21K 12 37 96K file_lock_cache
444 444 100% 0.21K 12 37 96K xfs_btree_cur
442 442 100% 0.94K 13 34 416K nfs_write_data
437 437 100% 0.69K 19 23 304K files_cache
432 432 100% 0.88K 12 36 384K task_group
432 432 100% 0.11K 12 36 48K khugepaged_mm_slot
420 420 100% 0.23K 12 35 96K xfs_trans
416 416 100% 0.25K 13 32 104K ip_dst_cache
408 408 100% 0.47K 12 34 192K xfs_da_state
408 408 100% 0.94K 12 34 384K nfs_read_data
399 399 100% 0.19K 19 21 76K kmalloc-cg-192
384 384 100% 0.25K 12 32 96K key_jar
384 384 100% 0.12K 12 32 48K seq_file
384 384 100% 0.25K 12 32 96K biovec-16
384 384 100% 1.00K 12 32 384K biovec-64
384 384 100% 0.25K 12 32 96K rpc_tasks
376 372 98% 8.00K 94 4 3008K biovec-max
372 372 100% 0.26K 12 31 96K numa_policy
364 320 87% 2.44K 28 13 896K TCP
360 360 100% 0.27K 12 30 96K tw_sock_TCP
360 360 100% 0.27K 12 30 96K tw_sock_TCPv6
352 352 100% 0.25K 11 32 88K pool_workqueue
340 329 96% 8.00K 85 4 2720K kmalloc-8k
340 340 100% 0.05K 4 85 16K ip_fib_trie
336 336 100% 1.13K 12 28 384K perf_event
336 336 100% 0.14K 12 28 48K blkdev_ioc
320 320 100% 0.12K 10 32 40K kmalloc-cg-128
315 315 100% 1.50K 15 21 480K mm_struct
312 312 100% 0.33K 13 24 104K request_sock_TCP
306 306 100% 0.94K 9 34 288K bdev_cache
300 300 100% 1.25K 12 25 384K UDP
300 300 100% 0.31K 12 25 96K ip6_dst_cache
292 292 100% 0.05K 4 73 16K ip_fib_alias
288 288 100% 0.33K 12 24 96K request_sock_TCPv6
280 280 100% 1.12K 10 28 320K RAW
276 276 100% 0.34K 12 23 96K taskstats
270 270 100% 0.27K 9 30 72K posix_timers_cache
264 264 100% 1.44K 12 22 384K UDPv6
264 264 100% 0.18K 12 22 48K xfs_log_ticket
256 256 100% 2.00K 16 16 512K kmalloc-cg-2k
256 256 100% 0.06K 4 64 16K fib6_nodes
256 256 100% 0.12K 8 32 32K scsi_sense_cache
252 252 100% 0.19K 12 21 48K uid_cache
252 252 100% 0.75K 12 21 192K nfs_commit_data
248 233 93% 4.00K 31 8 992K kmalloc-cg-4k
192 192 100% 2.00K 12 16 384K biovec-128
192 192 100% 2.00K 12 16 384K rpc_buffers
168 168 100% 1.31K 7 24 224K RAWv6
160 160 100% 0.25K 5 32 40K bio-248
144 144 100% 2.56K 12 12 384K TCPv6
102 102 100% 0.04K 1 102 4K fat_cache
100 100 100% 0.77K 5 20 80K fat_inode_cache
96 96 100% 4.00K 12 8 384K names_cache
92 92 100% 0.69K 4 23 64K kioctx
92 92 100% 0.69K 4 23 64K rpc_inode_cache
84 84 100% 0.19K 4 21 16K aio_kiocb
72 72 100% 0.88K 2 36 64K dax_cache
69 69 100% 0.69K 3 23 48K isofs_inode_cache
64 64 100% 0.06K 1 64 4K dmaengine-unmap-2
64 64 100% 0.06K 1 64 4K drbd_bm
63 63 100% 0.19K 3 21 12K kmalloc-rcl-192
63 63 100% 0.19K 3 21 12K bio-184
60 60 100% 0.13K 2 30 8K fscache_cookie_jar
52 52 100% 8.00K 13 4 416K irq_remap_cache
48 48 100% 8.00K 12 4 384K kmalloc-cg-8k
48 48 100% 2.65K 4 12 128K request_queue
48 48 100% 0.66K 2 24 32K hugetlbfs_inode_cache
32 32 100% 0.25K 1 32 8K bio_integrity_payload
32 32 100% 1.00K 1 32 32K mqueue_inode_cache
32 32 100% 0.25K 1 32 8K sgpool-8
32 32 100% 0.50K 1 32 16K sgpool-16
32 32 100% 1.00K 1 32 32K sgpool-32
30 30 100% 1.06K 1 30 32K dmaengine-unmap-128
28 28 100% 1.12K 1 28 32K PING
24 24 100% 1.31K 1 24 32K PINGv6
21 21 100% 0.19K 1 21 4K dmaengine-unmap-16
16 16 100% 2.00K 1 16 32K sgpool-64
15 15 100% 2.06K 1 15 32K dmaengine-unmap-256
8 8 100% 7.88K 2 4 64K net_namespace
8 8 100% 4.00K 1 8 32K sgpool-128
0 0 0% 0.25K 0 32 0K kmalloc-cg-256
0 0 0% 0.01K 0 512 0K kmalloc-rcl-8
0 0 0% 0.02K 0 256 0K kmalloc-rcl-16
0 0 0% 0.03K 0 128 0K kmalloc-rcl-32
0 0 0% 0.25K 0 32 0K kmalloc-rcl-256
0 0 0% 0.50K 0 32 0K kmalloc-rcl-512
0 0 0% 1.00K 0 32 0K kmalloc-rcl-1k
0 0 0% 2.00K 0 16 0K kmalloc-rcl-2k
0 0 0% 4.00K 0 8 0K kmalloc-rcl-4k
0 0 0% 8.00K 0 4 0K kmalloc-rcl-8k
0 0 0% 0.09K 0 42 0K dma-kmalloc-96
0 0 0% 0.19K 0 21 0K dma-kmalloc-192
0 0 0% 0.01K 0 512 0K dma-kmalloc-8
0 0 0% 0.02K 0 256 0K dma-kmalloc-16
0 0 0% 0.03K 0 128 0K dma-kmalloc-32
0 0 0% 0.06K 0 64 0K dma-kmalloc-64
0 0 0% 0.12K 0 32 0K dma-kmalloc-128
0 0 0% 0.25K 0 32 0K dma-kmalloc-256
0 0 0% 0.50K 0 32 0K dma-kmalloc-512
0 0 0% 1.00K 0 32 0K dma-kmalloc-1k
0 0 0% 2.00K 0 16 0K dma-kmalloc-2k
0 0 0% 4.00K 0 8 0K dma-kmalloc-4k
0 0 0% 8.00K 0 4 0K dma-kmalloc-8k
0 0 0% 0.05K 0 85 0K shared_policy_node
0 0 0% 0.43K 0 37 0K uts_namespace
0 0 0% 0.12K 0 34 0K iint_cache
0 0 0% 0.05K 0 73 0K avc_xperms_node
0 0 0% 0.05K 0 85 0K avc_xperms_decision_node
0 0 0% 0.03K 0 128 0K avc_xperms_data
0 0 0% 0.04K 0 102 0K avtab_extended_perms
0 0 0% 0.09K 0 46 0K configfs_dir_cache
0 0 0% 0.06K 0 64 0K skbuff_ext_cache
0 0 0% 0.06K 0 64 0K iommu_iova
0 0 0% 0.62K 0 25 0K user_namespace
0 0 0% 0.06K 0 64 0K ksm_rmap_item
0 0 0% 0.06K 0 64 0K ksm_stable_node
0 0 0% 0.05K 0 85 0K ksm_mm_slot
0 0 0% 0.25K 0 32 0K dquot
0 0 0% 0.88K 0 36 0K xfrm_state
0 0 0% 0.44K 0 36 0K xfrm_dst_cache
0 0 0% 0.12K 0 32 0K secpath_cache
0 0 0% 0.40K 0 20 0K request_sock_subflow
0 0 0% 2.12K 0 15 0K MPTCP
0 0 0% 1.25K 0 25 0K UDP-Lite
0 0 0% 0.19K 0 21 0K ip_mrt_cache
0 0 0% 0.23K 0 34 0K ip4-frags
0 0 0% 0.24K 0 33 0K pid_namespace
0 0 0% 0.08K 0 51 0K audit_tree_mark
0 0 0% 0.05K 0 85 0K fasync_cache
0 0 0% 0.62K 0 25 0K dio
0 0 0% 0.03K 0 128 0K dnotify_struct
0 0 0% 0.08K 0 51 0K dnotify_mark
0 0 0% 0.07K 0 56 0K fsnotify_mark
0 0 0% 0.05K 0 73 0K fanotify_event
0 0 0% 0.09K 0 46 0K fanotify_perm_event
0 0 0% 0.19K 0 21 0K userfaultfd_ctx_cache
0 0 0% 0.55K 0 29 0K bfq_queue
0 0 0% 0.23K 0 35 0K bfq_io_cq
0 0 0% 0.12K 0 32 0K btree_node
0 0 0% 0.05K 0 73 0K uhci_urb_priv
0 0 0% 1.44K 0 22 0K UDPLITEv6
0 0 0% 0.19K 0 21 0K ip6_mrt_cache
0 0 0% 0.22K 0 36 0K ip6-frags
0 0 0% 2.25K 0 14 0K MPTCPv6
0 0 0% 0.05K 0 73 0K zswap_entry
0 0 0% 2.57K 0 12 0K dm_uevent
0 0 0% 0.06K 0 64 0K io
0 0 0% 3.16K 0 10 0K kcopyd_job
0 0 0% 0.16K 0 24 0K xfs_rud_item
0 0 0% 0.66K 0 24 0K xfs_rui_item
0 0 0% 0.16K 0 24 0K xfs_cud_item
0 0 0% 0.41K 0 38 0K xfs_cui_item
0 0 0% 0.16K 0 24 0K xfs_bud_item
0 0 0% 0.20K 0 20 0K xfs_bui_item
0 0 0% 0.48K 0 33 0K xfs_dquot
0 0 0% 0.52K 0 31 0K xfs_dqtrx
0 0 0% 2.59K 0 12 0K x86_emulator
0 0 0% 0.12K 0 32 0K pte_list_desc
0 0 0% 0.19K 0 21 0K kvm_mmu_page_header
0 0 0% 8.95K 0 3 0K kvm_vcpu
0 0 0% 0.16K 0 24 0K kvm_async_pf
0 0 0% 0.03K 0 128 0K jbd2_revoke_record_s
0 0 0% 0.02K 0 256 0K jbd2_revoke_table_s
0 0 0% 0.12K 0 34 0K jbd2_journal_head
0 0 0% 0.05K 0 73 0K jbd2_journal_handle
0 0 0% 0.06K 0 64 0K jbd2_inode
0 0 0% 0.25K 0 32 0K jbd2_transaction_s
0 0 0% 0.05K 0 73 0K mbcache
0 0 0% 0.04K 0 102 0K ext4_extent_status
0 0 0% 0.03K 0 128 0K ext4_pending_reservation
0 0 0% 0.06K 0 64 0K ext4_io_end
0 0 0% 0.04K 0 102 0K ext4_system_zone
0 0 0% 0.10K 0 39 0K ext4_prealloc_space
0 0 0% 0.12K 0 32 0K ext4_allocation_context
0 0 0% 0.05K 0 73 0K ext4_free_data
0 0 0% 1.14K 0 28 0K ext4_inode_cache
0 0 0% 0.88K 0 36 0K fuse_inode
0 0 0% 0.15K 0 26 0K fuse_request
0 0 0% 0.22K 0 36 0K nf-frags
0 0 0% 0.24K 0 33 0K nf_conntrack_expect
0 0 0% 0.25K 0 32 0K nfs_direct_cache
0 0 0% 2.08K 0 15 0K nfs4_xattr_cache_cache
I also got an reply from OVH support - in short AMD EPYC cpu causes kernel takes more ram
p1u3o
December 2, 2024, 3:23pm
8
Did you ever figure out the reason for the large base memory usage (~5GB) on this process model. I couldn’t find anything useful in dmesg or slabtop to explain but it does seem extremely high to be that on a fresh boot.
unfortunately no solution, at this time I planed migration to non EPYC servers
system
February 1, 2025, 10:00am
10
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.