'Metadata corruption' at every 10 minutes

Rocky Linux General
1

Metadata corruption detected in nearly each 10 minutes.
here is the log from /var/log/messages

Jan 31 13:10:15 hostname kernel: XFS (dm-3): Metadata corruption detected at xfs_dinode_verify.part.0+0x3b3/0x690 [xfs], inode 0x492bba470 dinode
Jan 31 13:10:15 hostname kernel: XFS (dm-3): Unmount and run xfs_repair
Jan 31 13:10:15 hostname kernel: XFS (dm-3): First 128 bytes of corrupted metadata buffer:
Jan 31 13:10:15 hostname kernel: 00000000: 49 4e 81 80 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
Jan 31 13:10:15 hostname kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 31 13:10:15 hostname kernel: 00000020: 35 0b a1 d6 33 21 0a d6 35 0b a1 d6 4d 88 b5 35  5...3!..5...M..5
Jan 31 13:10:15 hostname kernel: 00000030: 35 0b a1 d6 4d 97 f7 78 00 00 00 00 00 00 05 46  5...M..x.......F
Jan 31 13:10:15 hostname kernel: 00000040: 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01  ................
Jan 31 13:10:15 hostname kernel: 00000050: 00 00 18 01 00 00 00 00 00 00 00 00 5b 4e 12 ce  ............[N..
Jan 31 13:10:15 hostname kernel: 00000060: ff ff ff ff 80 0b 05 8e 00 00 00 00 00 00 00 07  ................
Jan 31 13:10:15 hostname kernel: 00000070: 00 00 00 1a 00 37 c5 80 00 00 00 00 00 00 00 08  .....7..........
Jan 31 13:11:47 hostname kernel: XFS (dm-3): Metadata corruption detected at xfs_dinode_verify.part.0+0x3b3/0x690 [xfs], inode 0x492bba473 dinode
Jan 31 13:11:47 hostname kernel: XFS (dm-3): Unmount and run xfs_repair
Jan 31 13:11:47 hostname kernel: XFS (dm-3): First 128 bytes of corrupted metadata buffer:
Jan 31 13:11:47 hostname kernel: 00000000: 49 4e 81 80 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
Jan 31 13:11:47 hostname kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 31 13:11:47 hostname kernel: 00000020: 35 0b a7 4b 14 24 4a d8 35 0b a7 4b 1e 37 08 70  5..K.$J.5..K.7.p
Jan 31 13:11:47 hostname kernel: 00000030: 35 0b a7 4b 1e 46 4a b2 00 00 00 00 00 00 04 dd  5..K.FJ.........
Jan 31 13:11:47 hostname kernel: 00000040: 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01  ................
Jan 31 13:11:47 hostname kernel: 00000050: 00 00 18 01 00 00 00 00 00 00 00 00 37 d9 c7 26  ............7..&
Jan 31 13:11:47 hostname kernel: 00000060: ff ff ff ff d3 80 a1 b2 00 00 00 00 00 00 00 07  ................
Jan 31 13:11:47 hostname kernel: 00000070: 00 00 00 1a 00 38 30 88 00 00 00 00 00 00 00 08  .....80.........
Jan 31 13:12:06 hostname kernel: XFS (dm-3): Metadata corruption detected at xfs_dinode_verify.part.0+0x3b3/0x690 [xfs], inode 0x492bba471 dinode
Jan 31 13:12:06 hostname kernel: XFS (dm-3): Unmount and run xfs_repair
Jan 31 13:12:06 hostname kernel: XFS (dm-3): First 128 bytes of corrupted metadata buffer:
Jan 31 13:12:06 hostname kernel: 00000000: 49 4e 41 c0 03 01 00 00 00 00 00 00 00 00 00 00  INA.............
Jan 31 13:12:06 hostname kernel: 00000010: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 31 13:12:06 hostname kernel: 00000020: 35 0b ad 4b c4 82 fb e4 35 0b ad 4b c4 92 3e 28  5..K....5..K..>(
Jan 31 13:12:06 hostname kernel: 00000030: 35 0b ad 4b c4 92 3e 28 00 00 00 00 00 00 00 29  5..K..>(.......)
Jan 31 13:12:06 hostname kernel: 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 31 13:12:06 hostname kernel: 00000050: 00 00 18 01 00 00 00 00 00 00 00 00 28 50 ba 4e  ............(P.N
Jan 31 13:12:06 hostname kernel: 00000060: ff ff ff ff a7 c4 d4 a1 00 00 00 00 00 00 00 07  ................
Jan 31 13:12:06 hostname kernel: 00000070: 00 00 00 1a 00 38 b3 80 00 00 00 00 00 00 00 08  .....8..........
...
Jan 31 13:20:14 hostname kernel: XFS (dm-3): Metadata corruption detected at xfs_dinode_verify.part.0+0x3b3/0x690 [xfs], inode 0x492bba470 dinode
Jan 31 13:20:14 hostname kernel: XFS (dm-3): Unmount and run xfs_repair
Jan 31 13:20:14 hostname kernel: XFS (dm-3): First 128 bytes of corrupted metadata buffer:
Jan 31 13:20:14 hostname kernel: 00000000: 49 4e 81 80 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
Jan 31 13:20:14 hostname kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 31 13:20:14 hostname kernel: 00000020: 35 0b a1 d6 33 21 0a d6 35 0b a1 d6 4d 88 b5 35  5...3!..5...M..5
Jan 31 13:20:14 hostname kernel: 00000030: 35 0b a1 d6 4d 97 f7 78 00 00 00 00 00 00 05 46  5...M..x.......F
Jan 31 13:20:14 hostname kernel: 00000040: 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01  ................
Jan 31 13:20:14 hostname kernel: 00000050: 00 00 18 01 00 00 00 00 00 00 00 00 5b 4e 12 ce  ............[N..
Jan 31 13:20:14 hostname kernel: 00000060: ff ff ff ff 80 0b 05 8e 00 00 00 00 00 00 00 07  ................
Jan 31 13:20:14 hostname kernel: 00000070: 00 00 00 1a 00 37 c5 80 00 00 00 00 00 00 00 08  .....7..........
Jan 31 13:21:49 hostname kernel: XFS (dm-3): Metadata corruption detected at xfs_dinode_verify.part.0+0x3b3/0x690 [xfs], inode 0x492bba473 dinode
Jan 31 13:21:49 hostname kernel: XFS (dm-3): Unmount and run xfs_repair
Jan 31 13:21:49 hostname kernel: XFS (dm-3): First 128 bytes of corrupted metadata buffer:
Jan 31 13:21:49 hostname kernel: 00000000: 49 4e 81 80 03 02 00 00 00 00 00 00 00 00 00 00  IN..............
Jan 31 13:21:49 hostname kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 31 13:21:49 hostname kernel: 00000020: 35 0b a7 4b 14 24 4a d8 35 0b a7 4b 1e 37 08 70  5..K.$J.5..K.7.p
Jan 31 13:21:49 hostname kernel: 00000030: 35 0b a7 4b 1e 46 4a b2 00 00 00 00 00 00 04 dd  5..K.FJ.........
Jan 31 13:21:49 hostname kernel: 00000040: 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01  ................
Jan 31 13:21:49 hostname kernel: 00000050: 00 00 18 01 00 00 00 00 00 00 00 00 37 d9 c7 26  ............7..&
Jan 31 13:21:49 hostname kernel: 00000060: ff ff ff ff d3 80 a1 b2 00 00 00 00 00 00 00 07  ................
Jan 31 13:21:49 hostname kernel: 00000070: 00 00 00 1a 00 38 30 88 00 00 00 00 00 00 00 08  .....80.........
Jan 31 13:22:08 hostname kernel: XFS (dm-3): Metadata corruption detected at xfs_dinode_verify.part.0+0x3b3/0x690 [xfs], inode 0x492bba471 dinode
Jan 31 13:22:08 hostname kernel: XFS (dm-3): Unmount and run xfs_repair
Jan 31 13:22:08 hostname kernel: XFS (dm-3): First 128 bytes of corrupted metadata buffer:
Jan 31 13:22:08 hostname kernel: 00000000: 49 4e 41 c0 03 01 00 00 00 00 00 00 00 00 00 00  INA.............
Jan 31 13:22:08 hostname kernel: 00000010: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 31 13:22:08 hostname kernel: 00000020: 35 0b ad 4b c4 82 fb e4 35 0b ad 4b c4 92 3e 28  5..K....5..K..>(
Jan 31 13:22:08 hostname kernel: 00000030: 35 0b ad 4b c4 92 3e 28 00 00 00 00 00 00 00 29  5..K..>(.......)
Jan 31 13:22:08 hostname kernel: 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 31 13:22:08 hostname kernel: 00000050: 00 00 18 01 00 00 00 00 00 00 00 00 28 50 ba 4e  ............(P.N
Jan 31 13:22:08 hostname kernel: 00000060: ff ff ff ff a7 c4 d4 a1 00 00 00 00 00 00 00 07  ................
Jan 31 13:22:08 hostname kernel: 00000070: 00 00 00 1a 00 38 b3 80 00 00 00 00 00 00 00 08  .....8..........

This issuse occurs after configuring and using lvm-cache. The high-speed cache and metadata is in a nvme driver.
System version: Rocky Linux release 9.1 (Blue Onyx)

[root@hostname log]# lvs
  LV   VG     Attr       LSize   Pool          Origin       Data%  Meta%  Move Log Cpy%Sync Convert
  root rl_nas Cwi-aoC--- <14.52t [cache_cpool] [root_corig] 18.50  2.40            0.00
  swap rl_nas -wi-a-----  32.00g
[root@hostname log]# pvs
  PV             VG     Fmt  Attr PSize    PFree
  /dev/md126     rl_nas lvm2 a--    14.55t    0
  /dev/nvme0n1p2 rl_nas lvm2 a--  <412.94g 2.13g

It comes from the err of my nvme driver?

2

It is not resolved after disable lvm-cache. But it was finally solved through xfs_repair
Hours Relabeling in boot of rocky 9?

# boot into rescue mode (troubleshooting) from DVD.iso
# chose '1' to mount system as /mnt/sysroot
umount -l /mnt/sysroot
umount -l /mnt/sysimage
xfs_repair /dev/mapper/rl-root
reboot
# waiting for Relabeling complete, which may take hours. And then reboot, have fun!
3

This is used for the root file system (xfs) need to be repaired. Non-root file system ( not ‘/’ ) may not need rescue mode.