Hi. I hope I am in the right place. To cut to the chase, my server has 20 spinning disks installed; however, when I type lsblk, the output reports that there are 40 disks. I check the BMC and it reports that only 20 are installed. I’ve never seen anything like this before so I don’t even know where to begin to troubleshoot this. Has anyone ever experienced anything like this?
The first thing to do would be to post the actual output from lsblk.
Hi Frank. Thanks so much for the response. Here is the output. (According to the populated slots on my motherboard, I would have expected to see only sda → sdt).
lsblk
[output]
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 14.6T 0 disk
sdb 8:16 0 14.6T 0 disk
sdc 8:32 0 14.6T 0 disk
sdd 8:48 0 14.6T 0 disk
sde 8:64 0 14.6T 0 disk
sdf 8:80 0 14.6T 0 disk
sdg 8:96 0 14.6T 0 disk
sdh 8:112 0 14.6T 0 disk
sdi 8:128 0 14.6T 0 disk
sdj 8:144 0 14.6T 0 disk
sdk 8:160 0 14.6T 0 disk
sdl 8:176 0 14.6T 0 disk
sdm 8:192 0 14.6T 0 disk
sdn 8:208 0 14.6T 0 disk
sdo 8:224 0 14.6T 0 disk
sdp 8:240 0 14.6T 0 disk
sdq 65:0 0 14.6T 0 disk
sdr 65:16 0 14.6T 0 disk
sds 65:32 0 14.6T 0 disk
sdt 65:48 0 14.6T 0 disk
sdu 65:64 0 14.6T 0 disk
sdv 65:80 0 14.6T 0 disk
sdw 65:96 0 14.6T 0 disk
sdx 65:112 0 14.6T 0 disk
sdy 65:128 0 14.6T 0 disk
sdz 65:144 0 14.6T 0 disk
sdaa 65:160 0 14.6T 0 disk
sdab 65:176 0 14.6T 0 disk
sdac 65:192 0 14.6T 0 disk
sdad 65:208 0 14.6T 0 disk
sdae 65:224 0 14.6T 0 disk
sdaf 65:240 0 14.6T 0 disk
sdag 66:0 0 14.6T 0 disk
sdah 66:16 0 14.6T 0 disk
sdai 66:32 0 14.6T 0 disk
sdaj 66:48 0 14.6T 0 disk
sdak 66:64 0 14.6T 0 disk
sdal 66:80 0 14.6T 0 disk
sdam 66:96 0 14.6T 0 disk
sdan 66:112 0 14.6T 0 disk
sdao 66:128 1 15.1G 0 disk
├─sdao1 66:129 1 2G 0 part
└─sdao2 66:130 1 9.6M 0 part
The first bunch are using device driver ‘8’, which is normal, but then the next two sets are using ‘65’ and ‘66’. You may be able to identify by using
ls -l /dev
You can also try a different view
lsblk --nodeps --paths --output NAME,MAJ:MIN,MODEL,SERIAL
Interesting new development. I have CentOS 8.3 kicking around, so I installed it. Immediately after first boot. The drives look just fine. (I’ll include the system SSDs etc this time).
lsblk
[output]
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 14.6T 0 disk
sdb 8:16 0 14.6T 0 disk
sdc 8:32 0 14.6T 0 disk
sdd 8:48 0 14.6T 0 disk
sde 8:64 0 14.6T 0 disk
sdf 8:80 0 14.6T 0 disk
sdg 8:96 0 14.6T 0 disk
sdh 8:112 0 14.6T 0 disk
sdi 8:128 0 14.6T 0 disk
sdj 8:144 0 14.6T 0 disk
sdk 8:160 0 14.6T 0 disk
sdl 8:176 0 14.6T 0 disk
sdm 8:192 0 14.6T 0 disk
sdn 8:208 0 14.6T 0 disk
sdo 8:224 0 14.6T 0 disk
sdp 8:240 0 14.6T 0 disk
sdq 65:0 0 14.6T 0 disk
sdr 65:16 0 14.6T 0 disk
sds 65:32 0 14.6T 0 disk
sdt 65:48 0 14.6T 0 disk
sdu 65:64 1 15.1G 0 disk
├─sdu1 65:65 1 8.6G 0 part
└─sdu2 65:66 1 10M 0 part
sdv 65:80 0 1.9T 0 disk
├─sdv1 65:81 0 2G 0 part
│ └─md126 9:126 0 2G 0 raid1 /boot
└─sdv2 65:82 0 1.9T 0 part
└─md127 9:127 0 1.9T 0 raid1
├─cl_terrastr4-root 253:0 0 1024G 0 lvm /
├─cl_terrastr4-swap 253:1 0 5G 0 lvm [SWAP]
├─cl_terrastr4-var 253:2 0 726.6G 0 lvm /var
└─cl_terrastr4-home 253:3 0 150G 0 lvm /home
sdw 65:96 0 1.9T 0 disk
├─sdw1 65:97 0 2G 0 part
│ └─md126 9:126 0 2G 0 raid1 /boot
└─sdw2 65:98 0 1.9T 0 part
└─md127 9:127 0 1.9T 0 raid1
├─cl_terrastr4-root 253:0 0 1024G 0 lvm /
├─cl_terrastr4-swap 253:1 0 5G 0 lvm [SWAP]
├─cl_terrastr4-var 253:2 0 726.6G 0 lvm /var
└─cl_terrastr4-home 253:3 0 150G 0 lvm /home
nvme0n1 259:0 0 5.8T 0 disk
nvme1n1 259:1 0 5.8T 0 disk
Then I install the latest updates and reboot and it’s back to counting every spinning disk twice. WTH? I wonder if I have messed up hardware? It’s not counting every solid state disk twice. Here’s the latest output.
lsblk
[output]
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 15.1G 0 disk
├─sda1 8:1 1 8.6G 0 part
└─sda2 8:2 1 10M 0 part
sdb 8:16 0 14.6T 0 disk
sdc 8:32 0 14.6T 0 disk
sdd 8:48 0 14.6T 0 disk
sde 8:64 0 14.6T 0 disk
sdf 8:80 0 14.6T 0 disk
sdg 8:96 0 14.6T 0 disk
sdh 8:112 0 14.6T 0 disk
sdi 8:128 0 14.6T 0 disk
sdj 8:144 0 14.6T 0 disk
sdk 8:160 0 14.6T 0 disk
sdl 8:176 0 14.6T 0 disk
sdm 8:192 0 14.6T 0 disk
sdn 8:208 0 14.6T 0 disk
sdo 8:224 0 14.6T 0 disk
sdp 8:240 0 14.6T 0 disk
sdq 65:0 0 14.6T 0 disk
sdr 65:16 0 14.6T 0 disk
sds 65:32 0 14.6T 0 disk
sdt 65:48 0 14.6T 0 disk
sdu 65:64 0 14.6T 0 disk
sdv 65:80 0 14.6T 0 disk
sdw 65:96 0 14.6T 0 disk
sdx 65:112 0 14.6T 0 disk
sdy 65:128 0 14.6T 0 disk
sdz 65:144 0 14.6T 0 disk
sdaa 65:160 0 14.6T 0 disk
sdab 65:176 0 14.6T 0 disk
sdac 65:192 0 14.6T 0 disk
sdad 65:208 0 14.6T 0 disk
sdae 65:224 0 14.6T 0 disk
sdaf 65:240 0 14.6T 0 disk
sdag 66:0 0 14.6T 0 disk
sdah 66:16 0 14.6T 0 disk
sdai 66:32 0 14.6T 0 disk
sdaj 66:48 0 14.6T 0 disk
sdak 66:64 0 14.6T 0 disk
sdal 66:80 0 14.6T 0 disk
sdam 66:96 0 14.6T 0 disk
sdan 66:112 0 14.6T 0 disk
sdao 66:128 0 14.6T 0 disk
sdap 66:144 0 1.9T 0 disk
├─sdap1 66:145 0 2G 0 part
│ └─md126 9:126 0 2G 0 raid1 /boot
└─sdap2 66:146 0 1.9T 0 part
└─md127 9:127 0 1.9T 0 raid1
├─cl_terrastr4-root 253:0 0 1024G 0 lvm /
├─cl_terrastr4-swap 253:1 0 5G 0 lvm [SWAP]
├─cl_terrastr4-var 253:2 0 726.6G 0 lvm /var
└─cl_terrastr4-home 253:3 0 150G 0 lvm /home
sdaq 66:160 0 1.9T 0 disk
├─sdaq1 66:161 0 2G 0 part
│ └─md126 9:126 0 2G 0 raid1 /boot
└─sdaq2 66:162 0 1.9T 0 part
└─md127 9:127 0 1.9T 0 raid1
├─cl_terrastr4-root 253:0 0 1024G 0 lvm /
├─cl_terrastr4-swap 253:1 0 5G 0 lvm [SWAP]
├─cl_terrastr4-var 253:2 0 726.6G 0 lvm /var
└─cl_terrastr4-home 253:3 0 150G 0 lvm /home
nvme1n1 259:0 0 5.8T 0 disk
nvme0n1 259:1 0 5.8T 0 disk
As requested:
lsblk --nodeps --paths --output NAME,MAJ:MIN,MODEL,SERIAL
[output]
NAME MAJ:MIN MODEL SERIAL
/dev/sda 8:0 SAMBO 0000000000000373
/dev/sdb 8:16 MG08SCA16TE 5150A08ZFVAG
/dev/sdc 8:32 MG08SCA16TE 5140A043FVAG
/dev/sdd 8:48 MG08SCA16TE 5140A03PFVAG
/dev/sde 8:64 MG08SCA16TE 5140A0H7FVAG
/dev/sdf 8:80 MG08SCA16TE 5140A0JFFVAG
/dev/sdg 8:96 MG08SCA16TE 5150A04EFVAG
/dev/sdh 8:112 MG08SCA16TE 5150A03BFVAG
/dev/sdi 8:128 MG08SCA16TE 5140A0K1FVAG
/dev/sdj 8:144 MG08SCA16TE 5140A042FVAG
/dev/sdk 8:160 MG08SCA16TE 5150A0AGFVAG
/dev/sdl 8:176 MG08SCA16TE 5140A0J1FVAG
/dev/sdm 8:192 MG08SCA16TE 5140A06LFVAG
/dev/sdn 8:208 MG08SCA16TE 5150A09MFVAG
/dev/sdo 8:224 MG08SCA16TE 5150A0A4FVAG
/dev/sdp 8:240 MG08SCA16TE 5140A0JNFVAG
/dev/sdq 65:0 MG08SCA16TE 5150A07QFVAG
/dev/sdr 65:16 MG08SCA16TE 5150A02GFVAG
/dev/sds 65:32 MG08SCA16TE 5150A07MFVAG
/dev/sdt 65:48 MG08SCA16TE 5140A0KAFVAG
/dev/sdu 65:64 MG08SCA16TE 5140A03VFVAG
/dev/sdv 65:80 MG08SCA16TE 5150A08ZFVAG
/dev/sdw 65:96 MG08SCA16TE 5140A043FVAG
/dev/sdx 65:112 MG08SCA16TE 5140A03PFVAG
/dev/sdy 65:128 MG08SCA16TE 5140A0H7FVAG
/dev/sdz 65:144 MG08SCA16TE 5140A0JFFVAG
/dev/sdaa 65:160 MG08SCA16TE 5150A04EFVAG
/dev/sdab 65:176 MG08SCA16TE 5150A03BFVAG
/dev/sdac 65:192 MG08SCA16TE 5140A0K1FVAG
/dev/sdad 65:208 MG08SCA16TE 5140A042FVAG
/dev/sdae 65:224 MG08SCA16TE 5150A0AGFVAG
/dev/sdaf 65:240 MG08SCA16TE 5140A0J1FVAG
/dev/sdag 66:0 MG08SCA16TE 5140A06LFVAG
/dev/sdah 66:16 MG08SCA16TE 5150A09MFVAG
/dev/sdai 66:32 MG08SCA16TE 5150A0A4FVAG
/dev/sdaj 66:48 MG08SCA16TE 5140A0JNFVAG
/dev/sdak 66:64 MG08SCA16TE 5150A07QFVAG
/dev/sdal 66:80 MG08SCA16TE 5150A02GFVAG
/dev/sdam 66:96 MG08SCA16TE 5150A07MFVAG
/dev/sdan 66:112 MG08SCA16TE 5140A0KAFVAG
/dev/sdao 66:128 MG08SCA16TE 5140A03VFVAG
/dev/sdap 66:144 Micron_1300_MTFD 21032CA53502
/dev/sdaq 66:160 Micron_1300_MTFD 21032CA534FA
/dev/nvme1n1 259:0 Micron_9300_MTFDHAL6T4TDR 21152E429D30
/dev/nvme0n1 259:1 Micron_9300_MTFDHAL6T4TDR 21152E429D52
SAS drives, but just single link? Should not have “multipath”. (Some SAS drives have dual SAS for redundancy?)
There are subdirs in /dev/disk/ that have more elaborate names than the /dev/sd* Persistent block device naming - ArchWiki
What are those drivers “65” and “66”? How to find the driver (module) associated with a device on Linux? - Unix & Linux Stack Exchange
[EDIT] Alas, seem to be “sd” like the “8”: Major and Minor Numbers - Linux Device Drivers, Second Edition [Book]
[EDIT2] Ahh, each can have only 16 drives: https://www.kernel.org/doc/Documentation/admin-guide/devices.txt
Thanks for all the links! I will check them out. In the meantime, this is interesting. Here’s the whole works mapped by path:
cd /dev/disk; ls -l by-path/
[output]
total 0
lrwxrwxrwx. 1 root root 9 Jan 18 11:34 pci-0000:00:14.0-usb-0:1:1.0-scsi-0:0:0:0 → …/…/sda
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:00:14.0-usb-0:1:1.0-scsi-0:0:0:0-part1 → …/…/sda1
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:00:14.0-usb-0:1:1.0-scsi-0:0:0:0-part2 → …/…/sda2
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:00:17.0-ata-1 → …/…/sdap
lrwxrwxrwx. 1 root root 11 Jan 18 11:34 pci-0000:00:17.0-ata-1-part1 → …/…/sdap1
lrwxrwxrwx. 1 root root 11 Jan 18 11:34 pci-0000:00:17.0-ata-1-part2 → …/…/sdap2
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:00:17.0-ata-2 → …/…/sdaq
lrwxrwxrwx. 1 root root 11 Jan 18 11:34 pci-0000:00:17.0-ata-2-part1 → …/…/sdaq1
lrwxrwxrwx. 1 root root 11 Jan 18 11:34 pci-0000:00:17.0-ata-2-part2 → …/…/sdaq2
lrwxrwxrwx. 1 root root 13 Jan 18 11:34 pci-0000:18:00.0-nvme-1 → …/…/nvme0n1
lrwxrwxrwx. 1 root root 13 Jan 18 11:34 pci-0000:19:00.0-nvme-1 → …/…/nvme1n1
lrwxrwxrwx. 1 root root 9 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy0-lun-0 → …/…/sdv
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy10-lun-0 → …/…/sdaf
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy11-lun-0 → …/…/sdag
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy12-lun-0 → …/…/sdah
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy13-lun-0 → …/…/sdai
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy14-lun-0 → …/…/sdaj
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy15-lun-0 → …/…/sdak
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy16-lun-0 → …/…/sdal
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy17-lun-0 → …/…/sdam
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy18-lun-0 → …/…/sdan
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy19-lun-0 → …/…/sdao
lrwxrwxrwx. 1 root root 9 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy1-lun-0 → …/…/sdw
lrwxrwxrwx. 1 root root 9 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy2-lun-0 → …/…/sdx
lrwxrwxrwx. 1 root root 9 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy3-lun-0 → …/…/sdy
lrwxrwxrwx. 1 root root 9 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy4-lun-0 → …/…/sdz
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy5-lun-0 → …/…/sdaa
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy6-lun-0 → …/…/sdab
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy7-lun-0 → …/…/sdac
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy8-lun-0 → …/…/sdad
lrwxrwxrwx. 1 root root 10 Jan 18 11:34 pci-0000:3b:00.0-sas-exp0x5003048029fc3b3f-phy9-lun-0 → …/…/sdae
Just confirmed that all 20 SAS drives are listed twice (sorted by serial number and everything matches up). So now I need to figure out a way to tell the OS that it should only list each disk once… This has to be an OS problem, right? This can’t be some kind of hardware, bios, cable, adapter, firmware issue, could it?
lshw -class disk
[output]
…
[first disk]
*-disk:0
description: SCSI Disk
product: MG08SCA16TE
vendor: TOSHIBA
physical id: 0.0.0
bus info: scsi@7:0.0.0
logical name: /dev/sdb
version: 0101
serial: 5150A08ZFVAG
size: 14TiB (16TB)
capacity: 152TiB (168TB)
capabilities: 7200rpm
configuration: ansiversion=6 logicalsectorsize=512 sectorsize=4096
*-disk:13
description: SCSI Disk
product: MG08SCA16TE
vendor: TOSHIBA
physical id: 0.17.0
bus info: scsi@7:0.23.0
logical name: /dev/sdv
version: 0101
serial: 5150A08ZFVAG
size: 14TiB (16TB)
capacity: 152TiB (168TB)
capabilities: 7200rpm
configuration: ansiversion=6 logicalsectorsize=512 sectorsize=4096
…
[20th disk]
*-disk:22
description: SCSI Disk
product: MG08SCA16TE
vendor: TOSHIBA
physical id: 0.1f.0
bus info: scsi@7:0.31.0
logical name: /dev/sdad
version: 0101
serial: 5140A042FVAG
size: 14TiB (16TB)
capacity: 152TiB (168TB)
capabilities: 7200rpm
configuration: ansiversion=6 logicalsectorsize=512 sectorsize=4096
*-disk:39
description: SCSI Disk
product: MG08SCA16TE
vendor: TOSHIBA
physical id: 0.8.0
bus info: scsi@7:0.8.0
logical name: /dev/sdj
version: 0101
serial: 5140A042FVAG
size: 14TiB (16TB)
capacity: 152TiB (168TB)
capabilities: 7200rpm
configuration: ansiversion=6 logicalsectorsize=512 sectorsize=4096
Tried an experiment. Installed CentOS 8.3. Everything is cool. No duplicate drives listed after multiple reboots. Then I ran dnf update kernel. This installed only the following RPMS:
- kernel-4.18.0-348.7.1.el8_5.x86_64
- kernel-core-4.18.0-348.7.1.el8_5.x86_64
- kernel-modules-4.18.0-348.7.1.el8_5.x86_64
And then all hell breaks loose and the SAS drives are listed twice. So it’s the kernel update that triggers this. Not sure what to do about this…
EDIT 1
When I rebooted the older kernel (4.18.0-240.el8.x86_64), everything is fine and the drives map properly. So… is this a kernel bug?
You could try adding elrepo repository to Rocky, and then use the kernel-ml (mainline), which should be far newer, 5.x kernel. Then see if it returns to normal or not.
I have tried googling to find out if there is a way to load the kernel module for that particular disk with a kernel option that might have helped return it to displaying only once, unfortunately my google skills this morning must be a little weak, or not a lot of people have been experiencing this.
If you find the 5.x kernel from elrepo works, then it would then at least explain there was a minor regression that was later fixed in a newer kernel. I believe kernel-lt is 5.4 and kernel-ml is 5.16 currently.
Direct links to rpms when checking version: Index of /linux/kernel/el8/x86_64/RPMS but probably easier using the addrepo in the first elrepo link.
1 Like
CentOS had kernel 4.10.0-193, didn’t it? You have it still installed and/or can check with sudo dnf history list kernel
The RPMs have changelog, so I would check how many lines there are in the log before the “good” kernel:
rpm -q --changelog kernel-4.18.0-348.7.1.el8_5 | grep -n 4.18.0-193
Then one can look the part of log that is newer than the good kernel with something like:
rpm -q --changelog kernel-4.18.0-348.7.1.el8_5 | head -n 48941
But, is it “sas”, “scsi”, “sd”, “block”, or what that we should look from there?
Thing is, if RHEL 8.5 kernel has a regression, then it does affect users of RHEL. One could serch/file a bug in/to bugzilla.redhat.com
2 Likes