Issue with RAID on Rocky

[cxs]

Dear all, I have a serious issue with RAID on Rocky 9.8.

I am running a RAID1 with two 24TB disks.

cat /proc/mdstat

Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[2]
23437637568 blocks super 1.2 [2/2] [UU]
bitmap: 5/175 pages [20KB], 65536KB chunk

If I REMOVE 1 disk, it all works well. With 2 disks on the RAID1, I get ATA errors and the machine randomly reboots!!!

I even changed one of the disks. So I am convinced it is a motherboard/Rocky compatibility issue.

Here is what the log looks like:

Sep 24 02:06:29 CASGPUserver kernel: ata2: EH complete
Sep 24 02:06:29 CASGPUserver kernel: ata2.00: configured for UDMA/133
Sep 24 02:06:29 CASGPUserver kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Sep 24 02:06:29 CASGPUserver kernel: ata2: hard resetting link
Sep 24 02:06:29 CASGPUserver kernel: ata2.00: status: { DRDY }
Sep 24 02:06:29 CASGPUserver kernel: ata2.00: cmd 61/08:90:10:08:00/00:00:00:00:00/40 tag 18 ncq dma 4096 out
res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Sep 24 02:06:29 CASGPUserver kernel: ata2.00: failed command: WRITE FPDMA QUEUED
Sep 24 02:06:29 CASGPUserver kernel: ata2: SError: { CommWake 10B8B LinkSeq }
Sep 24 02:06:29 CASGPUserver kernel: ata2.00: exception Emask 0x0 SAct 0x40000 SErr 0x8c0000 action 0x6 frozen
Sep 24 02:06:13 CASGPUserver systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
Sep 24 02:06:03 CASGPUserver systemd[1]: Started Network Manager Script Dispatcher Service.

I get these ALL THE TIME, ever so often. Testing the disks with SMART shows NO ERRORs.

Is there a fix? Is this a kernel issue? Can I use an external hardware RAID? Which ones are compatible?

Thank you very much for all your help,
Christos.

[iwalker]

I expect this problem is exascerbated by the size of the disks and the mount of sync needed. Googling “linux software raid 1 24tb disks performance issues WRITE FPDMA QUEUED”, shows me some stuff like this:

The failed command: WRITE FPDMA QUEUED error indicates a low-level hardware or bus communication timeout. Because Native Command Queuing (NCQ) handles deep queues of concurrent commands, a hitch forces the Linux libata driver to stall and reset the link, crushing your mdadm RAID 1 write performance down to single digits or causing the array to hang. [1, 2, 3, 4, 5]

With massive 24TB disks, this behavior is magnified during heavy sequential array syncs, flushes, or heavy workloads. [1]

And recommendations:

1. Isolate the Root Cause

Run dmesg -T | grep -iE "ata|fpdma|exception" right after a slowdown to catch the exact error layout.

• If accompanied by SAct timeouts or Emask 0x4 (timeout): The drive is failing to respond to NCQ requests within the OS timeout window. This points to driver/firmware compatibility, controller stress, or power drops.
• If accompanied by SErr or BadCRC / ICRC: This confirms structural signal degradation. Data is corrupting while traveling across the physical bus. [1, 2, 3, 4]

---

2. Immediate Software Fixes

A. Force-Disable NCQ (The Common Fix)

Many high-capacity enterprise drives have quirks with specific motherboard SATA controllers under Linux NCQ implementations. Disabling NCQ forces legacy commands and usually stops the link resets: [1, 2, 3]

1. To test temporarily without a reboot, drop the queue depth to 1 (disabling queueing) for both drives (replace sdX and sdY with your actual drive letters):

   bash

   echo 1 | sudo tee /sys/block/sdX/device/queue_depth
   echo 1 | sudo tee /sys/block/sdY/device/queue_depth
   
   

   Use code with caution.

2. If performance stabilizes and errors stop, make this permanent by appending libata.force=noncq to your kernel boot parameters.

   • Edit /etc/default/grub.
   • Append to GRUB_CMDLINE_LINUX_DEFAULT: ... quiet splash libata.force=noncq.
   • Run sudo update-grub (Ubuntu/Debian) or sudo grub2-mkconfig -o /boot/grub2/grub.cfg (RHEL/Fedora) and reboot. [1, 2]

B. Adjust SATA Link Power Management (ALPM)

Modern kernels aggressively scale down SATA link voltages. Enterprise 24TB drives draw high bursts of power that can cause modern ALPM policies to trip and drop the link.
Disable ALPM link power management: [1]

bash

echo "max_performance" | sudo tee /sys/class/scsi_host/host*/link_power_management_policy

Use code with caution.

---

3. Critical Hardware Triaging

If noncq doesn’t stop the errors, 24TB mechanics require you to check physical layout constraints: [1, 2]

• Faulty SATA Cables: 24TB drives sustain extreme sequential bursts. Cheap or old SATA cables frequently cause CRC errors. Replace them with high-quality, locking SATA III cables. [1, 2]
• Underpowered 12V Rail / Splitters: 24TB Helium drives have significant rotational mass. When spinning up or executing heavy concurrent writes, they draw heavy current spikes on the 12V rail. If you are using cheap SATA power splitters or a weak power supply, the drive’s internal voltage drops, triggering a controller lockup. Do not use Molex-to-SATA splitters. [1, 2]
• SATA Controller Limitations: Many onboard consumer ports or cheap PCIe SATA expansion cards (like ASM106x chips) cannot properly negotiate or maintain deep queues with high-density enterprise commands. If possible, test by plugging them directly into native Intel/AMD CPU-bound motherboard SATA ports or an LBA-mode HBA card. [1, 2]

---

4. Check Drive Health (SMART)

Verify the physical media isn’t failing or reallocating sectors under the hood: [1]

bash

sudo smartctl -a /dev/sdX

Use code with caution.

Look closely at:

• UDMA_CRC_Error_Count → If this value is non-zero and growing, it is 100% a bad SATA cable, loose port, or bad controller.
• Reallocated_Sector_Ct or Current_Pending_Sector → If these are growing, the drive is physically failing and timing out because it’s stuck trying to read/write bad spots on the platters. [1, 2, 3, 4]

might be a good place to start. But yeah, if this is software raid connected to your motherboard it probably cannot handle it. You’d be looking at a raid controller no doubt instead. Looking at the RHEL hardware compatibility list should give you an idea of which raid controllers are supported.

You can check and filter for storage here: Search results - Red Hat Ecosystem Catalog