My server consists of a Ryzen 5 3600, Asus B550 Prime MB, 64gb ram, 2 M.2 2tb SSDs on the MB, and 12 total HDD.  4 are running off the onboard SATA, and 8 are running off of an LSI SAS2008.  I just finished up installing two new 2tb m.2 drives cache drive and configuring them in a zfs mirror cache pool.  I have an array of 8 drive with one parity.  Now that I have my new cache pool up and running, I am planning on swapping out my two oldest 2tb array disks and installing two 12tb disks in their place and reducing the number of disks in the array.  After doing that I'm planning on creating another raidz1 pool with four 2tb SSD's for a total of 8 HHDs, 4 SSDs, and 2 M.2's.

 

Years ago, when I built the system, I wasn't paying much attention to where and how I was hooking things up.  Currently I have the parity drive along with 2 other array disks running off of the MB controller and the others on the LSI controller.  Am I leaving performance on the table by having these drives split across two controllers? 

 

Another question I've always had is when doing things like moving files from one array disk to another or parity checks. Does the LSI controller move them directly between the drive without passing them up and then back down the PCI lane?

Edited February 8, 20251 yr by LavaSurfer

In theory yes... In practice, you may never see it...

 

Recommendations:

Move your parity and array disks to the LSI controller to consolidate performance management.

Keep SSDs and cache drives on the motherboard SATA ports to avoid controller bandwidth issues.

Data moving and parity checks in Unraid do pass through the PCIe bus but shouldn’t be a noticeable bottleneck in your case.

When creating your RAIDZ1 SSD pool, verify your controller and ZFS support for SSD features like TRIM.

 

 

                                                                Performance and Array Disk Distribution Across Controllers

It’s common for systems like yours to have multiple controllers, and typically, splitting drives between onboard SATA and a controller like the LSI SAS2008 isn't a big issue for Unraid. Unraid doesn’t use traditional RAID striping but instead accesses disks individually. Here's what matters most:

 

Controller Bandwidth:
Your Ryzen-based system likely runs your LSI card at PCIe 3.0 x8, which should provide more than enough bandwidth (around 6.4 GB/s) for several spinning HDDs. The LSI SAS2008 itself can handle 8x SATA or SAS drives at full speed.

 

SATA/Controller Load Distribution:
Since Unraid only reads one disk plus parity at a time during normal operations, having disks on separate controllers doesn’t impact most daily usage. However, during parity checks or high throughput operations (like large data migrations), a more balanced setup could marginally improve performance.

 

Recommendation:
If you're already swapping disks around, you might consider moving all array disks (including parity) to the LSI controller and keeping other drives like SSDs or cache pools on the motherboard SATA ports. This allows the LSI controller to handle parity and data disks efficiently, while SSDs won't saturate your onboard SATA connections.

 

                                                                    Direct Data Movement on the LSI Controller (PCI Lane Question)

LSI controllers don’t directly transfer data between drives without involving the CPU and RAM. Even though the controller can handle tasks like RAID in hardware mode, Unraid itself operates differently:

 

How Unraid Works:
Unraid is a software-defined storage system that doesn't utilize hardware RAID features. When moving files between array disks or doing a parity check, the data flows from the source disk to the CPU/memory and back to the destination disk via the PCIe bus.

 

Is This a Bottleneck?
While theoretically, there's a slight delay due to PCIe transfers, in practice, it's not noticeable for most users. PCIe 3.0 and modern CPUs are more than capable of handling this workload, especially with HDDs, which are slower than the PCIe bus can handle by a wide margin.

 

#############

 

Move off the array go pool devices only and setup zfs?

Creating a ZFS RAIDZ1 pool with 4 SSDs sounds like a good move for fast, redundant storage. Keep in mind:

 

ZFS and Controller Bandwidth:
SSDs can saturate SATA or SAS bandwidth quicker than HDDs. Ensure your SSDs are either on the motherboard's SATA ports (which may be closer to the CPU) or on a controller with sufficient bandwidth to avoid bottlenecks.

 

Trim and Compatibility:
ZFS may not always fully support TRIM on SSDs depending on your setup, especially with SAS controllers. Check your controller's support for SSD optimizations.

 

Awesome reply man!

 

I went through the manual for the MB again and I'm using the first 4 of the 6 sata ports there as when I use the second m.2 slot ports 5 & 6 are disabled.  So, I'm good there.  Also, when I go in System Devices under Tools all 4 of the SATA drive populate under the one SATA controller for the MB.  So just to make sure, that would tell me they are on the same controller?  I'm not really finding anything in the manual to directly tells me if that is the case or not.

 

As far as trim settings.  Is that something I would enable in Unraid or should I be looking in the BIOS?  Is it as simple as enabling Autotrim under the Pool settings?  Where would I look to make sure my controller supports everything needed?

Theoretical max throughput of the interface is meaningless when the storage medium itself can't hope to saturate it.

So in short, no, you would notice absolutely no difference going with sata over HBA sas....

 

so I use a LSI HBA with 3x 16 HDD. In Unriad yes they would appears as one controller as the HBA is telling it, but the HBA has other firmware that is telling the disk what to do.
My HBA is 6G meaning that it can handle crosstalk up to 6GB for a full saturated link.

 

what does  SCSI say?

 

^-HBA Thats Normal....

 

Trim is more a SSD issue. Unraid has setting to run ssd trim comands...

 

The SSD TRIM command simply marks the invalid data and tells the SSD to ignore it during the garbage collection process. The SSD then has fewer pages to move during garbage collection, which reduces the total number of program/erase cycles (P/E cycles) to the NAND flash media and prolongs the life of the SSD.

WebUI > Settings > Scheduler:

 

 

This will help SSD and NVME longevity...