I have some questions when building a system for maximum throughput and using a cache drive pool.  Essentially I think the underlying question is does 'where' you connect a drive matter much.  Probably not much for platter based drives, but SSD and M.2 drives are getting pretty fast, so could have an affect?

 

And is it worth investing in one of the new 12gb/s Motherboards (probably not...because...)

 

LAN throughput becomes a gating factor (who's ready to upgrade to 10gb/s!).

 

I'm conscious (but not fully educated) on the PCI bus structure and how data flows across the 16 channels, etc.  But I'm not aware how these all interact across the Mobo Sata ports and such.

 

Cache Drive Config

1. Better to use Cache Drives on Mobo (6gb/s) Ports or on 6GB/S PCI Sas Card?  Both MOBO and SAS HBA PCIe would be shared with Cache Drives?

2. If Cache on shared SAS card, is it better to put Cache drives on the same SAS controller or different cache controllers?

3. Better to use Cache Drives on Mobo (6gb/s) ports in a Raid 1 Bios created Raid (i.e. unraid sees one drive, BIOS handles redundancy) or let unRaid build the cache pool?

4. Better to separate Cache Drives - 1 on Mobo and 1 on SAS card?

 

Array Drive Config

1. Better to use Mobo (6gb/s) ports or 6gb/s PCI SAS Card?

2. Is there any sense to strategically placing drives based on where data is stored accessed considering Mobo ports and 1 to many SAS cards?

 

For instance, in my system, I have a SYBA Dual port SSD Caddy (6gb/s).  It can configure the drives in a hardware Raid 1, or the BIOS can configure the Raid 1 or I can let unraid do it.  Given Raid 1 is total redundancy, seems like having the cache drives on separate controllers or sas controller/mobo would make sense?

 

 

 

 

An overall answer:

 

Unraid is good at a lot of things, even has several things going for it that competitors have not.. Hower. Raw speed is not one of its strong points. The pure fact that you state in your first line that you are going for "maximum throughput" makes me wonder if unraid is for you to be honoust..

 

As far as your data drives (so not cache) is concerned, is of no importance how you connect them, the bottleneck is the unraid driver. This is the case as long as you want the array to be protected for drive failure by parity protection. When you do not do this, speed is as fast as your hardware. I can however not imagine that you want to run without parity protection.

 

As far as the SSD's are concerned. They will be as fast as you can connect them, so M2 will be faster then SATA. I am not sure if M2 will work in unraid though (I think they will but no experience myself).

 

If raw performance is important to you there are far better ways to achieve this... Think of raid arrays that distribute data and parity over a lot of drives, that will give you significant speed increase. Alas, that is not what unraid is for (unless you would want to build a hardware raid underneath of unraid..

 

Read up !

 

Most of us here are perfectly happy with SSD's for cache drive and SATA spinners for Data drives.

I have some questions when building a system for maximum throughput and using a cache drive pool.  Essentially I think the underlying question is does 'where' you connect a drive matter much.  Probably not much for platter based drives, but SSD and M.2 drives are getting pretty fast, so could have an affect?

 

And is it worth investing in one of the new 12gb/s Motherboards (probably not...because...)

 

LAN throughput becomes a gating factor (who's ready to upgrade to 10gb/s!).

 

I'm conscious (but not fully educated) on the PCI bus structure and how data flows across the 16 channels, etc.  But I'm not aware how these all interact across the Mobo Sata ports and such.

 

Cache Drive Config

1. Better to use Cache Drives on Mobo (6gb/s) Ports or on 6GB/S PCI Sas Card?  Both MOBO and SAS HBA PCIe would be shared with Cache Drives?

2. If Cache on shared SAS card, is it better to put Cache drives on the same SAS controller or different cache controllers?

3. Better to use Cache Drives on Mobo (6gb/s) ports in a Raid 1 Bios created Raid (i.e. unraid sees one drive, BIOS handles redundancy) or let unRaid build the cache pool?

4. Better to separate Cache Drives - 1 on Mobo and 1 on SAS card?

 

Array Drive Config

1. Better to use Mobo (6gb/s) ports or 6gb/s PCI SAS Card?

2. Is there any sense to strategically placing drives based on where data is stored accessed considering Mobo ports and 1 to many SAS cards?

 

For instance, in my system, I have a SYBA Dual port SSD Caddy (6gb/s).  It can configure the drives in a hardware Raid 1, or the BIOS can configure the Raid 1 or I can let unraid do it.  Given Raid 1 is total redundancy, seems like having the cache drives on separate controllers or sas controller/mobo would make sense?

I've been through a similar line of thinking and came to the conclusion I was over-thinking things.  I found that in my case my motherboard SATA ports are slightly faster than my SAS2LP, but there are so many variables I don't think you can generalize an answer.  It's dependent on your motherboard, PCIe version supported, motherboard PCIe lanes, memory performance, the particular SAS card, etc.  And unless you're making a big mistake like putting lots of drives on a PCI SATA card or putting SSDs on SATA1 ports I think it's unlikely that the setup you pick will have huge impact - and if it does we'd need to diagnose  that particular situation rather than re-write the generalization.

 

That said, johnnie.black has some some great work characterizing the maximum throughput of various SAS/SATA cards, that's worth looking for.  And tuning for VM performance is a lot trickier than setting up the storage array.