I'm busy speccing out some upgrades for my server and I'm currently working out what options would best suit me for my NVMe configuration.

For reference, the other specs are as follows:

 

• CPU: Intel Core i5 12400
• Motherboard: Asus TUF GAMING B760M-PLUS WIFI II
• Memory: 32GB Crucial Pro 6000mhz unbuffered dual channel memory

 

The motherboard has 2 x PCIe gen 4.0 x4 NVMe slots, and 1 gen 4.0 x2 slot.

 

I'm planning to use my NVMe drives as a ZFS pool to host 1-2 VM's, about 20 dockers, and to act as my array cache. I have a separate ssd for Plex. I figure my options are:

1. 2 x NVMe drives in the x4 slots in a mirror configuration or;
2. 3 x NVMe drives; 2 in the x4 slots and 1 in the x2 slot, probably in a RAIDz1 configuration

 

For option 2, how much of an issue would it be to lose 50% of the bandwidth of the 3rd slot? I'm not looking at top tier SSD's and the majority of the mid-range gen 4 SSDs I've looked at seem to support 4000-5000 MB/s read/write speeds. As the x2 slot should support 4 GB/s in theory I should be ok with that, and the 3 drive RAIDz1 configuration should still outperform a 2 drive mirror configuration, even if I lose a small amount of performance on the 3rd drive.

 

Does my logic make sense, or am I missing something, or does anyone else have any better ideas? I'm also trying to stick to a budget so want to avoid over-engineering my solution if I can.

While I don't know what drives do performance-wise when read and write are happening at the same time (does anyone benchmark that?), that's almost surely where you would find x2 a limitation (if there's heavy read AND write operations happening).

 

What are you doing where you think there is any benefit to RAIDz1? Is it just capacity? I'm not following super closely, but from chatter I'd not be choosing ZFS myself for this use case at all even on a mirror. Additionally, and if I'm not missing information (which I may well be), in the case of a failed drive RAIDz1 puts a pretty big hit on performance; a simple mirror wouldn't hit as hard on a drive loss, though rebuild certainly takes some (or at least, my past experiences with it suggest such...)

I'm not doing any kind of production workloads - only home-lab stuff, so I'm pretty comfortable with the risk and impact of losing a drive in the RAIDz1 pool. My reason for considering it is mainly that of practicality - I have a spare NVMe slot and RAIDz1 is the only way I can think to take advantage of it to increase the capacity of my cache.

I use an nvme > 6 port sata adapter - notably more power efficient than the 8 port hba I was using initially. They do have other uses. And no need to fill every slot. Obviously in the end it's up to you.