Hi all,

 

I recently lost my NVMe drive that I used for Docker and now I want to replace it with 2 or 3 new drives, most likely Samsung 990 Pros in zfs z1 configuration.

But my motherboard (Gigabyte B560M AORUS PRO) only has one PCI 4.0 and one PCI 3.0 M.2 slots.. So I'd like to get a PCI to quad M.2 expansion card, but I don't think my board supports bifurcation (it's not mentioned in the manual and I couldn't find a setting in BIOS), so I'm not sure which card to buy.. Or should I start thinking about replacing the board all together?

 

Any recommendations would be appreciated, I'm really a novice when it comes to these things.. And I've just randomly stumbled upon "bifurcation" in some card's reviews.

 

This is the full setup I'm currently running:

CPU: Intel i7-11700

MB: Gigabyte B560M AORUS PRO

RAM: 32 GB DDR4

3x 8TB HDD for the Array - WD Red

2x 1TB SSD for protected cache - Crucial MX500

Most consumer boards and processors don't support bifurcation (splitting of PCIe lanes), so you need the bifurcation to happen on the expansion card itself with something called a PCIe switch. Try googling 'm.2 to pcie switch' and it should come up with some. The ones i found used the ASM2824 chipset and use PCIe 3 instead of 4 but that should get you started.

I'm not that familiar with different chipsets.. Would something like this work?

https://sabrent.com/products/ec-p4bf

Or do you have other suggestions?

 

I wanted to go with Samsung 990 Pros for their PCI 4.0 speed and zfs z1 config to boost that even further with 3 drives.. So I wouldn't want to start with a 3.0 adapter 😅

Hello,

 

I ran one of these for a while within unRAID (purchased via AliExpress):

 

http://www.linkreal.com.cn/en/products/LRNV9547L4IPCIExpressx16toQuad.html

 

Linkreal also have a x8 slot version too.

 

It's only PCIe 3.0 speeds so may not get the full performance of gen4 NVMe SSDs (I ran it with gen3 NVMe SSDs).

 

I also chose to add small NVMe heatsinks to each SSD installed on this card as it was installed in a small case with limited airflow.

 

(Have since moved my NVMe disks to a Terramaster F8 plus all SSD NAS device.)

Well, I wanted to do it for the speed, but if there are no performance benefit without a huge investment (either in really expensive PCI 4.0 expansion card or switching to a different platform that supports bifurcation), I guess I'll have to do something else..

 

Do you know if PCI or M.2 to SATA expansion cards have the same bifurcation limitation? Or would these work at full bandwidth with like 4x 2.5 SATA SSDs?

I have 2x 1TB Crucial MX500, but only one more SATA port left available on my board.. So I might just get one of these cards and add 2 or 3 more MX500 in a RAIDz1 for caching.

 

PCI to SATA:

https://a.co/d/91Y4ti7

https://a.co/d/8RS5XZx

M.2 to SATA:

https://a.co/d/5BgNzmm

https://a.co/d/8I6BYzt

On 1/14/2025 at 7:17 PM, IMDS said:

Well, I wanted to do it for the speed.

Speed is really about bandwidth and getting stuff as fast as possible to the processor. So that means checking the block diagram in the manual, which says that the x16 slot and one M.2 slot are directly connected, so an adapter card in one of those provides the most bandwidth and the most speed per drive. Everything else has to share the DMI 3.0 link to the processor, creating a potential bottleneck.

 

As far as the cards you mentioned, you'll want to check this thread. In practice bifurcation only matters for M.2 PCIe devices, and graphics cards for AI or mining applications, if you don't use those, you don't need bifurcation, just a regular controller that moves stuff from whatever you want to connect to the PCIe bus, which those cards have (and other SATA/SAS controllers as well).

 

Since you're talking SATA devices, they get a speed of max 6gbit/s which is about 0.75GB/s, PCIE 3.0 (the M.2 slot) can transfer a little less than 1GB/s so generally you'll need as much PCIe lanes as you have drives to have everything working at full speed. For many of the cards, they only have an x1 or x2 connection, so it works for a few drives, but more than 2 drives, and you'll notice a bottleneck where things don't go as fast as they can.

 

Thanks @Wody!

The thing is that I have my x16 slot free, as I'm only using an integrated gpu, so I can do a faster card with 4 SATA SSDs at full speed, right?

 

Here's a diagram of my board:

 

And this is what I'm currently planning to do, so please let me know if it makes sense:

1. Keep my 3x 8TB HDD xfs array on motherboard's SATA slots (maybe expand it with 1 more drive if I run out of space, but I don't think that will happen any time soon)

2. Get an expansion card in the x16 PCI slot for 4x 1TB SATA SSDs in RAIDz1 as cache for the array (Or 6x 1TB in RAIDz2). I plan to have them set to run mover only if more than 80% full, which I shouldn't ever hit, or if files are older than a month.

3. Get 1x Samsung 990 Pro 2TB NVMe in the first PCI 4.0 slot for appdata, running zfs, so I can take snapshots and push them to the SSD pool.

 

That should all run at full speed with a proper card? (An actual card suggestion would be great 😅)

But also, does it make sense to run appdata snapshots like I suggested or should I just keep using appdata backup plugin?

And is it ok to run mover like suggested for a server with just nextcloud and plex type of media?

7 hours ago, IMDS said:

Thanks @Wody!

The thing is that I have my x16 slot free, An actual card suggestion would be great 😅)

One of the things I like about Unraid is the reliability, and the ability to run on almost anything. So when I had this issue, I didn't want to rely on the capabilities of the motherboard at all, so that when it failed, I could switch it out for another one and keep going. So instead of worrying about sata-slots, I got a SAS-controller card, which meant for sas/sata drives its full speed all the time, and because of much bigger buffers and queues, most of the data doesn't leave the card so the PCIe speed doesn't matter a lot either.

 

So, in your situation, since you're talking about 9 SATA drives, and 1 NVMe drive, I'd get something like a 9305-16i or 9400-16i or newer (the 9300-16i uses too much power and may get too warm, but an 9300-8i may do as well), put that in the 16x slot, and the NVMe in the M.2 slot. Only issue would be if you still had sata1 drives (1.5gbit), those aren't supported.

You'd be freeing up more bandwidth in the DMI link as well, and if you do need more drives, you can then get SAS drives as well which are often cheaper because companies get rid of them when they run out of warranty, or because of a yearly replacement.