Jump to content
billington.mark

4x NVMe drives on single PCI-e expansion card..... (UPDATE, IT WORKS!)

31 posts in this topic Last Reply

Recommended Posts

So I spotted this today... a single slot x16 PCI-e expansion card for 4x NVMe drives:

 

https://www.scan.co.uk/products/asus-hyper-m2-card-pcie-30-x16-4x-m2-pcie-2242-60-80-110-slots-intel-vroc-support-for-asus-x299-moth

https://www.asus.com/uk/Motherboard-Accessory/HYPER-M-2-X16-CARD/

 

Does anyone have one of these running on a non-X299 board? Would be very very useful if thats the case!  

At that price im tempted to just get it and see if it works anyway...

 

Ideally it would still split each NVMe into its own IOMMU group too.. but i think im asking a bit much there!

Edited by billington.mark

Share this post


Link to post

considering this a buy too. I guess as long as your mainboard support PCIe bifurcation (check your bios) it will work and split the lanes up.

on top of that individual iommu groups would be great. tell me more about it !:) not many reports out there....btw anybody with supermicro 2xnvme >pcie adapter experience here ?

Share this post


Link to post

I get the option to change the lane layout for my x16 slots (x16 or x8x8 or x4x4x4x4), so im hoping its supported.... 

Ive ordered the part and i'll report back once ive fitted it. 

 

For anyone curious this is the setup im using:

 

ASRock - EP2C602-4L/D16

2x E5-2670

Share this post


Link to post
On 5/26/2018 at 8:15 PM, billington.mark said:

I get the option to change the lane layout for my x16 slots (x16 or x8x8 or x4x4x4x4), so im hoping its supported.... 

Ive ordered the part and i'll report back once ive fitted it. 

 

For anyone curious this is the setup im using:

 

ASRock - EP2C602-4L/D16

2x E5-2670

 

In that case it will work without a problem! the Asus Hyper M2 is a really nice affordable piece of hardware, I did not check if my bios supported PCIe bifurcation  so i ended up buying 3 of them. they are not that much more expensive as a card for 1 M2 SSD and they are way nicer! (I had PCIe issues with cheap Chinese PCIe to M2 cards)

Share this post


Link to post

Sounds like this card (or multiple) will work on some X399 ThreadRipper motherboards even:

Share this post


Link to post

From my google-fu on this, it seems PCI-e bifurcation is a feature in most CPUs from the last few years, its just a case of your BIOS giving you the option to configure the x16 slot(s) to 4x4x4x4x, which only seems to be available in higher-end gaming or workstation motherboards. 

 

Its a public holiday in the UK today, so I'll not get this delivered until later in the week to test unfortunately. :(

Share this post


Link to post

So its arrived and fitted, and.. Success!

You DO NOT need an x299 based motherboard for this card to work, only a montherboard\cpu capable of splitting an x16 slot into 4x4x4x4x. 

This was found in my PCIe settings in the BIOS. 

 

Even better, each NVMe device is still in its own IOMMU group: 

 

IOMMU group 47:	[144d:a802] 81:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01)
IOMMU group 48:	[144d:a808] 82:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981

Only thing i had to to after installing was to update a VMs XML which has the 970evo passed through to it with the new PCIe address. 

 

So, ive free'd up a PCIe slot, and have the option to add 2 more NVMe devices in the future without much hastle at all. :)

IMG_20180531_184402.jpg

IMG_20180531_184850.jpg

IMG_20180531_185101.jpg

  • Upvote 1

Share this post


Link to post

has anyone some success with this or similar card on a Supermicro X9 board? 

Share this post


Link to post
1 hour ago, uldise said:

has anyone some success with this or similar card on a Supermicro X9 board? 

 

What model board do you have?

Share this post


Link to post
18 minutes ago, billington.mark said:

 

What model board do you have?

i have Supermicro X9DRH-7F..

Share this post


Link to post
5 minutes ago, uldise said:

i have Supermicro X9DRH-7F..

 

Page 4-10 of your motherboard manual....

Clearly says that bifurcation is supported on a couple of your PCIe slots.

 

Check your BIOS settings and see if you have the option to change things on your x16 slots. 

 

Share this post


Link to post
2 hours ago, billington.mark said:

Page 4-10 of your motherboard manual....

i think it is not so clear. i asked SM support some time ago about their AOC-SLG3-2M2 which is very similar and also need PCIe bifurcation to work properly. Look at the answer:

 

I'm sorry to tell you that you are not able to use the NVMe SSD with your current system since our X9 series only has BIOS that supports EFI only or Legacy only. You can try it out with your own discretion.

To use NVMe, you must have PCI-e slot where the NVME SSD installed with EFI enabled. Otherwise, the system will not detect it, usually there is an option to set only one slot as EFI if you need Legacy for your other cards, but since this X9 board is relatively old it must not have that feature. Our X9 motherboard series only has BIOS that supports either Legacy mode only or EFI mode only, you can’t combined.

Add-on Card AOC-SLG3-2M2 only compatible with the following motherboards as shown below. It will not work with any other motherboards and you can try it out with your own discretion.

https://www.supermicro.com/products/accessories/addon/AOC-SLG3-2M2.cfm

i'm not sure about my other cards: i have one IBM 1015 and one QLogic BR-1020 10Gbit Ethernet card.  

Share this post


Link to post

anyone else with Supermicro X9 and PCIe bifurcation?

Share this post


Link to post

I think they're getting at the point that you won't be able to boot from an nvme device as it won't be detected as a bootable device in the bios... Which if you're using it with unraid, that's not an issue.

 

That's how I read that response anyway.

Share this post


Link to post

yes, this was a question a asked them next: "if i don't need NVME SSD as boot device, but just wanna use it as VMWare ESXi datastore, do i still need a bios support for it?" and they replied: "Unfortunately, no."

so i really did not understand their answer, and was not asked any further..

Share this post


Link to post

You need Bios support for PCIe bifurcation without only the first NVMe SSD will work.

 

Bios support for NVMe is an different thing and would be needed if you want to boot from your NVMe drive, since unRAID boots from USB NVMe boot support is not needed!

Share this post


Link to post
Did you dig further into this with the X9 board? Curious to know the result.
+1

Sent from my SM-G955F using Tapatalk

Share this post


Link to post

I went ahead and picked up one of these and enabled bifurcation on PCIe slot 5 (CPU2 Slot5).  I setup 8x 8x since I'm only using 2 SSD's for now.  Unraid doesn't detect these drives so I must be doing something wrong.. 

 

Page 4-10 of the manual states, this option isn't present on the latest bios of 3.3

IIO 1 PCIe Port Bifurcation Control This submenu configures the following IO PCIe Port Bifurcation Control settings for IIO 1 PCIe ports to determine how the available PCI-Express lanes to be distributed between the PCI-Exp. Root Ports

 

 

Board: X9DRi-LN4F+

Manual: https://www.supermicro.com/manuals/motherboard/C606_602/MNL-1258.pdf

 

Share this post


Link to post

Have you tried running in x4/x4/x4/x4 mode to see if that fixes the issue?


If your on a slot that doesn't have that option, you may have the m.2s connected to the "second half" of the x8 instead of the "first half", changing the slots the m.2s are connected to may fix the issue.

Share this post


Link to post
27 minutes ago, tjb_altf4 said:

Have you tried running in x4/x4/x4/x4 mode to see if that fixes the issue?


If your on a slot that doesn't have that option, you may have the m.2s connected to the "second half" of the x8 instead of the "first half", changing the slots the m.2s are connected to may fix the issue.

Just flipped it to x4/x4/x4/x4 and the drives still don't show up for preclearing. The m.2's are in slots 1 and 2 on the hyper adapter. I'll try moving that to 1 and 3 with x8/x8 tomorrow. 

Share this post


Link to post

are you booting with UEFI?

i don't know if this is related, but when i asked about bifurcation on X9 boards to SM support some time ago, they answered, it's only works if you boot in UEFI mode..

Share this post


Link to post
6 hours ago, uldise said:

are you booting with UEFI?

i don't know if this is related, but when i asked about bifurcation on X9 boards to SM support some time ago, they answered, it's only works if you boot in UEFI mode..

Yes, PCIe mode is set to UEFI only.  Other settings remained the same for Slot 5 (x4/x4/x4/x4).  Still didn't pull up the drive in System Devices

 

-moved drive 2 from slow 2 to 3 on the hyper drive. Set to x8/x8 and neither show.  Maybe try another slot but the manual seems to state that all of the x16 slots are the same.:?!?

Edited by kron
additional details

Share this post


Link to post
2 hours ago, kron said:

Yes, PCIe mode is set to UEFI only.

That wasn't the question.

 

8 hours ago, uldise said:

are you booting with UEFI?

Please verify that you have enabled UEFI boot mode on the USB stick, and you have legacy boot disabled in the BIOS.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now