4x NVMe drives on single PCI-e expansion card..... (UPDATE, IT WORKS!)


Recommended Posts

So I spotted this today... a single slot x16 PCI-e expansion card for 4x NVMe drives:

 

https://www.scan.co.uk/products/asus-hyper-m2-card-pcie-30-x16-4x-m2-pcie-2242-60-80-110-slots-intel-vroc-support-for-asus-x299-moth

https://www.asus.com/uk/Motherboard-Accessory/HYPER-M-2-X16-CARD/

 

Does anyone have one of these running on a non-X299 board? Would be very very useful if thats the case!  

At that price im tempted to just get it and see if it works anyway...

 

Ideally it would still split each NVMe into its own IOMMU group too.. but i think im asking a bit much there!

Edited by billington.mark
Link to comment

considering this a buy too. I guess as long as your mainboard support PCIe bifurcation (check your bios) it will work and split the lanes up.

on top of that individual iommu groups would be great. tell me more about it !:) not many reports out there....btw anybody with supermicro 2xnvme >pcie adapter experience here ?

Link to comment
On 5/26/2018 at 8:15 PM, billington.mark said:

I get the option to change the lane layout for my x16 slots (x16 or x8x8 or x4x4x4x4), so im hoping its supported.... 

Ive ordered the part and i'll report back once ive fitted it. 

 

For anyone curious this is the setup im using:

 

ASRock - EP2C602-4L/D16

2x E5-2670

 

In that case it will work without a problem! the Asus Hyper M2 is a really nice affordable piece of hardware, I did not check if my bios supported PCIe bifurcation  so i ended up buying 3 of them. they are not that much more expensive as a card for 1 M2 SSD and they are way nicer! (I had PCIe issues with cheap Chinese PCIe to M2 cards)

Link to comment

From my google-fu on this, it seems PCI-e bifurcation is a feature in most CPUs from the last few years, its just a case of your BIOS giving you the option to configure the x16 slot(s) to 4x4x4x4x, which only seems to be available in higher-end gaming or workstation motherboards. 

 

Its a public holiday in the UK today, so I'll not get this delivered until later in the week to test unfortunately. :(

Link to comment

So its arrived and fitted, and.. Success!

You DO NOT need an x299 based motherboard for this card to work, only a montherboard\cpu capable of splitting an x16 slot into 4x4x4x4x. 

This was found in my PCIe settings in the BIOS. 

 

Even better, each NVMe device is still in its own IOMMU group: 

 

IOMMU group 47:	[144d:a802] 81:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01)
IOMMU group 48:	[144d:a808] 82:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981

Only thing i had to to after installing was to update a VMs XML which has the 970evo passed through to it with the new PCIe address. 

 

So, ive free'd up a PCIe slot, and have the option to add 2 more NVMe devices in the future without much hastle at all. :)

IMG_20180531_184402.jpg

IMG_20180531_184850.jpg

IMG_20180531_185101.jpg

  • Upvote 1
Link to comment
2 hours ago, billington.mark said:

Page 4-10 of your motherboard manual....

i think it is not so clear. i asked SM support some time ago about their AOC-SLG3-2M2 which is very similar and also need PCIe bifurcation to work properly. Look at the answer:

 

I'm sorry to tell you that you are not able to use the NVMe SSD with your current system since our X9 series only has BIOS that supports EFI only or Legacy only. You can try it out with your own discretion.

To use NVMe, you must have PCI-e slot where the NVME SSD installed with EFI enabled. Otherwise, the system will not detect it, usually there is an option to set only one slot as EFI if you need Legacy for your other cards, but since this X9 board is relatively old it must not have that feature. Our X9 motherboard series only has BIOS that supports either Legacy mode only or EFI mode only, you can’t combined.

Add-on Card AOC-SLG3-2M2 only compatible with the following motherboards as shown below. It will not work with any other motherboards and you can try it out with your own discretion.

https://www.supermicro.com/products/accessories/addon/AOC-SLG3-2M2.cfm

i'm not sure about my other cards: i have one IBM 1015 and one QLogic BR-1020 10Gbit Ethernet card.  

Link to comment

yes, this was a question a asked them next: "if i don't need NVME SSD as boot device, but just wanna use it as VMWare ESXi datastore, do i still need a bios support for it?" and they replied: "Unfortunately, no."

so i really did not understand their answer, and was not asked any further..

Link to comment
  • 3 months later...
  • 1 month later...

I went ahead and picked up one of these and enabled bifurcation on PCIe slot 5 (CPU2 Slot5).  I setup 8x 8x since I'm only using 2 SSD's for now.  Unraid doesn't detect these drives so I must be doing something wrong.. 

 

Page 4-10 of the manual states, this option isn't present on the latest bios of 3.3

IIO 1 PCIe Port Bifurcation Control This submenu configures the following IO PCIe Port Bifurcation Control settings for IIO 1 PCIe ports to determine how the available PCI-Express lanes to be distributed between the PCI-Exp. Root Ports

 

 

Board: X9DRi-LN4F+

Manual: https://www.supermicro.com/manuals/motherboard/C606_602/MNL-1258.pdf

 

Link to comment
27 minutes ago, tjb_altf4 said:

Have you tried running in x4/x4/x4/x4 mode to see if that fixes the issue?


If your on a slot that doesn't have that option, you may have the m.2s connected to the "second half" of the x8 instead of the "first half", changing the slots the m.2s are connected to may fix the issue.

Just flipped it to x4/x4/x4/x4 and the drives still don't show up for preclearing. The m.2's are in slots 1 and 2 on the hyper adapter. I'll try moving that to 1 and 3 with x8/x8 tomorrow. 

Link to comment
6 hours ago, uldise said:

are you booting with UEFI?

i don't know if this is related, but when i asked about bifurcation on X9 boards to SM support some time ago, they answered, it's only works if you boot in UEFI mode..

Yes, PCIe mode is set to UEFI only.  Other settings remained the same for Slot 5 (x4/x4/x4/x4).  Still didn't pull up the drive in System Devices

 

-moved drive 2 from slow 2 to 3 on the hyper drive. Set to x8/x8 and neither show.  Maybe try another slot but the manual seems to state that all of the x16 slots are the same.:?!?

Edited by kron
additional details
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.