Jump to content
casperse

Enough PCI express lanes?

12 posts in this topic Last Reply

Recommended Posts

Posted (edited)

Hi All

 

I did build my server some time ago and I always planned to get 10G network at some point!

BUT as always I need to do some compromise on how to place my PCI express cards?

 

My MB is a Gigabyte C246-WU4 (Its been rock solid, and working with everything I have thrown at it) its has these slots available:

image.thumb.png.2c67e7da268e0a11b93a518e9ed868ff.png

 

I have the following cards placed like this:

 

Slot 1 - Quadro P2000

Slot 2 - Intel Pro 1000 VT Quad Port NIC (EXPI9404VTG1P20)

Slot 3 - Empty

Slot 4 - LSI Logic SAS 9201-16i PCI-e 2.0 x8 SAS 6Gbs HBA Card

Slot 5 - Empty

 

If I buy a Intel 10gbps X540-t2 Ethernet Network Adapter? that would require x8? not work in 4x?

But would the Quadro P2000 work in a 4x slot?

 

Would this be my best option ?:

 

Slot 1 - 10 gbps X540-t2 Ethernet Network Adapter

Slot 2 - Quadro P2000

Slot 3 - Empty

Slot 4 - LSI Logic SAS 9201-16i PCI-e 2.0 x8 SAS 6Gbs HBA Card

Slot 5 - Empty

 

Sofar I don't have any need for a bigger controller the 16i + 10 onboard Sata ports and the two M2.SSD slots = 28 drives

Would this work? - The P2000 is only for Plex/Emby (The CPU has a IGP I can use for VM's)

 

Also I would like to extend my Cash drive (Needed for 10G) I think I have two options:

 

1) Buy one more Samsung SSD 970 EVO S467NX0KB27411E and do a RAID0? (The MB can do HW raid 0 would that be better?)

2) Or buy the new 2TB (Slower) Intel 600p Series SSDPEKNW020T801?

(No available Sata slots)

 

Looking forward to your input/comments

 

Cheers

Casperse

 

 

 

 

 

 

 

 

Edited by casperse

Share this post


Link to post

All should be fine. Just need to know slot 1,4 come from CPU and 2,5 from C246. And device in 1,4 will usually group in same IOMMU group in native. ( you can ignore this if you don't pass through PCIe device )

 

In general, it should no big difference which device in which slot, except you found large performance drop in real case.

Share this post


Link to post
3 minutes ago, Benson said:

All should be fine. Just need to know slot 1,4 come from CPU and 2,5 from C246. And device in 1,4 will usually group in same IOMMU group in native. ( you can ignore this if you don't pass through PCIe device )

 

In general, it should no big difference which device in which slot, except you found large performance drop in real case.

So performance impact/shared PCI lanes would be:

 

Slot 1 & 4 - 10 gbps X540-t2 Ethernet Network Adapter / LSI Logic SAS 9201-16i PCI-e 2.0 x8 SAS 6Gbs HBA Card

Slot 2 & 5 - Quadro P2000/Empty so I could potential add another 4X card in this slot? (Or keep my Intel Pro 1000 VT Quad Port NIC?)

 

Yes I did use the P2000 as passthrough at one time (But it's a hassle because I need to stop Plex/Emby 

I would actually like some GPU for passthrough but I would be limited to Slot 5 with 4x

 

Is there any 10G card available that would run in a 4x slot? ;-) (Dont need a dual 10G card)

Share this post


Link to post

I think Slot one has higher priority so maybe I should place the HD controller there?

 

Slot 1 - LSI Logic SAS 9201-16i PCI-e 2.0 x8 SAS 6Gbs HBA Card

Slot 2 - Quadro P2000

Slot 3 - Empty

Slot 4 - 10 gbps X540-t2 Ethernet Network Adapter

Slot 5 - Empty

 

I have read around the forum and the 10G X540-t1 or t2 should be plug and play in unraid?

Share this post


Link to post
Posted (edited)
16 minutes ago, casperse said:

I think Slot one has higher priority so maybe I should place the HD controller there?

Yes, me too, I place two disk controller in those direct connect CPU's PCIe slot.

 

16 minutes ago, casperse said:

I have read around the forum and the 10G X540-t1 or t2 should be plug and play in unraid?

Should be. All my 10G NIC ( optical not RJ45 ) also plug and play with Unraid.

 

One question I want confirm from you. Seems your existing setup disk controller and P2000 in slot 1,4 and you pass through P2000. Does they are in same IOMMU group and you need apply something like ACS override to seperate it ?

 

I use Z390 and never play C246.

Edited by Benson

Share this post


Link to post
Posted (edited)
36 minutes ago, casperse said:

I would actually like some GPU for passthrough but I would be limited to Slot 5 with 4x

I don't think have performance drop or even negatable.

 

36 minutes ago, casperse said:

Is there any 10G card available that would run in a 4x slot?

Most were 8x, you may got some Aquantia base (RJ45) in 4x. But pls further check.

Edited by Benson

Share this post


Link to post
25 minutes ago, Benson said:

One question I want confirm from you. Seems your existing setup disk controller and P2000 in slot 1,4 and you pass through P2000. Does they are in same IOMMU group and you need apply something like ACS override to seperate it ?

 

I use Z390 and never play C246.

 

Yes PCIe ACS Override setting enabled and a P2000 bios file for the VM works fine did movie editing on it :-)

 

I almost bought the Z390 but I really wanted ECC RAM I don't think that is supported on Z390?

and at this point I also wanted a IGP in my CPU for encoding before I got the P2000

 

What about the cash? what are you using? (I really need a bigger cash drive and raid 0 should cover speed and double the size)

Share this post


Link to post
2 minutes ago, casperse said:

I almost bought the Z390 but I really wanted ECC RAM I don't think that is supported on Z390?

Yes, Z390 doesn't support ECC memory. The question just to exame does C246 have same behaviour with Z390 in IOMMU part.

 

For about cache, in fact I just use it for VM, it is a low end SSD.

 

I use UD and pool disks in RAID0 for capacity and speed need.

 

In fact, you don't need found x4 NIC, because you have a x16 slot. Stay in Intel NIC.

Share this post


Link to post
Posted (edited)
3 hours ago, Benson said:

Yes, Z390 doesn't support ECC memory. The question just to exame does C246 have same behaviour with Z390 in IOMMU part.

In fact, you don't need found x4 NIC, because you have a x16 slot. Stay in Intel NIC.

OMMU - Intel VT-d what behaviour are you talking about?

 

So you are saying stay with Intel NIC an use one of the fast slots 8x (shared with the LSI controller)

I guess I just have to get one and try! ;-) Guess Ebay is still best option

 

I found this card which looks like its 4x? and have two 2 NICS?

Intel Ethernet Converged Network Adapter X550-T2

image.png.9f667b027a7222aca9cb4d41acafe539.png

 

 

 

Edited by casperse

Share this post


Link to post
Posted (edited)
18 hours ago, casperse said:

OMMU - Intel VT-d what behaviour are you talking about?

When device connect to CPU or Chipset PCIe, their IOMMU grouping will be different.

 

18 hours ago, casperse said:

stay with Intel NIC

Reason are may be less trouble, Intel have main stream support in market.

 

18 hours ago, casperse said:

I found this card which looks like its 4x? and have two 2 NICS?

Yes, the card physically x4, it can plug in x16 x8 x4 slot, this will got max compatible.

 

X550-T2 native was a x4 chipset ( except X550-BT2 )

https://ark.intel.com/content/www/us/en/ark/products/88209/intel-ethernet-converged-network-adapter-x550-t2.html

 

So even X550 physically was x8, the card would max at x4 actually.

Edited by Benson

Share this post


Link to post
2 hours ago, Benson said:

When device connect to CPU or Chipset PCIe, their IOMMU grouping will be different.

 

Reason are may be less trouble, Intel have main stream support in market.

 

Yes, the card physically x4, it can plug in x16 x8 x4 slot, this will got max compatible.

 

X550-T2 native was a x4 chipset ( except X550-BT2 )

https://ark.intel.com/content/www/us/en/ark/products/88209/intel-ethernet-converged-network-adapter-x550-t2.html

 

So even X550 physically was x8, the card would max at x4 actually.

Yes IOMMU is different but only if I want to pass through my P2000 - using it for Plex/Emby no worries

I think I can Pass he IGP on the CPU as a graphiccard without IOMMU problems?

 

Yes I found this list (Look at the x4 column, this would mean that the new 550 only requires 4x and the old one requires 8x)

Or the physical slot is 8x for the old card and it oly uses 4x?)

image.thumb.png.5a4fd9c1ba0feebccf12fd354075baee.png

 

Anyway I am going to get a X550T2 if someone can confirm its plug and play with Unraid? Anyone....? :-)

Share this post


Link to post

I miss that, most X550-T2 were x4 ( physical and electrical ), old gen usually x8 and PCIe 2.x only ( physical and electrical )

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.