Jump to content
flaggart

Ryzen 3000 series build / Struggling with PCI-e lanes

92 posts in this topic Last Reply

Recommended Posts

Posted (edited)

@fr05ty  thanks for the post,  you answer one question. and that is  to only have on M.2 slot and if I want any more then to look into having a PCI expansion card.

 

and to anyone else, what about the ideal of my CPUS, that I go back a Gen for more threads ?

 

I think I answer my own question.     Ryzen 5 3600

 

https://www.cpubenchmark.net/compare/AMD-Ryzen-5-3600-vs-AMD-Ryzen-7-2700/3481vs3240

Edited by chris_netsmart

Share this post


Link to post
On 7/8/2019 at 10:39 PM, Pducharme said:

Hi guys! I just pulled the trigger on a ASUS Pro WS X570-ACE + a Ryzen 9 3900x.  I have a 21 drives Unraid and plan on upgrading my stuff with this new Ryzen 3900x and board.  I currently have 2 9211-8i and 1 GPU (for PLEX transcoding, but in a 1x slot with a Mining Riser that I reuse, works fine).  I might add a GTX 1080Ti for a VM (passthru) when that will be fix on the new platform.  I'll probably run that on a "test" Unraid until everything works fine, pretty sure i'll have some issues :)  Also, i still need to choose RAM for the new system and a NVMe PCIe Gen4 for the Cache drive.

 

I'll keep posting to this topic what my findings are.  Hope to at least be able to boot current 6.7.2 Unraid on my board.

Do you know if the USB 2 are on a separate controller or are they all combined? If it's on a separate controller I can save a pcie slot. 

Share this post


Link to post
Just now, phat_cow said:

Do you know if the USB 2 are on a separate controller or are they all combined? If it's on a separate controller I can save a pcie slot. 

Don't know yet.  I should receive my ASUS board, my 3900x, my Corsair RAM and Corsair MP600 on friday.  If it arrives really friday, i'll be able to play with everything this week-end.  I would be able to do some testing for people too, if you need it.   I won't put in "PRODUCTION" this new plateform until i'm sure it works stable.

Share this post


Link to post
1 hour ago, Pducharme said:

Don't know yet.  I should receive my ASUS board, my 3900x, my Corsair RAM and Corsair MP600 on friday.  If it arrives really friday, i'll be able to play with everything this week-end.  I would be able to do some testing for people too, if you need it.   I won't put in "PRODUCTION" this new plateform until i'm sure it works stable.

Thanks! This will be the most expensive build I have ever built and I'm super excited about it! 

  • Like 1
  • Upvote 1

Share this post


Link to post
On 6/8/2019 at 7:11 PM, flaggart said:

Hi all

 

Like many here (I assume), I am planning on a Ryzen 3000 series build in the next couple of months, to replace my existing setup.  This brings with it some choices around chipsets, pci-e lanes, etc.  My current setup is:

Why the 3000 series? first gen Threadripper is an absolute bargain and no worries about PCIe lanes

Share this post


Link to post
10 minutes ago, mrbilky said:

Why the 3000 series? first gen Threadripper is an absolute bargain and no worries about PCIe lanes

Cause we want to spoil ourselves with the newest tech :)

Share this post


Link to post
On 7/7/2019 at 11:21 PM, fr05ty said:

@BLKMGK have you looked into getting an intel res2sv240, its a sas extender you can plug 1 or 2 8087 cable(s) from say a 9211-8i to the res2sv240 and have 16-20 (if only using 1 cable) drives in your system i use 1 of these in my current unraid box. You could then have a max of 24 drives connected through a 8i HBA card.

The res2sv240 can be powered from a molex 4 pin or from a spare pcie slot. It only requires the power to work so if you need all your slots it can be placed somewhere else in the case.

 

I'm currently looking at doing the same ASUS board with either a 3900x or 3950x making it my gaming rig(2080ti), plex(p2000), and storage(lsi 9207-8e). i just want to see a few people that jump in first give feedback to see if it's worth it

I have an Extender already in a box but based on what I saw in Johnnie's test it looks like using one slows transfer speeds down too. Right now I use 3x Perc H310 cards in my existing system and have IBM M?? cards as well which are pretty close to the same thing. We need better cards I think lol! 

Share this post


Link to post
On 7/8/2019 at 9:39 PM, Pducharme said:

Hi guys! I just pulled the trigger on a ASUS Pro WS X570-ACE + a Ryzen 9 3900x.  I have a 21 drives Unraid and plan on upgrading my stuff with this new Ryzen 3900x and board.  I currently have 2 9211-8i and 1 GPU (for PLEX transcoding, but in a 1x slot with a Mining Riser that I reuse, works fine).  I might add a GTX 1080Ti for a VM (passthru) when that will be fix on the new platform.  I'll probably run that on a "test" Unraid until everything works fine, pretty sure i'll have some issues :)  Also, i still need to choose RAM for the new system and a NVMe PCIe Gen4 for the Cache drive.

 

I'll keep posting to this topic what my findings are.  Hope to at least be able to boot current 6.7.2 Unraid on my board.

Can you give any additional details on the riser card? I've tried looking for something like that but no luck as I'm not sure what the heck to call it? :(

Share this post


Link to post
Can you give any additional details on the riser card? I've tried looking for something like that but no luck as I'm not sure what the heck to call it?


One of those risers. It connect in a 1x slot to x16 slot. Requires separate power too (molex) :


Ubit 6 Pack Latest PCI-E Riser Express Cable 16X to 1X (6pin / MOLEX/SATA) with Led Graphics Extension Ethereum ETH Mining Powered Riser Adapter Card+60cm USB 3.0 Cable https://www.amazon.ca/dp/B076KLJR2Y/ref=cm_sw_r_cp_api_i_tgPjDbKPT47GP



Envoyé de mon iPhone en utilisant Tapatalk

Share this post


Link to post
3 hours ago, BLKMGK said:

I have an Extender already in a box but based on what I saw in Johnnie's test it looks like using one slows transfer speeds down too. Right now I use 3x Perc H310 cards in my existing system and have IBM M?? cards as well which are pretty close to the same thing. We need better cards I think lol!

@BLKMGK sorry if this seems a little long but hopefully it will clear a few things up for you

 

IBM M1015 and Perc H310 are basically 9211-8i from what i understand, so for 1 card quick math is 8 drives x 600 MB/s = 4800 MB/s max throughput, but as it is PCIe 2.0 x8 its max speed is 4096 MB/s so 8 drives could do up to 512 MB/s each.

 

if you take both outputs from 1 card into an extender and have 16 drives off the extender you would have a max speed of 256MB/s per drive if you are only connecting spinning rust to them you may only just get to the limits of the card if doing something like a scrub when reading all drives at once. of course there could be some overhead info that may drop the speed a little.

 

if you had a 9207-8i or similar card that is a PCIe 3.0 x8 card the PCIe 3.0 x8 max throughput is 7.88GB/s so 8 x 600 = 4800 / 16 = 300 MB/s which is still plenty for HDD's.

 

if you only had 1 HBA IBM M1015 or Perc H310 with 1 cable to the extender and 1 to 4 drives 4096 / 2 = 2048 / 20 drives = 102 MB/s and 2048 / 4 drives = 512 MB/s

 

When a scrub is started on my server it tops out around 160MB/s per drive on all 15 disks which is 2400MB/s in total i only have 1 cable to the extender which is 2400MB/s max speed and as it continues through the 4tb drives it will drop to match the drives as the further you get through a disk the read speed slows down when they have finished scrubbing and all the 8tb drives are left to continue it will pick up in speed again.

 

I have a 9207-8i in my server as i didn't want to fuss around with trying to flash a card, these ones just work otb as jbod/i.t. mode. i just started a scrub to take some screen shots which are attached below all my drives are connected/detected as sata3 6Gb/s disks for a bit more info.

 

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 

@mrbilky a 1950x is not that easy or cheap to come by in my area they are selling for about $1000 kangaroo dollars brand new, a 2950x is about $200 more, I have always had a look online for second hand but never see them for sale and the M/B are about $450 and up, but i never worried to much about price on motherboardsyou pay for features. If these new zen 2 CPU's are as good as all these reviews say they draw about a third (3700x 65w) to about half (3900x 105w) less the power than a 1950x's 180w and these new boards have all the features that i want even if it doesn't cut the mustard for unraid just yet i'm sure it wont take to long before it does catch up for what we want (if it never does then i have a super new gaming pc) and the 6700k still goes to dad rocking an old i3-530. either way i win

 

https://www.cpubenchmark.net/compare/AMD-Ryzen-Threadripper-1950X-vs-AMD-Ryzen-7-3700X-vs-AMD-Ryzen-9-3900X/3058vs3485vs3493

Screen Shot 2019-07-11 at 3.15.01 pm.png

Screen Shot 2019-07-11 at 3.14.39 pm.png

Share this post


Link to post
On 7/9/2019 at 11:55 AM, Pducharme said:

Currently, I have :

 

1 card x GTX 1060 6GB in a 1x (riser card) - Dedicated to PLEX

2 cards x LSI 9211-8i in IT mode - For 16 disks (other 5 disks are from onbard SATA controller)

 

Might add :

 

1 card x GTX 1080ti in a x16 slot - Passthrough to a Win10 VM (Not sure yet i'll do that).  I think I would also need a USB card for Keyboard/mouse and I would miss slots because of that ? The ASUS WS Pro X570-ACE  has 3 x x16 slots and 1 x x1 slot.

 

With the cards you have.... Does that mean your primary pcie x16 will run at 8x?

Share this post


Link to post
11 minutes ago, fr05ty said:

if you only had 1 HBA IBM M1015 or Perc H310 with 1 cable to the extender and 1 to 4 drives 4096 / 2 = 2048 / 20 drives = 102 MB/s and 2048 / 4 drives = 512 MB/s

Single wide band SAS2 link has 2400MB/s max theoretical bandwidth, of those 2200MB/s are usable, dual link doubles that.

 

Your other numbers are correct but you need to account for protocol overhead, which with PCIe is about 20/25%, so max usable speed is usually around 75/80% max theoretical bandwidth, some real world benchmarks here.

Share this post


Link to post

thanks for that reply and the link, i thought there might have been protocol overheads but i didn't know how much so i was just using the max theoretical bandwidth to try to work out a few number and trying to judge from the speeds i get reported in the unraid gui when doing a scrub, and i wasn't sure how much off a difference there is from a 9211 style card to a 9207 as i have the latter in my system

Share this post


Link to post
16 hours ago, phat_cow said:

Thanks! This will be the most expensive build I have ever built and I'm super excited about it! 

 

Update on my Shipping :

 

CPU + Motherboard :  New date for Scheduled Delivery -->  Next Tuesday (2019-07-16)

RAM : Next Tuesday (2019-07-16)

PCIe gen4 NVMe SSD : Next Tuesday (2019-07-16)

 

It seems that my testing will only starts next week :(  But I won't wait for the week-end ;) If I really receive everything on monday, during the evening i will assemble everything and do some basic testing (I'll drop a Win10 on the SSD for the test).  Since i'll have to "canibalise" my Office PC with the GPU (1080ti) since the 1060 is doing it's work in my PROD UNRAID for PLEX, if my RAM or SSD is not arriving as supposed, I have a spare 256GB NVMe I can use, and I can take 1 RAM stick or 2 from my Office PC too (it's a Ryzen 2700x, so DDR4 too).

 

I'll keep you guys posted.

Share this post


Link to post
7 hours ago, johnnie.black said:

Single wide band SAS2 link has 2400MB/s max theoretical bandwidth, of those 2200MB/s are usable, dual link doubles that.

 

Your other numbers are correct but you need to account for protocol overhead, which with PCIe is about 20/25%, so max usable speed is usually around 75/80% max theoretical bandwidth, some real world benchmarks here.

 

@johnnie.black I have 4 Disks 5-in-3 for my Data and Parity disks.  They each have 5 SATA ports (1 per disk).  I currently have 2 LSI 9211-8i (IT mode), so 4 big cable of SFF to 4xSATA and use the Onboard SATA ports to complete.   

 

If I would like to get more speed out of Unraid for "I/O" and disks, what would you recommend? Do I already have the "best" way to optimize my storage with Unraid ?  Thanks !

Share this post


Link to post
2 minutes ago, Pducharme said:

If I would like to get more speed out of Unraid for "I/O" and disks, what would you recommend? Do I already have the "best" way to optimize my storage with Unraid ?  Thanks !

As long the the LSI HBAs are using x8 CPU slots (possibly even x4 slots depending on the disks) there won't be any controller/bus bottlenecks, there might be if they are on PCH/chipset lanes, would need more info, like which board and slots are in use.

Share this post


Link to post

@johnnie.black, currently, they are on my old Supermicro X10SLL-F-O slots, the 2 top one.  I have the 1x used by a Riser that goes to a x16 Slot with a GTX 1060 (i know it runs at 1x probablly, but enough for PLEX transcoding).  

 

The plan is to "optimize" as most as possible, on my new ASUS WS Pro X570-ACE Motherboard + Ryzen 3900x.   I still can use a 1x slot with a riser for the GTX1060 for the transcoding if that leaves me enough of bandwith for the HBAs.   I have 21 disks in my Unraid, 20 Spinning disks + 1 SATA SSD (that will be replaced by the NVMe Gen4 MP600).  Do you think I would need to get a PCIe Gen4 HBA or i'm not even saturating the current HBAs I have with my setup ?

Share this post


Link to post
3 minutes ago, Pducharme said:

currently, they are on my old Supermicro X10SLL-F-O slots, the 2 top one.

That's good, both CPU slots.

 

6 minutes ago, Pducharme said:

The plan is to "optimize" as most as possible, on my new ASUS WS Pro X570-ACE Motherboard + Ryzen 3900x.   I still can use a 1x slot with a riser for the GTX1060 for the transcoding if that leaves me enough of bandwith for the HBAs.   I have 21 disks in my Unraid, 20 Spinning disks + 1 SATA SSD (that will be replaced by the NVMe Gen4 MP600).  Do you think I would need to get a PCIe Gen4 HBA or i'm not even saturating the current HBAs I have with my setup ?

You should be able to use one CPU slot and one chipset slot for the HBAs without any bottlenecks, since the chipset has a x4 PCIe 4.0 connection to the CPU, no need for a PCIe 4.0 HBA, and I don't know of any yet.

Share this post


Link to post
Posted (edited)

@johnnie.black I can only find this new chipset for such hypothetical HBA :  https://www.servethehome.com/microchip-24g-sas-for-pcie-gen4-servers/

 

How can I know if a PCIe slot is a "CPU" slot or a "Chipset" slot ? On the new ASUS I mean.  Also, except by trying, how can I know the IOMMU groups of the new board, is there a list online of that ?

Edited by Pducharme

Share this post


Link to post
14 minutes ago, Pducharme said:

How can I know if a PCIe slot is a "CPU" slot or a "Chipset" slot ?

Check the manual but usually the top 2 slots are CPU slots, one @ x16 or both @ x8, third slot is a chipset slot.

Share this post


Link to post
32 minutes ago, Pducharme said:

Also, except by trying, how can I know the IOMMU groups of the new board, is there a list online of that ?

No way to know without someone actually owning the board. Even different BIOS versions can result in different IOMMU groups.

 

Share this post


Link to post

@johnnie.black Here is the screenshot of the PCIe slots :

145090200_E15611_Pro_WS_X570-ACE_UM_V2_WEB.pdf2019-07-1110-47-17.png.9eee2ad7d835302af3cc24fd23935f25.png

 

There is 3 x16 PCIe slots.  According to the details, this is how it operate.  I know i'll use M2_1 for my MP 600 PCIe Gen4 x4 SSD.  Still not understanding if I want to have, 2 x HBA (LSI 9211-8i), 1 x GTX 1060 (PLEX docker transcoding).  Can I still add a GTX 1080ti at full speed in one of those slots for PCIe Passthru to a Win10 VM without loosing Storage Bandwidth?  Also, there is a U.2 connector that they say in the manual can be use with a SFF to 4 SATA cable to connect 4 disks, so it means it can have a total of 8 SATA (4 Sata ports onboard + those 4 from the U.2).  Are they usually as fast as 4 inidividual SATA port on the motherboard?

 

2070450378_E15611_Pro_WS_X570-ACE_UM_V2_WEB.pdf2019-07-1110-45-52.thumb.png.97f69defbc7c3982a4cabe75298c86c6.png

Share this post


Link to post
4 minutes ago, testdasi said:

No way to know without someone actually owning the board. Even different BIOS versions can result in different IOMMU groups.

 

Thanks, i'll do some testing next week when everything will come IN.  I just got the MP 600 SSD in my hand 😃

Share this post


Link to post
4 minutes ago, Pducharme said:

Can I still add a GTX 1080ti at full speed in one of those slots for PCIe Passthru to a Win10 VM without loosing Storage Bandwidth?

I believe so.

 

4 minutes ago, Pducharme said:

Are they usually as fast as 4 inidividual SATA port on the motherboard?

Yes, assuming it's using a x4 lane for the U.2 connection, though it might be shared with something else, that's usually mentioned din the manual if that's the case, shame one thing usually missing from consumer board's manual, a system block diagram, those make connections/bandwidth much clearer.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.