Ryzen 3950x build - is this viable?

[eribob]

I am planning for a server upgrade. My current build is 2 x e5 2670 and I run my main computer as a VM from them. I am really happy with the general setup bu I feel that the 2.6-3.3ghz speed is a little too slow. For example cities skylines runs at about 15fps now that my city has grown a bit. Lowering the graphics settings does not improve that, which makes me think it is a cpu issue, a lot of computations for all the inhabitants in my city perhaps? 

The 3950x is such a powerful chip and even though I am not editing 4K video I think this kind of build has the potential to fully utilise all the fast cores it provides! I use my computer for programming, some statistical calculations for my research, and sometimes gaming.
 

I would like to be able to do the following:
- Run 2 VM:s, with a dedicated gpu for each of them. I mainly use win 10 but want to have a Mac as well for fun
- run my 20 or so docker containers
- Use my SAS expansion card for additional hdd:s
- Preferrably have at least 2 nvme drives.
- Be able to upgrade to 128gb ram later
 

I don’t think I need pcie gen 4 so I would like to use a much cheaper x470 board if possible (or do you see a need for x570 here?)

 

I only need cpu motherboard and ram. The rest I will re-use. 

 

I made the following list but I would like some input from you experts as it is a big purchase! Will the board accept 32gb ram sticks for example? Do you see any other major issues here?

https://pcpartpicker.com/user/eribob/saved/8y3j23
 

what about gaming on a vm on a ryzen server? I read somewhere here that there might be performance issues? I am no competitive gamer, just want to be able to play for fun occationally. 
 

looking forward to your insights!

[eribob]

Has noone tried a similar setup akd can give me any advice? I am most interested in potential latency on VMs that I heard could be caused by a ryzen cpu. 
 

Thanks! 
Erik

[testdasi]

I don't think you can do 2xGPU + 2xNVMe + 1xHBA with X470. Check the mobo owner manual carefully because in most cases, some slots are deactivated if other slots are occupied.

 

With RAM stick support, again check manual / spec sheet. If it doesn't say it supports, assume it doesn't. There may be support added in BIOS update but ultimately if it's not on the spec sheet, you have no ground to argue for a replacement / return / refund.

This particularly applies to 32GB single DIMM support. It only starts to appear in X570 spec sheets and while I have heard that some made it work with X470 but do you want to take the risk?

 

Also I would recommend getting a Gigabyte motherboard so you have flexbility to pick any x16 slot for the GPU that Unraid boots with. This provides flexibility with GPU pass-through-ability in case a card would not work with VM if Unraid boots with it.

 

In terms of latency, it's there and is higher than Intel CPU (even when all workarounds have been applied).

It's unlikely to translate to benchmark figures because it manifests as inconsistent fps (aka stuttering). Some can't tell the diff (e.g. me) and some can (e.g. my friend, whom I have done a blind test on and I'm 99% sure he can).

Side note is with Ryzen, always remember that the best performing config is not necessarily the most consistent config. The fastest F1 car on Monza isn't necessarily the fastest around Monte Carlo.

[eribob]

Many thanks for the answers. 

 

Perhaps the scales are in favor of a x570 board then. I have been curious about the PCI-E 4.0 that they support. That would double the bandwidth of the only 24 PCI-E lanes that the 3950x has in theory, but I dont know if it will be true in practice... 

 

I will look at Gigabyte boards. And do modest tweaks for improved latency then. Is the latency a problem with KVM or is it specific to unraid? I expect it has to do with KVM, just wanted to ask... Also, will the upgrade to the newer kernel in 6.9 help in any way? 

 

Too bad threadrippers are so expensive hehe... 

[eribob]

4 hours ago, testdasi said:

I don't think you can do 2xGPU + 2xNVMe + 1xHBA with X470. Check the mobo owner manual carefully because in most cases, some slots are deactivated if other slots are occupied.

I looked at a manual for the Asrock Taichi x570 board that I have been thinking about buying and this it what it says with regards to PCI-E and M.2 slots: 

Quote

AMD Ryzen series CPUs (Matisse): 3 x PCI Express 4.0 x16 Slots (PCIE1/PCIE3/PCIE5: single at x16 (PCIE1); dual at x8 (PCIE1) / x8 (PCIE3); triple at x8 (PCIE1) / x8 (PCIE3) / x4 (PCIE5))*

This should mean that if I populate all of them using two GPU:s and a HBA I would get PCIE 4.0 x8 bandwidth (16 GB/s) for the GPUs and x4 speed (8 GB/s) for the HBA.

 

The bandwidth for the GPU:s should probably be fine right? I will not be running the latest and greatest GPU:s since I am not a gamer (currently have a rx580 4GB card I bought used). 

 

8 GB/s max for up to 8 drives that could be attached to the HBA, thats 1 GB/s per drive, which is almost 10x more than what most mechanical drives can deliver anyway, and rarely are all drives accessed at the same time right? 

 

Quote

1 x Hyper M.2 Socket (M2_1), supports M Key type 2242/2260/2280 M.2 SATA3 6.0 Gb/s module and M.2 PCI Express module up to Gen4x4 (64 Gb/s) (with Matisse)

1 x Hyper M.2 Socket (M2_2), supports M Key type 2260/2280 M.2 PCI Express module up to Gen4x4 (64 Gb/s)*

1 x Hyper M.2 Socket (M2_3), supports M Key type 2230/2242/2260/2280/22110 M.2 SATA3 6.0 Gb/s module and M.2 PCI Express module up to Gen4x4 (64 Gb/s)* *

Quote

If M2_3 is occupied, PCIE5 slot will be disabled

So this means that I can not use the third M.2 slot, but I was looking for max 2 slots anyway (one for each VM is what I envision...)

 

Quote

Ryzen CPU lanes are used to feed the top two full-length expansion slots in the usual x16/x0 or x8/x8 format. As such, Crossfire and SLI is supported. The bottom full-length connector gets its PCIe 4.0 x4 connection from the X570 chipset and shares that link with the lowest M.2 connector. Both PCIe 4.0 x1 slots are also fed by the X570 chipset.

So the 24 lanes provided by the CPU + the extra lanes provided by the x570 chipset should be enough for everything???? 

 

This is a quite expensive board though. I will look at a cheaper option as well and come back with how that would work. 

 

Another problem is, since I want to use the 2 GPU:s for the 2 VM:s, I need to be able to pass both through even though one might be used by Unraid? But I think I have read somewhere that this can be acieved? Perhaps spaceinvader one has a tutorial... 

Edited April 20, 2020 by eribob

[juan11perez]

HI, I'm not knowledgeable enough to comment on some of your questions, but I'll share info on what Im running now. 

Asus ROG STRIX X470-F, Ryzen 3900x, 64GB RAM, 3 x 4TB Seagate, 500GB NvMe, GPU1 - Saphire RTX 570, GPU2- Gigabyte GTX1650

I can definitely run 2 vms at the same time without issue in addition to about 40 dockers.

 

[testdasi]

6 minutes ago, eribob said:

Many thanks for the answers. 

 

Perhaps the scales are in favor of a x570 board then. I have been curious about the PCI-E 4.0 that they support. That would double the bandwidth of the only 24 PCI-E lanes that the 3950x has in theory, but I dont know if it will be true in practice... 

 

I will look at Gigabyte boards. And do modest tweaks for improved latency then. Is the latency a problem with KVM or is it specific to unraid? I expect it has to do with KVM, just wanted to ask... Also, will the upgrade to the newer kernel in 6.9 help in any way? 

 

Too bad threadrippers are so expensive hehe... 

You will not see any real life impact of PCIe 4.0, at least for now (and for consumer uses). Even an RTX 2080Ti overclocked to instability won't saturate PCIe 3.0 x8 so 4.0 x8 is more than enough.

 

Latency is an inevitable result of Ryzen CCX/CCD design. It's not specific to any platform or OS - as long as it's a like-for-like comparison. E.g. you can't compare bare metal latency vs VM latency (which naturally is always higher). But between 2 VM or between 2 bare metal configs, Ryzen (and Threadripper) will always have higher latency.

On the subject of Threadripper, latency is worse. Having 1 other die to comm with is a naturally faster than having 3 other dies to comm with.

 

Not sure about 6.9 update impact on latency but as I said, it's a intentional design compromise so it's impossible to eliminate.

 

You will have problem with the RX580. It is a known problem child if Unraid boots with it.

So if you use a mobo that is not Gigabyte, your only choice is to have it on the 2nd PCIe x16 slot, HBA on 3rd slot and another GPU in 1st slot.

Not a big deal if your RX 580 width is double-slot. A big deal if your RX 580 is the 2.5-slot width variety.

 

Unraid does not need a graphics card at all. That's a misconception when people refer to a card that is "used" by Unraid. It's a function of the motherboard BIOS. If the BIOS allows it, Unraid can boot completely headless e.g. no graphics card at all.

All BIOS that I know of will force graphical boot if there is ANY graphics card plugged in at all. So if you don't want a certain card to be initiated at boot (e.g. the RX 580 so you can pass it through to a VM), the only choice is to have another card that the BIOS would boot with (and thus "used" by Unraid). You could run KVM / QEMU in any Linux distro or even Windows Server Hyper-V or VMWare ESXi and it will be exactly the same situation.

[eribob]

3 hours ago, juan11perez said:

HI, I'm not knowledgeable enough to comment on some of your questions, but I'll share info on what Im running now. 

Asus ROG STRIX X470-F, Ryzen 3900x, 64GB RAM, 3 x 4TB Seagate, 500GB NvMe, GPU1 - Saphire RTX 570, GPU2- Gigabyte GTX1650

I can definitely run 2 vms at the same time without issue in addition to about 40 dockers.

 

That is super cool to hear! Very close to what I am aming for. I have acctually considered the 3900x as a budget option. 

 

How did you manage the 2 GPU:s? Which one do you have as primary? As testdasi said, it needs to be one that can be passed through to a VM despite also being used by unraid at startup? 

 

How many CPU cores do you allocate to each VM and to the dockers? 

Do you game on your VM:s? In that case, does it work ok with more modern titles? 

 

The RAM is 4 x 16GB?

 

Are there any major issues that you have had during the setup? 

 

Thanks! 

/Erik

[eribob]

3 hours ago, testdasi said:

Not a big deal if your RX 580 width is double-slot. A big deal if your RX 580 is the 2.5-slot width variety.

It seems to be 2.2 slot: https://www.sapphiretech.com/en/consumer/nitro-rx-580-4g-g5#Design 

I will look into gigabyte options! 

 

Can you recommend a GPU that is known to function so that you can pass it trough even though it is the primary GPU in the system?

3 hours ago, testdasi said:

Latency is an inevitable result of Ryzen CCX/CCD design. It's not specific to any platform or OS - as long as it's a like-for-like comparison. E.g. you can't compare bare metal latency vs VM latency (which naturally is always higher). But between 2 VM or between 2 bare metal configs, Ryzen (and Threadripper) will always have higher latency.

I see. So the latency is there even when you run bare metal? I guess my question is if the latency on a VM in a ryzen CPU server is so much

 

higher than the latency on a intel CPU server that intel is clearly a better option if you want the ability to game on your VM?

 

3 hours ago, testdasi said:

Unraid does not need a graphics card at all. That's a misconception when people refer to a card that is "used" by Unraid.

I understand that. Is there a workaround so that you can pass through your primary GPU even though the BIOS snags it at startup? EDIT: Sorry 0.5s of googling found me the spaceinvader video... So I will try to get my hands on a decent used NVIDIA gpu then.

Edited April 20, 2020 by eribob

[testdasi]

12 hours ago, eribob said:

It seems to be 2.2 slot: https://www.sapphiretech.com/en/consumer/nitro-rx-580-4g-g5#Design 

I will look into gigabyte options! 

 

Can you recommend a GPU that is known to function so that you can pass it trough even though it is the primary GPU in the system?

I see. So the latency is there even when you run bare metal? I guess my question is if the latency on a VM in a ryzen CPU server is so much

 

higher than the latency on a intel CPU server that intel is clearly a better option if you want the ability to game on your VM?

 

I understand that. Is there a workaround so that you can pass through your primary GPU even though the BIOS snags it at startup? EDIT: Sorry 0.5s of googling found me the spaceinvader video... So I will try to get my hands on a decent used NVIDIA gpu then.

I know my Zotac GTX 1070 Mini (and Zotac GT 710 but that's a low-end model not suitable for gaming) can be passed through.

• I used to boot Unraid with the GT710 and pass through the 1070 to the main workstation VM and the GT 710 to a MacOS VM.
• Currently, the 1070 is the only GPU in the system (i.e. Unraid boots with it) and it is also passed through to the same workstation VM after boot.

I do dump and use my own vbios (watch SpaceInvader One tutorial on how to dump your own vbios).

Note: the fact that both are Zotac is completely coincidental. I used to have a small case and the cards that fit happened to be Zotac.

Caveat: pass-through is very hardware dependent so without having the exact same hardware, it is impossible to be 100% certain.

 

I know factually Intel is better in terms of latency; however, I can't speak for YOU whether the latency would matter.

As mentioned, I can't tell the diff but I have a friend who can. This is totally guessing but I think if you are really good at shooting / twitchy games (i.e. like my friend), you are probably more likely to notice the stuttering.

 

 

 

 

 

 

[juan11perez]

@eribob

GPU1 - Saphire RTX 570 (primary)

GPU2- Gigabyte GTX1650

 

It's only used during start. As soon as unraid is online, it's released and I can start the windows or mac vm using it.

The sencond GPU I use for my Arch vm which is the daily driver.

 

I allocate 8 cores (16 threads) to the vms and the first 4 cores to unraid (dockers)

 

I dont game, sorry.

 

Yes I have 4 x 16GB rams sticks

 

No issues, it's very stable. no lockups or anything else. The only problem which everyone has right now is I cant get cpu and motherbouard temperature data as unraid 6.8.3 doesnt use the right kernel. Hopefully 6.9 will resolve that.

 

 

[eribob]

1 hour ago, juan11perez said:

GPU1 - Saphire RTX 570 (primary)

You mean RX 570? 

 

8 hours ago, testdasi said:

Currently, the 1070 is the only GPU in the system (i.e. Unraid boots with it) and it is also passed through to the same workstation VM after boot.

It seems like an NVIDIA card should work then as primary, after dumping the vbios. The 1070 is not so bad, at least it seems stronger than my Rx 580 and I do not need super GPU power...

 

1 hour ago, juan11perez said:

I dont game, sorry.

w00t? 😃

 

It is all coming together nicely I think! I will wait until my wallet feels full enough and then upgrade and then harass you again once everything fails...

[juan11perez]

@eribob

apologies, that’s correct rx570.