Jump to content
lusitopp

Help with hardware for my first "all in one" machine

14 posts in this topic Last Reply

Recommended Posts

Hi, 

I'm kind of new to unraid, have been running it on a xeon 1240l v3, 32gb ecc ram, intel motherboard for a few months now, and I want to take the next step.

Today I have that server, used for storage, 4 dockers and a few virtual machines.

I have an old Intel NUC for some tasks.
I have a Zyxel 4bay NAS.
We have two gaming desktops (will be replaced with one new)

So I want to have all of this in one physical machine.

I was looking into buying the Ryzen 3950x and a Asus Pro WS X570-Ace motherboard, and begin with 2x32gb ecc ram.
Now I have some questions about this, regarding both compability and pcie-lanes
I heard that there is a problem with GPU passthrough with new ryzen bios, is this still a problem and is it a big problem or just a problem for few users?
How do i need to think regarding pcie-lanes?

To summarize what I'm going to put into my computer:
8 sata HDD (there are four sata ports and then one u.2 mini sas port on the motherboard)
2 m.2 SSD (nvme for virtual machines, the other as unraid cache)
1 quad GB nic, pcie x4
1 usb-c controller card, pcie x4
1 gpu (X5700XT or something like that)

Now I need help thinking

 

M.2_1 goes over the CPU's pcie x4
Pcie x16_1 and _2 go over the CPU's pcie x16 in x8 / x8 if I use both.

 

Pcie x16_3 uses the chipset x4.
M.2_2 shares lanes with pcie x4, so if both ports are used, M.2_2 will go into x1. This goes over the chipsets x4.
4 sata and u.2 mini sas (up to 4 sata) go over the chipsets x4.

 

So if I stop a nvme in M.2_1. GPU in x16_1 and quad nic in x16_2 then graphics cards will run in x8, which I have read does not matter at all unless i buy a 2080ti.

 

The following will be using the 4 lanes between chipset and cpu

USB-C controller in x16_3

SSD in m.2_2

4 disks in the sata ports and then further 4 disks in the u.2 port.
 

Will there be a bottleneck in my setup?

Regards

Share this post


Link to post
  • The BIOS issue that I remember had to do with X470 update to support next gen Ryzen that broke it. I don't think it is related to the X570 chipset. Nevertheless, it is still a good idea to pre-download all the BIOS's and keep them in a handy USB stick in case you need to downgrade.
  • Your path with X5700XT will be fraught with difficulties (if it is even possible to pass through Navi right now). The Navi reset patch proved unreliable and was removed from 6.8.0-rc so the issue is still there.
  • Your mobo only has 4 SATA. The U.2 is for PCIe NVMe SSD and not a SAS breakout. The maximum available SATA on a X570 motherboard without a separate controller is 6.

Now questions for you:

  1. Why do you need separate Quad gigabit card? - using your server as a router too?
  2. Why do you need separate USB-C card? - what's wrong with your motherboard controller?
  3. You had 2 gaming desktops but replace it with 1 gaming desktop?
  4. Do you really need 2 NVMe? Do you actually need all SATA for the array?
    • Generally with HDD, it is better to have fewer large capacity HDDs. Just use 1 SATA port for a cache i.e. SATA SSD and your problem with PCIe lane is resolved.
    • From personal experience, NVMe cache doesn't offer any perceivable speed diff - it's a more a "I have it so I use it".

 

Also, in my opinion, the first choice for motherboard with Unraid beginners should be Gigabyte due to one single feature: "Initial Display Output".

  • Gigabyte BIOS allows you to pick any of the x16 slot (including the bottom x16 slot that only runs at x4 speed - check the owner's manual pdf from Gigabyte website to confirm) as initial display i.e. what Unraid boots with.
  • That allows you flexibility in case you actually need a 2nd cheapo GPU for Unraid (e.g. when you have problems passing through primary GPU). Otherwise, you will have to waste a fast PCIe slot - of course if you are running Unraid Nvidia then it doesn't matter.

All the motherboard brands are more or less similar in terms of features and gimmicks so why not consider something that MAY make your life easier?

 

 

 

Share this post


Link to post

Thank you for very great feedback! =)

This build will happend around feb/mars, but i need to start reading up on things.

 

Regarding the X5700XT, i can choose an nvidia too if it is better, lets wait and see if the issue with navi is resolved until i will build my machine. do you know where i can follow up on the progress?

 

Regarding the u.2, this is what the manual says on page 1-4:
 

Quote

The Mini-SAS HD connector allows you to connect a Mini-SAS HD cable to support configurations such as U.2 devices or four SATA devices.


1: The reason for quad nic is that im running 2 pfsense machines and also I like VMs and dockers to be on separate nic from unraid server. 1 nic for management and other nics for other stuff, just a habit i have.

2: i did some reading and as i understood it, it can be difficult to passthrough usb ports attached to the motherboard to a specific VM, so that is not an issue?

3: me and my girl almost never play on computer at the same time, most of the time one is on the computer and the other play xbox/ps.

4: i dont really need 2 nvme. my plan was to have one nvme assigned only to the gaming VM as physical passthrough and let other VMs have virtual harddrive on the other drive which will also serve as cache. So you would say to have maybe a 2tb NVMe as cache and have all VMs, dockers and file transfer on that single cache drive?

 

"Initial Display Output" sounds like a good feature, i was looking for information about a scenario like this:
"what if my VM with gpu passthrough doesn't start and i can't access the unraid webpage"

so, talking about that scenario, what would happen if my VM doesn't start and i don't have a second GPU?

 

Share this post


Link to post
1 hour ago, lusitopp said:

Regarding the X5700XT, i can choose an nvidia too if it is better, lets wait and see if the issue with navi is resolved until i will build my machine. do you know where i can follow up on the progress?

 

Regarding the u.2, this is what the manual says on page 1-4:

1: The reason for quad nic is that im running 2 pfsense machines and also I like VMs and dockers to be on separate nic from unraid server. 1 nic for management and other nics for other stuff, just a habit i have.

2: i did some reading and as i understood it, it can be difficult to passthrough usb ports attached to the motherboard to a specific VM, so that is not an issue?

3: me and my girl almost never play on computer at the same time, most of the time one is on the computer and the other play xbox/ps.

4: i dont really need 2 nvme. my plan was to have one nvme assigned only to the gaming VM as physical passthrough and let other VMs have virtual harddrive on the other drive which will also serve as cache. So you would say to have maybe a 2tb NVMe as cache and have all VMs, dockers and file transfer on that single cache drive?

 

"Initial Display Output" sounds like a good feature, i was looking for information about a scenario like this:
"what if my VM with gpu passthrough doesn't start and i can't access the unraid webpage"

so, talking about that scenario, what would happen if my VM doesn't start and i don't have a second GPU?

 

There isn't really a progress checker. The issue lies with AMD and the fix is in the kernel so whenever AMD fixes it and/or someone fixes the kernel and/or someone has a patch available then we'll see how things progress. Given how long the issue has been around, I am not optimistic.

Nvidia isn't pain-free. Passing through primary / only Nvidia GPU (GTX) is likely to give you error 43. Having a second GPU for Unraid to boot with is one of the potential solutions (and even so it is not a guarantee that you won't get error 43).

 

With regards to the U.2 - the Asus manual is very ambiguous because its spec in the same manual says "U.2 connector (supports U.2 NVMe device)".

Given the manual says "such as" and "or", it can be interpreted as just 2 examples of the Mini SAS connector and the spec clearly says it only support 1 of the 2 examples.

I don't have the board so of course take it with a grain of salt.

However, I am comfortable that my conclusion is correct since even the Anandtech review specifically only says U.2 NVMe with no mentioning of additional 4 SATA ports.

 

1. Make sure you do a test with your NIC passthrough with Unraid before assuming it would work. I have seen several posts reporting problems with Pfsense VM lately on the forum..

 

2. The pattern I have seen with Ryzen is there is usually at least 1 USB 3.0 controller that can be passed through (2 controllers for Threadripper).

Based on this post on the forum, I don't see why the 06:00 device can't be passed through. It is in its own IOMMU group.

You should always try the motherboard controller first. Only buy what you need if the motherboard controller doesn't work.

 

3. Expect strain on your relationship. Girls who game are already rare. Girls who game and still be nice when she wants to game while you are also gaming and can't give her the PC back is on the endangered list.

 

4. You can pass through the NVMe (as a PCIe device) to the VM as it's main storage. There is no need for a vdisk in that case.

For the cache, you don't need NVMe performance. A SATA SSD is more than good enough for docker uses.

Share this post


Link to post

once again testdasi. thank you for great feedback! 😃

 

regarding the asus board, i will check with asus support about this. the gigabyte board only have 6 sata but 3 m.2, need to verify pcie lanes.

 

Nic passthrough is not needed as long as i can bridge ex. eth2 to br2 and so on. i'm using bridge today even though passthrough *would* be nice.

 

i will try the mobo controller first, thanks 😃

 

if my girl have complaints she can buy her own computer! noo, just kidding :P but if it will be a problem i can always get a second computer or get a eGPU dock for my laptop, the options is there 😃 feel that if i would run 2 gaming desktops on unraid i would need to buy threadripper or skylake-x, and that will be expensive :D

 

i will have atlest 4 VMs except the gaming VM. and sometimes i have more because im testing things for work (at most i had around 20 on previously win server). so my thought is either to run everything (vdisk and dockers) on nvme or have atlest two, one for the gaming and then one for everything else.
but im thinking if i get a 2tb nvme i should have no problem using that for everything.

one question, does unraid have any dedup? alot of space to save when running virtual win servers.

Edited by lusitopp

Share this post


Link to post
1 hour ago, lusitopp said:

regarding the asus board, i will check with asus support about this. the gigabyte board only have 6 sata but 3 m.2, need to verify pcie lanes.

 

if my girl have complaints she can buy her own computer! noo, just kidding :P but if it will be a problem i can always get a second computer or get a eGPU dock for my laptop, the options is there 😃 feel that if i would run 2 gaming desktops on unraid i would need to buy threadripper or skylake-x, and that will be expensive :D

 

i will have atlest 4 VMs except the gaming VM. and sometimes i have more because im testing things for work (at most i had around 20 on previously win server). so my thought is either to run everything (vdisk and dockers) on nvme or have atlest two, one for the gaming and then one for everything else.
but im thinking if i get a 2tb nvme i should have no problem using that for everything.

one question, does unraid have any dedup? alot of space to save when running virtual win servers.

I just found out recently that the Asus ROG Crosshair VIII Hero has 8 SATA ports but only 2 M.2.

So if SATA ports are important for you then perhaps consider that.

I still think Gigabyte is better for the reasons I have said but I understand why having 8 SATA ports are important for some.

 

The 3950X can handle 2 gaming VM's. It has the same number cores (16) as Threadripper 1950X / 2950X.

The model numbering does suggest the 3950X to be the successor of the Threadripper 2950X.

 

Whatever 2TB SSD you get, don't get Intel (660p and 760p).

Intel has the anti-consumer practice of locking up the entire SSD in read-only mode after all the reserve has been used, under the pretext of data loss protection.Both 660p and 760p models have also been reported on here to be not happy with PCIe pass through.

And the 660p is QLC (NVMe QLC is like driving a Ferrari off road - avoid it. There's a use case for QLC, just not as an NVMe SSD).

 

No, Unraid doesn't have dedup.

Share this post


Link to post

Late response but i think i have decided now, and thanks for all the help.

  • Gigabyte x570 Aourus Master (i believe 😃 who knows, maybe choose the ultra or pro)
  • Ryzen 3900x
  • 2x Kingston A2000 NVMe 1tb (for cache and VMs)
  • 4x16GB Corsair Vengeance LPX Black / 3200MHz / CL16
  • continue to use my GTX 1070 until there is a valid amd card out there.
  • GeForce GT 710 for my unraid needs

I will primary use this hardware to build my new unraid machine and move some VMs and my other NAS to it.
I will then try to get a gaming VM up and running. if i succeed, YAY. If i dont, no worries 😃

Edited by lusitopp

Share this post


Link to post

Not sure i will pass through, will probably use them both in raid1 so i have backup on my VMs.

Which SSD would you recommend?

Share this post


Link to post
5 minutes ago, lusitopp said:

will probably use them both in raid1 so i have backup on my VMs.

RAID is not technically a backup, and isn't a substitute for real backups. It only helps in 1 case, the clean failure of a device. There are many more ways to lose data, so you would probably be better off setting up a real backup, there is a VM backup plugin that is a work in progress in Community Applications.

Share this post


Link to post
33 minutes ago, jonathanm said:

RAID is not technically a backup, and isn't a substitute for real backups. It only helps in 1 case, the clean failure of a device. There are many more ways to lose data, so you would probably be better off setting up a real backup, there is a VM backup plugin that is a work in progress in Community Applications.

 

yes i know that, but it is good enough for me.
the vm backup plugin would not be a great backup either because it will (probably) store the backup on the array, and if my server goes boom boom my backup goes boom boom 😃

 

but yeah, i will check that out, if i can save the money or use that second ssd to something else it could be good, thx

 

Edited by lusitopp

Share this post


Link to post
26 minutes ago, lusitopp said:

the vm backup plugin would not be a great backup either because it will (probably) store the backup on the array, and if my server goes boom boom my backup goes boom boom 😃

 

"Great" is subjective. There are uses for everything.

  1. RAID-1 will only save you if 1 of your 2 NVMe fails cleanly (i.e. without actual data corruption).
    It won't save you if, let's say, your VM is infected with crypto virus.
  2. A VM backup on the array will restore you quickly from a crypto virus infection and a NVMe failure (even partial corruption).
    It won't save you if, like you said, your server goes boom boom.
  3. An external drive backup of your VM vdisk (that is only plugged in while doing the backing up) will save you from NVMe failure, crypto virus infection and your server going boom boom.
    It won't save you if, god forbids, your house is on fire.
  4. A cloud backup will save you from NVMe failure, cryptovirus infection, your server going boom boom and your house on fire.
    It won't save you if the data center worker is dumped by his wife, gets angry and decides to set the entire data center on fire, killing off the storage unit that has your backup.

(2), (3), (4) are examples of online, offline and offsite backups.

There are always some scenarios that you can't recover your data regardless of what (and how many) backups you have. However, having a backup is still better than having no backup.

 

That's why I'm running single-drive cache and focus on making sure everything important to me have the "4 O's" copies - original, online, offline, offsite.

 

 

 

 

Edited by testdasi

Share this post


Link to post

yeah i understand the basics of backup. nothing on my unraid will be that critical. but i will look into the backup so i can free one nvme device up, didnt event thought about looking for backup in the apps 😃

 

haven't had the time to look for the app yet, but is it possible for the app to backup online (onedrive or azure (have msdn subscription, so free space =)). or do i have to get the backup online with rclone or similar?

Edited by lusitopp

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.