Just another LTT inspired gaming workstation nas build, hw recommendations?

[Divi]

So I currently have separate Windows 10 gaming pc, and freenas server running pihole, syncthing etc. I'm looking to put all under one machine, and so far unraid looks like best option for me. 

 

Current freenas box is running 2x4Tb mirror, and this is what I would continue to use. I like the fact that expanding would be easy with unraid. I wanna run Windows 10 vm and dedicate gpu, usb, audio etc for it.. all other vms and containers will be running on background and require only ssh/vnc or similar 

 

Parts I have already in my disposal:

AMD Ryzen 5 2600

Gigabyte Aorus X470 Ultra Gaming

Nvidia GT520 for host

AMD RX580 8Gb to passthrough for windows

24Gb of ram

1tb Intel 660p nvme m.2 ssd

2x 120gb Kingston A400 sata ssd

2x 4Tb Seagate Ironwolf

Dual port pcie intel nic for 2x1gbe lacp with vlans

Silent Base 801 case

Seasonic Focus Plus 550w psu.

 

How would you suggest setting it all up?

1tb ssd would be quite fast cache but will it hold up being cheap qlc drive? Better to sell all ssds and get something like 2x 500gb samsung evo or similar? Those would be limited to sata3 tho.. Problem is that motherboard dont support two big nvme ssd, and they will also eat so many pcie lanes. CPU only has 20 and GT520 need to be in first 16x slot i think?

 

Edited January 7, 2020 by Divi

[Divi]

I've been looking through documentation of unraid and the forums but its hard to put all information together without experience of unraid.

Most sources point the user to use just array, and cache (i guess main usecase for the os is simple nas box). Cache pool of two drives seems to be the preferred way and default option for pool is mirror if I got this correct. However I found some posts that lead me to think there is possibility of using other configurations aswell with more ssds.

 

In my use case, would it be possible and good practise to use existing 120gb kingston ssds as cache for array, and mount 1tb intel nvme drive separate from array and cache, and put all vm's, dockers etc there? I found some information that backing up vm's should be quite simple task, so I could get away with one cheap QLC ssd as a primary drive for vm's. For me it would be secure enough if I could automate backing up the vm's and dockers maybe once a week to the hdd array. This way I could use cheap fast nvme drive for vms and dont need it to me mirrored. Im sure its not optimal, but I'd like to get some use out of that drive that I have already.

 

Another option I was thinking... If I ditch those 120gb ssd's and only use 1Tb nvme drive as cache for both array and vm's/dockers etc. I suppose I can pool it with slower 1tb sata ssd, but can I somehow choose that nvme drive to be the primary drive where I want everything to be written first, so I could benefit from the speed? Or will this pool limit speed of the cache to the slowest member of the pool?

I found a post saying that pool speed will be limited to slowest. Need to test if I can use second M.2 slot (M2B) for similar nvme drive and make a pool of 2 that way. Its only pci-e 2.0 x4, M2A slot is 3.0 x4 which im using at the moment.

Edited January 9, 2020 by Divi

[testdasi]

On 1/7/2020 at 8:17 PM, Divi said:

So I currently have separate Windows 10 gaming pc, and freenas server running pihole, syncthing etc. I'm looking to put all under one machine, and so far unraid looks like best option for me. 

 

...

 

AMD Ryzen 5 2600

Gigabyte Aorus X470 Ultra Gaming

Nvidia GT520 for host

AMD RX580 8Gb to passthrough for windows

24Gb of ram

1tb Intel 660p nvme m.2 ssd

2x 120gb Kingston A400 sata ssd

2x 4Tb Seagate Ironwolf

Dual port pcie intel nic for 2x1gbe lacp with vlans

Silent Base 801 case

Seasonic Focus Plus 550w psu.

 

How would you suggest setting it all up?

...

Let's start with the good news.

You have a Gigabyte mobo which has a BIOS option for you to pick which PCIe slot as "Initial Display Output" i.e. for Unraid host.

That will save you needing to plug the GT520 on the fast 1st PCIe slot. IIRC the 520 is a single-slot GPU so you may even be able to use the bottom PCIe slot for it.

It will also allow your main GPU to be secondary GPU (despite being plugged in the fast 1st PCIe slot) which tends to make life a lot easier with PCIe passthrough.

 

Now the not-so-good news.

The RX580 is notoriously not happy with being passed through - I have seen many posts with issues on here.

There is a "Solved" post which reported success with having the RX 580 as secondary GPU + vfio-stub it so you might still be ok.

I would highly recommend watching SpaceInvaderOne guides on youtube, particularly the ones about passing through GPU (I think he has 3 vids varying from basic to advanced tweaks, you should watch them all)

 

Now with regards to storage.

• It is impossible to pass through the 660p via PCIe method to a VM on Linux host (e.g. Unraid) due to Linux kernel not liking the controller). So the best you can do is mount it as Unassigned Devices and put a vdisk on it (or pass it through via ata-id method).
  • The 660p real life performance varies from "as good as" to "worse than" a good SATA SSD. Keep that in mind if you are considering purchasing another SSD. QLC for SSD is like SMR for HDD i.e. cheap but not cheerful.
  • I personally prefer a vdisk instead of ata-id passthrough since I can somewhat benefit more from TRIM with a qcow2 vdisk + scsi device (see johnnie's guide in the VM FAQ of the VM forum).
• You do NOT need cache for the array but you should still have at least a cache drive.
  • The original intent for the cache pool to serve as write cache came about before (a) reconstruct write aka turbo write was implemented and (b) HDD were still ultra slow. Nowadays with turbo write on, you can run pretty close to the max speed of your slowest HDD while writing.
  • The cache drive is now used for docker image, libvirt, appdata etc. They are pretty important for a smooth experience with Unraid if you are using it beyond a simple NAS.
• Mirrored cache pool is only required if you have critical data to protect against drive failure. I used to run a pool but now go for single-drive cache and instead focus on making sure my backup strategy is up to par.
  • One of the main reasons is that I have found SSD (that is not Intel-based) incredibly resilient to catastrophic failure. They tend to fail very gracefully as dead cells are gradually replaced with reserved cells and eventually you just lose capacity as the reserve runs out.
  • Intel, in contrast, engages in the anti-consumer practice of locking your SSD in read-only mode if all reserves are used under the pretext of data protection. While it does take a relatively long time to use up all the reserve, it is still anti-consumer and the practice is even more concerning with QLC.

Edited January 9, 2020 by testdasi

[Divi]

39 minutes ago, testdasi said:

Let's start with the good news.

You have a Gigabyte mobo which has a BIOS option for you to pick which PCIe slot as "Initial Display Output" i.e. for Unraid host.

That will save you needing to plug the GT520 on the fast 1st PCIe slot. IIRC the 520 is a single-slot GPU so you may even be able to use the bottom PCIe slot for it.

It will also allow your main GPU to be secondary GPU (despite being plugged in the fast 1st PCIe slot) which tends to make life a lot easier with PCIe passthrough.

 

Now the not-so-good news.

The RX580 is notoriously not happy with being passed through - I have seen many posts with issues on here.

There is a "Solved" post which reported success with having the RX 580 as secondary GPU + vfio-stub it so you might still be ok.

I would highly recommend watching SpaceInvaderOne guides on youtube, particularly the ones about passing through GPU (I think he has 3 vids varying from basic to advanced tweaks, you should watch them all)

 

Now with regards to storage.

• It is impossible to pass through the 660p via PCIe method to a VM on Linux host (e.g. Unraid) due to Linux kernel not liking the controller). So the best you can do is mount it as Unassigned Devices and put a vdisk on it (or pass it through via ata-id method).
  • The 660p real life performance varies from "as good as" to "worse than" a good SATA SSD. Keep that in mind if you are considering purchasing another SSD. QLC for SSD is like SMR for HDD i.e. cheap but not cheerful.
  • I personally prefer a vdisk instead of ata-id passthrough since I can somewhat benefit more from TRIM with a qcow2 vdisk + scsi device (see johnnie's guide in the VM FAQ of the VM forum).
• You do NOT need cache for the array but you should still have at least a cache drive.
  • The original intent for the cache pool to serve as write cache came about before (a) reconstruct write aka turbo write was implemented and (b) HDD were still ultra slow. Nowadays with turbo write on, you can run pretty close to the max speed of your slowest HDD while writing.
  • The cache drive is now used for docker image, libvirt, appdata etc. They are pretty important for a smooth experience with Unraid if you are using it beyond a simple NAS.
• Mirrored cache pool is only required if you have critical data to protect against drive failure. I used to run a pool but now go for single-drive cache and instead focus on making sure my backup strategy is up to par.
  • One of the main reasons is that I have found SSD (that is not Intel-based) incredibly resilient to catastrophic failure. They tend to fail very gracefully as dead cells are gradually replaced with reserved cells and eventually you just lose capacity as the reserve runs out.
  • Intel, in contrast, engages in the anti-consumer practice of locking your SSD in read-only mode if all reserves are used under the pretext of data protection. While it does take a relatively long time to use up all the reserve, it is still anti-consumer and the practice is even more concerning with QLC.

Awesome information!

 

I've watched few spaceinvader videos but will look more, there was 100+ on the playlist so its a bit challenging to find the relevant ones.

I will setup the thing in testbench and see what I can do with it. Maybe try the vdisk method to use nvme drive for vms, and possibly one sata ssd as a cache. Everything critical stays on the array anyway, and I have offline backup also.

[Divi]

Throw the parts on test bench... RX580 in first 16x slot, GT520 in second and Pro1000/PT in last one. Seems like a nice setup, everything has room to breathe and managed to set GT520 as primary in bios. Time to update bios, make usb stick and get everything else ready. Hopefully next weekend I'll have the case to build, and start figuring out how to get drives and data from freenas without screw-ups. I have couple offline backups (wd green in the picture and another 1tb usb hdd) that are luckily still able to hold all of my most precious data so should be fine... :)

 

[Divi]

Could not wait, had to put some old extra hdd in, make array and see how the passthrough goes. So far we are looking good.

 

Got windows installed in novnc, configured some power management settings, installed virtio drivers. Then passthrough, pcie acs override broke native groups out really nice. Had some freezes and other problems after installing amd drivers, but changing machine type to Q35 seemed to fix it all. Also windows start menu stopped working couple times but has not happened anymore. 

 

Now I dont have the 1tb nvme drive in yet, so well see it that one will mess up the passthrough.

 

edit: Also tried passing through USB controller: Chipset USB controller has 4x external USB2.0, 4x internal USB2.0, 2x internal USB3.1 and internal USB-C 3.1 Gen 2. This has [RESET] and is in its own IOMMU group 13 = perfect thing for passing through, no problems at all. USB sound card (DacMagic 100) works fine now, keyboard and mouse are passed as separate devices so they wont eat 2 ports from the back leaving more room for other stuff. There is also two more controllers, one in CPU and one ASMedia. ASMedia controller only has one external USB 3.1 Gen2 and one external USB-C 3.1 Gen2, I dont use these so I put boot stick here so I could also pass thru 4-port usb 3.1 Gen1 controller which is in CPU, but its not needed atm. Might still want USB ports for host, maybe for UPS or something like that.

 

I think I'll stop posting in this thread now as everything seems to work great and hardware is up for the task. When build is finished I'll post it in UCD section. If anyone else is looking to use Gigabyte Aorus X470 Ultra Gaming for something like this, go for it! Its not bad at all!

 

Big time thanks to testdasi, your answer was big encouragement for me to go on with Unraid!

 

 

Edited January 12, 2020 by Divi