The Black Bear - Threadripper 2990WX build

testdasi · February 15, 2020

Final updates on the VMWare adventure:

I reverted back to running Windows VM under Unraid 🤣 The limitations were too much to justify a little bit better performance
- It only took 3 hours to roll back (having backups and available storage are pretty big factors behind the rapid recover).
Created a storage pool for my 2x PM983 running in RAID-0 equivalent just so I don't have to manage 2 different drives. Now it makes me itchy to buy another PM983 so I can run RAID-5 equivalent.
Also doing hardware transcoding in Windows now with GPU (instead of docker with CPU).
- I am in awe of the number of streams my GTX 1070 is capable of (with the helped of Dr.G 😉 ). What used to take an hour, now takes 5 minutes.
- Have to resist moving Plex to my workstation instead of docker.
  - The most important reason is I don't have a Plex Pass 😅.
  - Having to convert Linux-based Plex db to Windows-based is a pain in the backside.
  - My docker with about 1/4 the CPU power is fully capable of doing what I need it to do for media consumption.

Edited February 15, 2020 by testdasi

testdasi · February 16, 2020

Update on the storage pool:

I reverted back to non-RAID (i.e. 2 independent drives) after much tinkering with storage pool and ReFS.
- The main reason I wanted to try storage pool was ReFS promised data integrity feature i.e. it checksum data i.e. similar to btrfs / ZFS. Then after many tries, I gave up because scrubbing doesn't work. What is the effing point of having hashing without the ability to scrub. Sure I know I can't recover data since I run RAID-0 but all the other FS allows me to scrub regardless. I can always restore from a backup!
- While trying to make scrubbing work (to no avail), my frustration was only heightened by the discovery that ReFS can silently delete files that it finds integrity break on. This is an atrocious design for a file system. Sure the file may be corrupt but at least partial recovery is better than no recovery. And deleting silently (regardless if it will or not) is a big no-no for any FS.
- I now suspect Microsoft decision to remove ReFS creation on Windows Pro wasn't a marketing decision to push Windows Workstation. They just realised ReFS is shit and want to reduce the chance of class-action lawsuits.
- Btw, Windows Enterprise will allow formatting any drive to ReFS without the need of storage pool so there's really no need to add to the risk by running RAID-0 anyway.
I am now back to square one to solve a very much first-world problem of trying to hash my data on Windows (and hopefully in the process not have to deal with 2 independent drives).

jbartlett · February 17, 2020

I played around with GPU transcoding but ended up shying away from it. Even though it's faster, there is a quality decrease and I haven't yet managed to work my "gotta be high quality" mindset yet. 😂 (crying tears, not laughing tears). I spent hours performing multiple Handbrake 4K HDR encodes looking at zoomed in screen caps comparing the compression quality against the original to find one that gives a good size reduction with minimal quality loss.

I run Plex in a Win10 VM that also does other stuff and I have had zero issues with it. I made that move because I had to do a couple rebuilds on the Docker version though that was way back when unraid first started supporting Docker. I've got a separate virtual drive configured for it so it's not bloating the VM OS/data drives.

testdasi · February 18, 2020

15 hours ago, jbartlett said:

I spent hours performing multiple Handbrake 4K HDR encodes looking at zoomed in screen caps comparing the compression quality against the original to find one that gives a good size reduction with minimal quality loss.

Does Handbrake NVENC support HDR?

I thought it only does 8-bit.

jbartlett · February 18, 2020

8 hours ago, testdasi said:

Does Handbrake NVENC support HDR?

I thought it only does 8-bit.

It's still 8-bit. I should have broken that up or clarified better that the HDR part was just me demonstrating my "issues" 😁 Since I process the blu-ray along with the 4K and don't post both at the same time, I went even higher quality on the blu-ray. I also create a 1080p HDR version.

That said, if NVENC does ever support HDR, I'll probably swallow that pill and switch. Waiting nearly a day for a 4K to process on my TR2...

One thing to note is if you want to maximize your core to encode and do your encodes in a VM like I do, set up the VMs to utilize only six cores (12 threads). More cores have diminishing returns with Handbrake.

testdasi · February 26, 2020

More tinkering:

I got a bit annoyed with needing to plugin 3 USB devices for my offline backup so had (another) rethink.
- Bought a 2-bay non-RAID USB enclosure for the 2x 8TB HDD to serve as my offline backup (so I only need to plugin one device). It is off most of the time at the back of the bookshelf so a bit less HDD noise on a daily basis but the enclosure has terrible coil whine when on.
- Reuse my old 15mm 2.5" USB enclosure for the 5TB Seagate 2.5" SMR HDD for "archival" storage (i.e. things that I'm only 99% sure I can delete).
- Plug in all of the SATA SSD's back onto the server and use them for online backup. I originally had these in the array but then decided to use mergerfs instead. The main reasons are (a) I can trim the SSD and (b) without the need for parity, mergerfs and Unraid shfs essentially offer the same functionality.
- My array, as a result, now only has the 10TB HDD, which is on 3-hour-spin-down setting.
I now have 2 mergerfs bash scripts. One to pool the internal SSD's to create a online backup location. One to pool the external USB HDD's when they are connected to create the offline backup location.
While running btrfs scrubs, I noticed that disk activities do not respect my isolcpus settings. It's still using the core that I isolated (and behaved just like when one assign an isolated core to a docker i.e. max it out at 100%). So I raised a bug report and it seems to have been fixed by adding some additional options to syslinux e.g.
```
isolcpus=32-63 nohz_full=32-63 rcu_nocs=32-63
```
- Maybe placebo effect but I also noticed things to be a bit springier in my VM after the tweaks too.

jbartlett · February 26, 2020

Since your backup drive is a bit away due to noise, check out the optical USB extensions. I've got this product but I've yet to actually hook it up though. No power over USB so it'll need to hook up to a powered hub or in your case, a powered device.

https://www.amazon.com/gp/product/B00JOJRF6K/

I'd shy away from the USB3 active extensions. I've had a LOT of trouble finding cables that'll handle a high data load over USB3 (Logitech Brio 4k web cam). Even same brand has mixed results, likely a "silicone lottery" deal.

testdasi · March 13, 2020

Update:

I found good deals on 2 additional 4TB Samsung 860 Evo. My array now consists of 3x 4TB SSD and the 10TB HDD.
- I originally wanted to use the 10TB as parity but that would waste 60% of the drive capacity (which I can use for other purposes) while reducing write speed (which would defeat the purpose of getting SSD). Then I realised I don't need parity protection for all the data on those SSD's (just my main backup, which doesn't change that often, less than daily) and 12TB is kinda close to 10TB anyway.
- So instead of having 10TB parity, I setup a regular script to create copies of each of the 3x 4TB SSD on the 10TB. This would protect me from a single drive failure, just like parity without compromising speed and available capacity. This only works because the 3x 4TB SSD don't contain regularly updating data (that would require live parity).
Then found another good deal on a 960GB Intel Optane 905p so pulled (another) trigger on that and use it as my working drive for the VM.
- The non-boot 905p drops latency by about 4us vs the boot drive, about 7% faster.
- The old working drive (2TB Samsung 970 Evo) becomes the temp drive.
- The old temp drive (1.2TB Intel 750) is removed from the VM and mounted unassigned. I moved all the static data out of cache and into this UD drive. With the helps of symlinks, I still can use Unraid share feature with the data stored on UD.
- My cache now has 98% available space for write-heavy activities (which I am using it for).
The 2x 2TB SATA SSD (Samsung 850 Evo and Crucial MX300) are pooled using mergerfs for a 2nd array (that supports trim). This is used for data that I don't need protection. They will also get written to more frequently than the ones in the actual array.
I also set up another mergerfs pool that pool the available space of (in order) the HDD, MX300, 850 Evo, 3x 860 Evo to create a 3rd space-priority array. This will be used when I need to dump a very large (>10TB) amount of data to the server with speed being less of a concern. This is pretty niche use but I'm glad mergerfs helps with this.
It would help if Limetech can implement multiple-array feature asap (preferably with trim!) but mergerfs is good enough for now.

Today is Friday the 13th. COVID-19 is a pandemic. Prayers to the human race.

Edited March 13, 2020 by testdasi

DZMM · March 13, 2020

1 hour ago, testdasi said:

With the helps of symlinks, I still can use Unraid share feature with the data stored on UD.

Can you give an example of your symlink command please. I do a similar thing with my UD drives, but I use bind mounts e.g. I have my plex metadata on a UD 970 evo plus:

mount --bind "/mnt/disks/ud_970evoplus/appdata/dockers/plex" "/mnt/cache/appdata/dockers/plex"

Also, I think you should do a seperate post when you have time of all the TR changes you've made as there's a lot of useful stuff in this thread.

testdasi · March 13, 2020

2 hours ago, DZMM said:

Can you give an example of your symlink command please. I do a similar thing with my UD drives, but I use bind mounts e.g. I have my plex metadata on a UD 970 evo plus:

I use ln.

Let's say you want to point /mnt/cache/personal/doc to /mnt/disks/ssd/documents then do:

ln -sv /mnt/disks/ssd/documents /mnt/cache/personal/doc

Unlike mount, it's important that you don't have a folder called "doc" at /mnt/cache/personal or the command will create a documents symlink under /mnt/cache/personal/doc (i.e. it would become /mnt/cache/personal/doc/documents).

I like symlinks for simple stuff since I just have to do it once and it persists across reboots.

I think bind mount needs to be redone after reboot.

But then Plex db has a lot of links too (if I remember correctly) so you might not be able to use symlink with it.

With dockers though, I just point the path directly to the location in the docker config. I find that even more straightforward.

Kurkoko · March 19, 2020

Hello, maybe is already answered but I just want to make sure about your build.

Does the SATA and M.2 share lanes? I mean for example on my workstation the m.2 on cpu lanes shares bandwith with sata 0,1. Does threadripper solves that with all his lanes?

I searched on you motherboard manual but also can't find about PCIE slots sharing lanes.

I'm right then about that this x399 designare ex doesn't share lanes between PCIE, SATA and M.2? I mean if you use a M.2 slot a SATA port is disabled, if use the PCIE number X the M.2 slot is disabled,....?

All my questions are related about if I finally go for your build but with a 2920X without an extra LSI HBA. The 8 sata ports + M.2 slots are great.

You still recommend this motherboard? Even for a single Gaming Windows 10 VM and lots of dockers?

Sorry about my english and if any of my questions are stupid.

testdasi · March 25, 2020

On 3/19/2020 at 6:15 PM, Kurkoko said:

Hello, maybe is already answered but I just want to make sure about your build.

Does the SATA and M.2 share lanes? I mean for example on my workstation the m.2 on cpu lanes shares bandwith with sata 0,1. Does threadripper solves that with all his lanes?

I searched on you motherboard manual but also can't find about PCIE slots sharing lanes.

I'm right then about that this x399 designare ex doesn't share lanes between PCIE, SATA and M.2? I mean if you use a M.2 slot a SATA port is disabled, if use the PCIE number X the M.2 slot is disabled,....?

All my questions are related about if I finally go for your build but with a 2920X without an extra LSI HBA. The 8 sata ports + M.2 slots are great.

You still recommend this motherboard? Even for a single Gaming Windows 10 VM and lots of dockers?

Sorry about my english and if any of my questions are stupid.

Hi, sorry forgot to reply.

I used all M.2 slots, all PCIe slots and all SATA ports at one point in the past so I can confirm.

M.2 NVMe does not disable SATA ports.
M.2 NVMe also does not disable any PCIe slot (and vice versa).

The whole point of getting Threadripper is the number of PCIe lanes so whether I would recommend the board or not depends on how many PCIe lanes you need. For example, I have a rather unusual number of NVMe drives compared to a typical user so I do need all of those lanes.

Kurkoko · March 26, 2020

Thank you very much!!!

After all of your updates, how is the system stability now? Same Bios firmware?

About PCIe lanes I'm finishing my build list. I need the 8 sata, extra M.2 slots for direct drives for VMs and pcie slots for win10 VM gaming daily driver/moonlight to laptop.

3900X seems don't have enought lanes for my needs.

Thanks again!

testdasi · March 26, 2020

37 minutes ago, Kurkoko said:

Thank you very much!!!

After all of your updates, how is the system stability now? Same Bios firmware?

About PCIe lanes I'm finishing my build list. I need the 8 sata, extra M.2 slots for direct drives for VMs and pcie slots for win10 VM gaming daily driver/moonlight to laptop.

3900X seems don't have enought lanes for my needs.

Thanks again!

Rock solid. No instability to speak of.

I'm running an older F12e BIOS due to issues I reported on page 3 about F12i BIOS.

There is the latest F12 (no "e" nor "i") BIOS which I can't be bothered to update to since as mentioned, everything is rock solid.

For a primarily gaming build, I would recommend you also consider Intel single-die CPU offering too. While having lower maximum performance, Intel single-die design means you get more consistent gaming performance (e.g. lower latency, less fps variability aka stuttering etc.).

My VM is workstation-first, gaming-second (and I can't tell the diff with fps variability but I know someone who can) so TR is perfect for me.

Edited March 26, 2020 by testdasi

Spritzup · March 29, 2020

So I'm at a loss here and hoping that someone who has been through this can assist. Long story short, built a Threadripper 2970wx system using this motherboard. However, where I'm struggling is setting up two VM's with USB passthrough, as well as passing through two Western Digital Black NVMe to the above VM's.

Any help or guidance that you guys can offer will be greatly appreciated!

Thanks.

~Spritz

testdasi · March 29, 2020

44 minutes ago, Spritzup said:

So I'm at a loss here and hoping that someone who has been through this can assist. Long story short, built a Threadripper 2970wx system using this motherboard. However, where I'm struggling is setting up two VM's with USB passthrough, as well as passing through two Western Digital Black NVMe to the above VM's.

Any help or guidance that you guys can offer will be greatly appreciated!

Thanks.

~Spritz

Start a new topic in the General forum and I'll try to help.

Don't forget to attach diagnostics (Tools -> Diagnostics -> attach zip file) as well as describing what you did, errors etc.

When copy-paste text taken from the GUI / error / syslog, please use the forum code functionality (the </> button next to the smiley button) so the text is formatted correctly.

Spritzup · March 29, 2020

@testdasi Thank you for your help. I didn't attach the zip file because I had rebuilt the box since posting, so no error logs.

That said, here is the topic --> ThreadRipper Build Issue's

~Spritz

Edited March 29, 2020 by Spritzup

SimonG · May 12, 2020

I've just swapped out my motherboard for a Gigabyte X399 Designare EX with F12 BIOS

With IOMMU in the BIOS is set to "On" (but without ACS Override multifunction) my groups appear very different to yours making it difficult to passthrough the graphics card. Any idea why this might be?

IOMMU group 0:	[1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
[1022:1453] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
[1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
[1022:43ba] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset USB 3.1 xHCI Controller (rev 02)
[1022:43b6] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset SATA Controller (rev 02)
[1022:43b1] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset PCIe Bridge (rev 02)
[1022:43b4] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
[1022:43b4] 02:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
[1022:43b4] 02:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
[1022:43b4] 02:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
[1022:43b4] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
[8086:24fd] 05:00.0 Network controller: Intel Corporation Wireless 8265 / 8275 (rev 78)
[8086:1539] 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
[10de:128b] 07:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1)
[10de:0e0f] 07:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
[1000:0072] 08:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
IOMMU group 1:	[1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
IOMMU group 2:	[1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
IOMMU group 3:	[1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
IOMMU group 4:	[1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
[1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
[1022:145a] 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function
[1022:1456] 09:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor
[1022:145c] 09:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller
IOMMU group 5:	[1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
[1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
[1022:1455] 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function
[1022:7901] 0a:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
[1022:1457] 0a:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller
IOMMU group 6:	[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 7:	[1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0
[1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1
[1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2
[1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3
[1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4
[1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5
[1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6
[1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7
IOMMU group 8:	[1022:1460] 00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0
[1022:1461] 00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1
[1022:1462] 00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2
[1022:1463] 00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3
[1022:1464] 00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4
[1022:1465] 00:19.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5
[1022:1466] 00:19.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6
[1022:1467] 00:19.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7
IOMMU group 9:	[1022:1452] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
[1022:1453] 40:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
[126f:2263] 41:00.0 Non-Volatile memory controller: Silicon Motion, Inc. Device 2263 (rev 03)
IOMMU group 10:	[1022:1452] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
IOMMU group 11:	[1022:1452] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
[1022:1453] 40:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
[10de:21c4] 42:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1660 SUPER] (rev a1)
[10de:1aeb] 42:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1)
[10de:1aec] 42:00.2 USB controller: NVIDIA Corporation Device 1aec (rev a1)
[10de:1aed] 42:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev a1)
IOMMU group 12:	[1022:1452] 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
IOMMU group 13:	[1022:1452] 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
[1022:1454] 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
[1022:145a] 43:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function
[1022:1456] 43:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor
[1022:145c] 43:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller
IOMMU group 14:	[1022:1452] 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
[1022:1454] 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
[1022:1455] 44:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function
[1022:7901] 44:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

Setting ACS override splits everything into it's own individual group but I'd rather not have to do this.

testdasi · May 13, 2020

15 hours ago, SimonG said:

I've just swapped out my motherboard for a Gigabyte X399 Designare EX with F12 BIOS

With IOMMU in the BIOS is set to "On" (but without ACS Override multifunction) my groups appear very different to yours making it difficult to passthrough the graphics card. Any idea why this might be?

Setting ACS override splits everything into it's own individual group but I'd rather not have to do this.

There's no diff if you account for the diff in the actual devices plugged into your system vs mine.

For example, you have the GTX 1660 and I have the GTX 1070.

Yours has 4 "functions" (42:00.0 -> 3, which are the GPU, the HDMI audio and 2x USB devices respectively).

Mine only has 2 "functions" (GPU + HDMI Audio).

All 4 functions of your GTX 1660 must be passed through together to the same VM for it to work.

(in the same way that all 2 functions of my 1070 must be passed through together to the same VM).

You need to watch SpaceInvaderOne tutorial on Youtube on how to stub (aka bind) PCIe devices (e.g. "functions") so they show up on the VM template for you to include in the pass through. (I think it's in the vid on passing through USB controller).

And don't forget to dump your own vbios.

If those things still don't work then share your xml and I can have a look.

PS: ignore any PCIe "bridge" device. They should not be passed through and will not interfere with pass through.

Edited May 13, 2020 by testdasi

SimonG · May 13, 2020

Thanks for your help. I've now managed to passthrough the graphics card and associated sound.

Had a quick try at passing through the USB controller on IOMMU 4 using vfio-pci.ids=1022:145c but I haven't been able to get it to work yet. Any further advice on this?

testdasi · May 14, 2020

14 hours ago, SimonG said:

Thanks for your help. I've now managed to passthrough the graphics card and associated sound.

Had a quick try at passing through the USB controller on IOMMU 4 using vfio-pci.ids=1022:145c but I haven't been able to get it to work yet. Any further advice on this?

You also need to bind 1022:1456 and 1022:145a.

You can pass through just 1022:145c but the other 2 devices also need to be stubbed for the USB controller to go through without IOMMU error.

SimonG · May 14, 2020

8 hours ago, testdasi said:

You also need to bind 1022:1456 and 1022:145a.

You can pass through just 1022:145c but the other 2 devices also need to be stubbed for the USB controller to go through without IOMMU error.

Am I write that in order to bind the three devices I use vfio-pci.ids=1022:145c,1022:1456,1022:145a ?

When I tried this Unraid lost it's Ethernet connection. Think I'm going to have to read up on this more.

testdasi · May 14, 2020

20 minutes ago, SimonG said:

Am I write that in order to bind the three devices I use vfio-pci.ids=1022:145c,1022:1456,1022:145a ?

When I tried this Unraid lost it's Ethernet connection. Think I'm going to have to read up on this more.

Those 3 devices have nothing to do with your network. If you lose connection, it's something else happening.

SimonG · May 16, 2020

On 5/14/2020 at 9:56 PM, testdasi said:

Those 3 devices have nothing to do with your network. If you lose connection, it's something else happening.

Strange because it's very repeatable. Add the line in and I get a DHCP network IP address. Remove it and it reverts to the default IP address.

I finally got everything working by switching to using the vfio-pci plugin. Just as I was tidying everything up and SpaceInvader releases a video using the exact method I'd used.

Thanks for your help. Apologies for gatecrashing your thread. It's a great read by the way and it helped me out a lot.

Gsusking2 · May 21, 2020

Hello @testdasi,

Amazing thread! A lot of great information here. I am using the same MOBO currently (had to replace a x399 that died recently).

Can i ask you what settings you used on the BIOS to get the USB to boot ? I created it using the UNRAID boot maker. It has been years since i had to finagle with the gigabyte bios and i am completely at a loss after reading the Limeware 'getting started guide'

If you could help shed any light on this situation it would be greatly appreciated.

The Black Bear - Threadripper 2990WX build

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

testdasi

testdasi

testdasi

Posted Images

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation