Trx40 motherboards


Dent_

Recommended Posts

In case someone stumbles across this and needs the info. I recently upgraded from a 1950x to a 3960x on a Aorus Extreme TRX40 and struggled the last couple of days getting my VMs back up running. Passing through both GPUs (1080ti, 1050ti) to different VMs, no issues, onboard USB controller also no issue. The problems started with the onboard audio controller. The common way is to find the controller in the devices list, stub/bind them and you should be good to go. In my case the controller is reported as

[1022:1487] 23:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller

BUT

this isn't the real audio controller that can be passed to a VM. I tried the 6.9 beta 1, tried different patched kernels for 6.8.3 with fixes for 3rd gen Ryzens (AMD onboard audio/usb controller flr patch, Vega-Navi-Patch etc.). During all my testings I completely missed, that this board provides 2 separate USB audio devices. There is no need for passing through the Starship/Matisse controller or using a patched kernel. It works straight out of the box with Unraid 6.8.3. No ACS patch or unsafe interrupts needed. Simply pick one of the 2 USB audio devices in the VM template and you're good to go. Windows will recognize it as "Realtek USB Audio" and on Linux (Zorin15.1) it also works straight out of the box. 👍

 

grafik.png.677d7a041b2034e9b966ea244972a8a5.png

 

The first device is for the front panel audio and the second one for the back panel 5.1/7.1 links. And guess what....

you can use them in 2 different VMs and run them simultaneously without issues or any extra tweaking needed. 😍

 

For reference my initial posting, trying to get help in the custom kernel thread:

 

Link to comment
  • 1 month later...
On 1/2/2020 at 1:49 PM, Nomy said:

I wanna throw a question out there!

Having a 3970x on a Asus Zenith 2 extreme. 
It's impossible to get a Firewire card to work on that motherboard. Computer wont boot pass bios. Tried everything I could find on google. Nothing changes.

Anyone got this to work and could tell me which firewire card you are using?

Appriciate any help!
/Patrick

It wont work. What you need to do to get firewire 800 to work is buy  startech thunderbolt 3 to thunderbolt 2 adapter and buy a Apple  thunderbolt 2 to firewire 800 1394B . Apple sell it for 29.99 and the startech on ebay between 50 bucks  and hundred bucks.  I got lucky an bought mine for 25 bucks.  Connect the startech to the thunderbolt titan ridge card and the apple to the startech thunderbolt 2 port.  In my case I went from the apple fire wire 800 port on my UAD apollo duo sound  external sound card. My apollo duo also has thunderbolt 1 gen connector and my titan ridge will not pick it up. I read you need to update  the TB card on the uad  sound card to thunderbolt 2 or 3 generation. Not worth it. Windows 10 will pick up the end fire wire 800 device and create drivers in device manager. Hope this helps. Also you have to use non doch drivers. need to use the legacy drivers for the thunderbolt titan ridge. It works flawlessly and stable. Love It!!!!!!

If you do it  like this WIndows 10 wont have any problem picking up anything connected to FW 800 cable or apple FW 800 port.

Link to comment
  • 3 weeks later...

Just setup a Asrock trx40 creator with 3970x. Plan was 2 x 2070 supers for vm's and a rtx 980 for bit of plex lol, No chance of getting them all in the system see photo's.

 

Passed through 1 x usb 3 controller fine to first vm but i had the FLR issue if tried to pass another one. Got round it by passing the usb c pcie to the other vm. The 2070 gpu's passed to vm's with no issues and i am using the gpu's audio as well. I don't need to use the onboard audio as gpu's are going to monitors via 20m fibre diplayports and 20m usb 3 active cables.

 

Also got 128gb of Corsair lpx ddr4 c16 no chance of getting it to run at 3200 crashes vm's and all sorts, but rock solid on 2666! what is everyone else running their memory at?

 

IMG_20200625_122601.thumb.jpg.89beceb5c95010939d76d828c222641a.jpgIMG_20200625_122532.thumb.jpg.c4b543206896f2c0116fec7521d4885b.jpg

IOMMU group 0:[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 1:[1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 2:[1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 3:[1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 4:[1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 5:[1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 6:[1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 7:[1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 8:[1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 9:[1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 10:[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)

[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)

IOMMU group 11:[1022:1490] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0

[1022:1491] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1

[1022:1492] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2

[1022:1493] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3

[1022:1494] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4

[1022:1495] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5

[1022:1496] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6

[1022:1497] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7

IOMMU group 12:[8086:10c9] 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

IOMMU group 13:[8086:10c9] 01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

IOMMU group 14:[1022:148a] 02:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function

IOMMU group 15:[1022:1485] 03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

IOMMU group 16:[1022:148c] 03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller

IOMMU group 17:[1022:1482] 20:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 18:[1022:1482] 20:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 19:[1022:1482] 20:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 20:[1022:1482] 20:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 21:[1022:1482] 20:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 22:[1022:1482] 20:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 23:[1022:1484] 20:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 24:[1022:1482] 20:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 25:[1022:1484] 20:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 26:[1022:148a] 21:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function

IOMMU group 27:[1022:1485] 22:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

IOMMU group 28:[1022:1486] 22:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP

IOMMU group 29:[1022:148c] 22:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller

IOMMU group 30:[1022:1487] 22:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller

IOMMU group 31:[1022:1482] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 32:[1022:1483] 40:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 33:[1022:1483] 40:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 34:[1022:1483] 40:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 35:[1022:1482] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 36:[1022:1482] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 37:[1022:1483] 40:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 38:[1022:1483] 40:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 39:[1022:1482] 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 40:[1022:1482] 40:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 41:[1022:1482] 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 42:[1022:1484] 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 43:[1022:1482] 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 44:[1022:1484] 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 45:[1022:57ad] 41:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream

IOMMU group 46:[1022:57a3] 42:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

IOMMU group 47:[1022:57a3] 42:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

IOMMU group 48:[1022:57a3] 42:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

IOMMU group 49:[1022:57a3] 42:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

IOMMU group 50:[1022:57a3] 42:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

IOMMU group 51:[1022:57a4] 42:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

[1022:1485] 48:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

[1022:149c] 48:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

[1022:149c] 48:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

IOMMU group 52:[1022:57a4] 42:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

[1022:7901] 49:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

IOMMU group 53:[1022:57a4] 42:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

[1022:7901] 4a:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

IOMMU group 54:[1987:5016] 43:00.0 Non-Volatile memory controller: Phison Electronics Corporation E16 PCIe4 NVMe Controller (rev 01)

IOMMU group 55:[1b21:3242] 44:00.0 USB controller: ASMedia Technology Inc. Device 3242

IOMMU group 56:[1d6a:07b1] 45:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)

IOMMU group 57:[8086:2723] 46:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)

IOMMU group 58:[10ec:8125] 47:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 01)

IOMMU group 59:[8086:f1a8] 4b:00.0 Non-Volatile memory controller: Intel Corporation SSD 660P Series (rev 03)

IOMMU group 60:[8086:f1a8] 4c:00.0 Non-Volatile memory controller: Intel Corporation SSD 660P Series (rev 03)

IOMMU group 61:[10de:1e84] 4d:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] (rev a1)

IOMMU group 62:[10de:10f8] 4d:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)

IOMMU group 63:[10de:1ad8] 4d:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)

IOMMU group 64:[10de:1ad9] 4d:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)

IOMMU group 65:[10de:1e84] 4e:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] (rev a1)

IOMMU group 66:[10de:10f8] 4e:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)

IOMMU group 67:[10de:1ad8] 4e:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)

IOMMU group 68:[10de:1ad9] 4e:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)

IOMMU group 69:[1022:148a] 4f:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function

IOMMU group 70:[1022:1485] 50:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

IOMMU group 71:[1022:1482] 60:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 72:[1022:1482] 60:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 73:[1022:1482] 60:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 74:[1022:1482] 60:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 75:[1022:1482] 60:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 76:[1022:1482] 60:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 77:[1022:1484] 60:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 78:[1022:1482] 60:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 79:[1022:1484] 60:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 80:[1022:148a] 61:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function

IOMMU group 81:[1022:1485] 62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

 

Link to comment
  • 2 months later...
On 6/25/2020 at 1:59 PM, turnipisum said:

Just setup a Asrock trx40 creator with 3970x. Plan was 2 x 2070 supers for vm's and a rtx 980 for bit of plex lol, No chance of getting them all in the system see photo's.

For your third card (980) you could use a pci-e cable. These are about 30 to 40 Euro.

 

I would like to know, how you configured your two gpus and your two vms. I am planing to buy a new system, a beefy one. What were the needed steps to get the two GPUs work? Thanks in advance.

Link to comment

Glad I stumbled upon this thread.  I am just upgrading my server from an old Dual Xeon build to a TR 3990X on a Zenith II Extreme Alpha.  Dockers seem to run fine after I disabled C-States in the BIOS but I cannot install a new VM to save my life.  Every time I set up a new VM and dedicate some cores to it it takes like 5-10 min to get to the windows install logo.  Whatever my lowest core selected in the VM template goes to 100% for that entire 5-10 min.  Anyone have any suggestions?  

 

 

- NEW AMD Ryzen Threadripper 3990X 64-Core, 128-Thread Unlocked Desktop Processor

- NEW ASUS ROG Zenith II Extreme Alpha TRX40 Gaming AMD 3rd Gen Ryzen Threadripper sTRX4 EATX Motherboard (Running on Newest Firmware)

- NEW 128 GB of Kingston KSM26ED8/16ME Server Premier - DDR4-16 GB - DIMM 288-pin - 2666 MHz / PC4-21300 - CL19-1.2 V - unbuffered - ECC

- NEW (2) Samsung 970 EVO Plus SSD 2TB - M.2 NVMe Interface Internal Solid State Drives - (want to pass through one to each gaming VM's via unassigned devices hopefully)

- (2) Cache Drives Samsung SSD 860 EVO 1TB 2.5 Inch SATA III Internal SSD's

- Existing 50TB Array transferred over to new build consisting of (5+Parity) Western Digital Gold 10TB Enterprise Class Internal Hard Drives

- NEW LSI 9211-8i P20 IT Mode for ZFS FreeNAS unRAID Dell H310 6Gbps SAS HBA for Managing the Array (PCIe Slot 2)

- Existing ZOTAC GeForce GT 730 Low Profile 4GB 64-Bit DDR3 PCI Express 2.0 x16 (PCIe Slot 1) Eventually will upgrade when I do a second Gaming VM via BIOS Dump

- NEW EVGA GeForce RTX 2060 SUPER SC BLACK GAMING, 8GB GDDR6, Dual Fans, 08G-P4-3062-KR (Slot 3)

- NEW ASUS Rog Thor 1200 Certified 1200W Fully-Modular RGB Power Supply

- EXISTING 4U Supermicro 730 Server Rack Mount Chasis.

 

Link to comment
  • 2 weeks later...
On 8/28/2020 at 11:22 AM, Michael Baecker said:

For your third card (980) you could use a pci-e cable. These are about 30 to 40 Euro.

 

I would like to know, how you configured your two gpus and your two vms. I am planing to buy a new system, a beefy one. What were the needed steps to get the two GPUs work? Thanks in advance.

I ditched the 980 in the end it seems to cause issues whenever i put in a unraid system! But yes, i didn't think of pcie extension lol.

The vm's just have the 2070's passed through. Works no problems it was the usb pass through from the MB that was a pain! there is a few threads about this on here.

Link to comment
On 8/28/2020 at 5:03 PM, Joshewing02 said:

Glad I stumbled upon this thread.  I am just upgrading my server from an old Dual Xeon build to a TR 3990X on a Zenith II Extreme Alpha.  Dockers seem to run fine after I disabled C-States in the BIOS but I cannot install a new VM to save my life.  Every time I set up a new VM and dedicate some cores to it it takes like 5-10 min to get to the windows install logo.  Whatever my lowest core selected in the VM template goes to 100% for that entire 5-10 min.  Anyone have any suggestions?  

 

 

- NEW AMD Ryzen Threadripper 3990X 64-Core, 128-Thread Unlocked Desktop Processor

- NEW ASUS ROG Zenith II Extreme Alpha TRX40 Gaming AMD 3rd Gen Ryzen Threadripper sTRX4 EATX Motherboard (Running on Newest Firmware)

- NEW 128 GB of Kingston KSM26ED8/16ME Server Premier - DDR4-16 GB - DIMM 288-pin - 2666 MHz / PC4-21300 - CL19-1.2 V - unbuffered - ECC

- NEW (2) Samsung 970 EVO Plus SSD 2TB - M.2 NVMe Interface Internal Solid State Drives - (want to pass through one to each gaming VM's via unassigned devices hopefully)

- (2) Cache Drives Samsung SSD 860 EVO 1TB 2.5 Inch SATA III Internal SSD's

- Existing 50TB Array transferred over to new build consisting of (5+Parity) Western Digital Gold 10TB Enterprise Class Internal Hard Drives

- NEW LSI 9211-8i P20 IT Mode for ZFS FreeNAS unRAID Dell H310 6Gbps SAS HBA for Managing the Array (PCIe Slot 2)

- Existing ZOTAC GeForce GT 730 Low Profile 4GB 64-Bit DDR3 PCI Express 2.0 x16 (PCIe Slot 1) Eventually will upgrade when I do a second Gaming VM via BIOS Dump

- NEW EVGA GeForce RTX 2060 SUPER SC BLACK GAMING, 8GB GDDR6, Dual Fans, 08G-P4-3062-KR (Slot 3)

- NEW ASUS Rog Thor 1200 Certified 1200W Fully-Modular RGB Power Supply

- EXISTING 4U Supermicro 730 Server Rack Mount Chasis.

 

There was a issue when setting up windows VM's. fix was setting it to use 1 core for setup then change back to full amount of cores. But i can't remember if that was for blank screen issue or what you are getting. 

Link to comment
  • 1 month later...

Has anybody managed to successfully pass through a GPU to a KVM/Qemu VM on an Asus Rog Zenith II Extreme Alpha TRX40 board?

I've has success with the Asrock Creator TRX40 board, but this new one is not working out so well.

Linux just refuses to use anything other than software bounce buffers for DMA, and no DMAR in sight.

I've tried every combination of kernel command line settings and related BIOS setting that I can find, but nothing seems to matter.

 

Rather than go through every combination I've tried, I figured I'd first ask if anybody managed to get it to work, and if so, what were the related BIOS settings and/or kernel command line used that ended up working for them.

 

Thanks!

Link to comment
1 hour ago, WolfRunner said:

Has anybody managed to successfully pass through a GPU to a KVM/Qemu VM on an Asus Rog Zenith II Extreme Alpha TRX40 board?

I've has success with the Asrock Creator TRX40 board, but this new one is not working out so well.

Linux just refuses to use anything other than software bounce buffers for DMA, and no DMAR in sight.

I've tried every combination of kernel command line settings and related BIOS setting that I can find, but nothing seems to matter.

 

Rather than go through every combination I've tried, I figured I'd first ask if anybody managed to get it to work, and if so, what were the related BIOS settings and/or kernel command line used that ended up working for them.

 

Thanks!

I use a VM as my daily driver with this motherboard and I don’t have any issues. I’m using a dumped vbios and I pass the GPU through as a multifunction device. I’m running the latest unRAID beta.

Link to comment
On 12/27/2019 at 3:38 PM, jwoolen said:

I finally got my 3970X and I've begun testing with an Asus Zenith II Extreme and an Aorus TRX40 Master. My main interests were in the rear panel USB controllers and M.2 connections to the CPU. My GPUs have water blocks on them and it will be a while before I'm able to test passthrough using them.

 

Zenith II Extreme:

 

Surprisingly, the Zenith II Extreme only offers one dedicated M.2 connection to the CPU (M.2_1). M.2_2 connects directly to the CPU, but it shares 4 lanes of bandwidth with PCIE_X16_4, which makes it unusable for me because I will need all 8 lanes of that slot for an HBA.

 

USB Controllers:

 

1541361647_ROGZENITHIIEXTREME-BACKIOlabeled.thumb.png.963b8e95879c6efdde342372a8340e66.png


03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
22:00.0 USB controller: ASMedia Technology Inc. Device 3242
24:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
49:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
49:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

The ports circled in red are connected to 49:00.3, the ports circled in blue are connected to  24:00.3, the ports circled in yellow are connected to 49:00.1, the ports circled in purple are connected to 03:00.3, and the port circled in pink is connected to 22:00.0.

 

The ASMedia controller and the AMD Starship controllers are in their own IOMMU group and can be reset. The AMD Matisse controllers are in the same IOMMU group and only one of them can be reset.

 

PCI Devices and IOMMU Groups:


IOMMU group 0:	[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 1:	[1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 2:	[1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 3:	[1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 4:	[1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 5:	[1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 6:	[1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 7:	[1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 8:	[1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 9:	[1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 10:	[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
		[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 11:	[1022:1490] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0
		[1022:1491] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1
		[1022:1492] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2
		[1022:1493] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3
		[1022:1494] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4
		[1022:1495] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5
		[1022:1496] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6
		[1022:1497] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7
IOMMU group 12:	[10de:1380] 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2)
		[10de:0fbc] 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)
IOMMU group 13:	[1022:148a] 02:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 14:	[1022:1485] 03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 15:	[1022:148c] 03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
IOMMU group 16:	[1022:1482] 20:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 17:	[1022:1482] 20:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 18:	[1022:1482] 20:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 19:	[1022:1483] 20:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 20:	[1022:1483] 20:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 21:	[1022:1482] 20:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 22:	[1022:1482] 20:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 23:	[1022:1482] 20:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 24:	[1022:1484] 20:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 25:	[1022:1482] 20:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 26:	[1022:1484] 20:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 27:	[144d:a804] 21:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961
IOMMU group 28:	[1b21:3242] 22:00.0 USB controller: ASMedia Technology Inc. Device 3242
IOMMU group 29:	[1022:148a] 23:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 30:	[1022:1485] 24:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 31:	[1022:1486] 24:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
IOMMU group 32:	[1022:148c] 24:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
IOMMU group 33:	[1022:1487] 24:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
IOMMU group 34:	[1022:1482] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 35:	[1022:1483] 40:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 36:	[1022:1482] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 37:	[1022:1482] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 38:	[1022:1482] 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 39:	[1022:1482] 40:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 40:	[1022:1482] 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 41:	[1022:1484] 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 42:	[1022:1482] 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 43:	[1022:1484] 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 44:	[1022:57ad] 41:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57ad
IOMMU group 45:	[1022:57a3] 42:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
IOMMU group 46:	[1022:57a3] 42:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
IOMMU group 47:	[1022:57a3] 42:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
IOMMU group 48:	[1022:57a3] 42:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
IOMMU group 49:	[1022:57a3] 42:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
IOMMU group 50:	[1022:57a3] 42:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
IOMMU group 51:	[1022:57a4] 42:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
		[1022:1485] 49:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
		[1022:149c] 49:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
		[1022:149c] 49:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
IOMMU group 52:	[1022:57a4] 42:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
		[1022:7901] 4a:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 53:	[1022:57a4] 42:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
		[1022:7901] 4b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 54:	[144d:a808] 43:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
IOMMU group 55:	[1d6a:07b1] 44:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
IOMMU group 56:	[1b21:0612] 45:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
IOMMU group 57:	[1b21:0612] 46:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
IOMMU group 58:	[8086:1539] 47:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
IOMMU group 59:	[8086:2723] 48:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)
IOMMU group 60:	[1022:148a] 4c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 61:	[1022:1485] 4d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 62:	[1022:1482] 60:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 63:	[1022:1482] 60:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 64:	[1022:1482] 60:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 65:	[1022:1482] 60:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 66:	[1022:1482] 60:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 67:	[1022:1482] 60:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 68:	[1022:1484] 60:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 69:	[1022:1482] 60:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 70:	[1022:1484] 60:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 71:	[1022:148a] 61:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 72:	[1022:1485] 62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

 

TRX40 Aorus Master:

 

This board is much more practical for me. It offers the same VRM configuration as Gigabyte's top end boards and two M.2 drives from the CPU, which seems to be the most possible without reallocating lanes dedicated to the expansion slots.

 

USB Controllers:

 

23246299_GIGABYTETRX40AorusMasterRearPanel.png.b80de4a7ea2398ba1d7a1db2bc5ac6f4.png


03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
23:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
47:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
47:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

The ports circled in red are connected to 47:00.1, the ports circled in blue are connected to  23:00.3, and the ports circled in yellow are connected to 03:00.3.

 

The AMD Starship controllers are in their own IOMMU group and can be reset. The AMD Matisse controllers are in the same IOMMU group and only one of them can be reset.

 

PCI Devices and IOMMU Groups:


IOMMU group 0:	[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 1:	[1022:1483] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 2:	[1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 3:	[1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 4:	[1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 5:	[1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 6:	[1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 7:	[1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 8:	[1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 9:	[1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 10:	[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
		[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 11:	[1022:1490] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0
		[1022:1491] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1
		[1022:1492] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2
		[1022:1493] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3
		[1022:1494] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4
		[1022:1495] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5
		[1022:1496] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6
		[1022:1497] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7
IOMMU group 12:	[144d:a804] 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961
IOMMU group 13:	[1022:148a] 02:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 14:	[1022:1485] 03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 15:	[1022:148c] 03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
IOMMU group 16:	[1022:1482] 20:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 17:	[1022:1482] 20:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 18:	[1022:1482] 20:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 19:	[1022:1483] 20:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 20:	[1022:1482] 20:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 21:	[1022:1482] 20:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 22:	[1022:1482] 20:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 23:	[1022:1484] 20:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 24:	[1022:1482] 20:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 25:	[1022:1484] 20:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 26:	[10de:1380] 21:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2)
		[10de:0fbc] 21:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)
IOMMU group 27:	[1022:148a] 22:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 28:	[1022:1485] 23:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 29:	[1022:1486] 23:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
IOMMU group 30:	[1022:148c] 23:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
IOMMU group 31:	[1022:1487] 23:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
IOMMU group 32:	[1022:1482] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 33:	[1022:1483] 40:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 34:	[1022:1482] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 35:	[1022:1482] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 36:	[1022:1482] 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 37:	[1022:1482] 40:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 38:	[1022:1482] 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 39:	[1022:1484] 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 40:	[1022:1482] 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 41:	[1022:1484] 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 42:	[1022:57ad] 41:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57ad
IOMMU group 43:	[1022:57a3] 42:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
IOMMU group 44:	[1022:57a3] 42:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
IOMMU group 45:	[1022:57a3] 42:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
IOMMU group 46:	[1022:57a3] 42:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
IOMMU group 47:	[1022:57a4] 42:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
		[1022:1485] 47:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
		[1022:149c] 47:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
		[1022:149c] 47:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
IOMMU group 48:	[1022:57a4] 42:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
		[1022:7901] 48:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 49:	[1022:57a4] 42:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
		[1022:7901] 49:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 50:	[144d:a808] 43:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
IOMMU group 51:	[8086:1539] 44:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
IOMMU group 52:	[1d6a:11b1] 45:00.0 Ethernet controller: Aquantia Corp. AQC111 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
IOMMU group 53:	[8086:2723] 46:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)
IOMMU group 54:	[1022:148a] 4a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 55:	[1022:1485] 4b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 56:	[1022:1482] 60:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 57:	[1022:1482] 60:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 58:	[1022:1482] 60:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 59:	[1022:1482] 60:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 60:	[1022:1482] 60:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 61:	[1022:1482] 60:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 62:	[1022:1484] 60:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 63:	[1022:1482] 60:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 64:	[1022:1484] 60:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 65:	[1022:148a] 61:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 66:	[1022:1485] 62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

 

I hope this information is helpful to someone. I'll be trying more stuff like setting up VMs soon.

 

Hi there,

 

Did you ever do any further testing with the Gigabyte TRX40 Aorus Master in regards to GPU passthrough?

 

I'm looking at this board it seems like one of the better non-Extreme choices.

Link to comment
7 hours ago, Cadal said:

@jwoolen that goes for secondary passthrough of GPU? I have a 1080ti and 970 and cant for the life of me get the 970 to passthrough without an error 43 - got everything else working though.

I’ve only tried with a single GPU, using a dumped vbios and passing the GPU as a multifunction device. I’ll try adding a second GPU and see how it goes.

Edited by jwoolen
Link to comment

Has anybody else seen their GPU and HD audio device in the same card split into two separate IOMMU groups?

I can't seem to find a way to merge them:

 

[root@tiamat kvm]# lspci | grep 4b
4b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (rev c4)
4b:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio
[root@tiamat kvm]# iommu | grep 4b
/sys/kernel/iommu_groups/53/devices/0000:4b:00.0
/sys/kernel/iommu_groups/54/devices/0000:4b:00.1
[root@tiamat kvm]# 
 

I don't know if this is what is causing me grief or what...

 

At one point I managed to get the GPU passed through to Windows10, but the audio over HDMI did not work.

Link to comment

I've just got this board setup with a 3970x and am trying to get some VM passthrough going. My goal is to utilize the 2 CPU linked NVME ports each to be passed to their own VM. Can anyone help me understand which of these IOMMU groups would apply to the NVMEs? Every other example of NVME passthrough from spaceinvader etc seems to specifically mention "NVME" in the IOMMU group description, but this Aorus master doesn't have anything like that from what I can tell with or without ACS override.

 

i believe it is Group 13/14, but if that's the case is there any way to know which one belongs to which drive? There is also a 3rd NVME not through the CPU on this board (not installed at this time).

 

Here are my groupings currently:

 

IOMMU group 0:	[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 1:	[1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 2:	[1022:1483] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 3:	[1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 4:	[1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 5:	[1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 6:	[1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 7:	[1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 8:	[1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 9:	[1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 10:	[1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 11:	[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 12:	[1022:1490] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0
[1022:1491] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1
[1022:1492] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2
[1022:1493] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3
[1022:1494] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4
[1022:1495] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5
[1022:1496] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6
[1022:1497] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7
IOMMU group 13:	[15b7:5006] 01:00.0 Non-Volatile memory controller: Sandisk Corp Device 5006
IOMMU group 14:	[15b7:5006] 02:00.0 Non-Volatile memory controller: Sandisk Corp Device 5006
IOMMU group 15:	[1022:148a] 03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 16:	[1022:1485] 04:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 17:	[1022:148c] 04:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
IOMMU group 18:	[1022:1482] 20:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 19:	[1022:1482] 20:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 20:	[1022:1482] 20:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 21:	[1022:1483] 20:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 22:	[1022:1482] 20:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 23:	[1022:1482] 20:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 24:	[1022:1482] 20:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 25:	[1022:1484] 20:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 26:	[1022:1482] 20:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 27:	[1022:1484] 20:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 28:	[10de:1c02] 21:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] (rev a1)
[10de:10f1] 21:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)
IOMMU group 29:	[1022:148a] 22:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 30:	[1022:1485] 23:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 31:	[1022:1486] 23:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
IOMMU group 32:	[1022:148c] 23:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
IOMMU group 33:	[1022:1487] 23:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
IOMMU group 34:	[1022:1482] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 35:	[1022:1483] 40:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 36:	[1022:1482] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 37:	[1022:1482] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 38:	[1022:1482] 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 39:	[1022:1482] 40:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 40:	[1022:1482] 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 41:	[1022:1484] 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 42:	[1022:1482] 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 43:	[1022:1484] 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 44:	[1022:57ad] 41:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream
IOMMU group 45:	[1022:57a3] 42:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 46:	[1022:57a3] 42:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 47:	[1022:57a3] 42:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 48:	[1022:57a4] 42:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 49:	[1022:57a4] 42:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 50:	[1022:57a4] 42:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 51:	[8086:1539] 43:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
IOMMU group 52:	[1d6a:11b1] 44:00.0 Ethernet controller: Aquantia Corp. AQC111 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
IOMMU group 53:	[8086:2723] 45:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)
IOMMU group 54:	[1022:1485] 46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
[1022:149c] 46:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
[1022:149c] 46:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
IOMMU group 55:	[1022:7901] 47:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 56:	[1022:7901] 48:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 57:	[1022:148a] 49:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 58:	[1022:1485] 4a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 59:	[1022:1482] 60:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 60:	[1022:1482] 60:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 61:	[1022:1482] 60:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 62:	[1022:1482] 60:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 63:	[1022:1482] 60:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 64:	[1022:1482] 60:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 65:	[1022:1484] 60:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 66:	[1022:1482] 60:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 67:	[1022:1484] 60:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 68:	[1022:148a] 61:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 69:	[1022:1485] 62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

 

Edited by Vaggeto
Link to comment
On 10/15/2020 at 10:46 PM, jwoolen said:

LAANUNf.png

 

I was able to pass a secondary GPU without the error 43.

Thanks, so there's some hope for me. Able to share details on how? I mean slot used on Gigbyte extreme mainboard, xml's. If using PCIe ACS override or/and VFIO allow unsafe interrupts.

 

I'm using a vbios, multi-function (basically as all of spaceinvader one advice). But haven't tired the override or unsafe interrupts as it all breaks up fine on its own.

Link to comment
15 hours ago, Cadal said:

Thanks, so there's some hope for me. Able to share details on how? I mean slot used on Gigbyte extreme mainboard, xml's. If using PCIe ACS override or/and VFIO allow unsafe interrupts.

 

I'm using a vbios, multi-function (basically as all of spaceinvader one advice). But haven't tired the override or unsafe interrupts as it all breaks up fine on its own.

I did my testing with the Asus Zenith II Alpha. I used the two x16 slots. No overrides or unsafe interrupts. I dumped the bios files in a bare metal windows installation and removed the header from the ROM files with HxD.

Link to comment
On 10/18/2020 at 4:10 PM, Vaggeto said:

I've just got this board setup with a 3970x and am trying to get some VM passthrough going. My goal is to utilize the 2 CPU linked NVME ports each to be passed to their own VM. Can anyone help me understand which of these IOMMU groups would apply to the NVMEs? Every other example of NVME passthrough from spaceinvader etc seems to specifically mention "NVME" in the IOMMU group description, but this Aorus master doesn't have anything like that from what I can tell with or without ACS override.

 

i believe it is Group 13/14, but if that's the case is there any way to know which one belongs to which drive? There is also a 3rd NVME not through the CPU on this board (not installed at this time).

Those are the correct devices. With the latest unRAID beta you can determine which drive is which using System Devices if you know the serial number, since it gives you the device name.

 

image.thumb.png.71e9a28289dd265921fa015437a40bc0.png

 

image.thumb.png.d5f301922e36d948650cf1c5ef6265c1.png

Link to comment
49 minutes ago, jwoolen said:

Those are the correct devices. With the latest unRAID beta you can determine which drive is which using System Devices if you know the serial number, since it gives you the device name.

 

image.thumb.png.71e9a28289dd265921fa015437a40bc0.png

 

image.thumb.png.d5f301922e36d948650cf1c5ef6265c1.png

Thanks!

 

I did finally get this resolved. Essentially I did find that both of the WD Black Sn750 NVMEs were the following devices:

IOMMU group 13:	[15b7:5006] 01:00.0 Non-Volatile memory controller: Sandisk Corp Device 5006
IOMMU group 14:	[15b7:5006] 02:00.0 Non-Volatile memory controller: Sandisk Corp Device 5006

One is a cache drive and the other is an unassigned drive for dual-booting bare-metal Windows and a VM.

Because they share the same Vendor ID I couldn't stub only 1, but I found a video from SpaceInvaderOne using the VFIO-PCI Config tool which allowed me to (by process of elimination) pass-through a single drive to be pickable in PCI-e devices in the VM setup area.

So I got that working and into the VM!

 

Now I still don't know where the 3rd NVME is within the IOMMU groupings, but I suspect if I put an NVME in that slot, it would reveal itself.

Edited by Vaggeto
Link to comment
On 4/26/2020 at 6:45 AM, bastl said:

In case someone stumbles across this and needs the info. I recently upgraded from a 1950x to a 3960x on a Aorus Extreme TRX40 and struggled the last couple of days getting my VMs back up running. Passing through both GPUs (1080ti, 1050ti) to different VMs, no issues, onboard USB controller also no issue. The problems started with the onboard audio controller. The common way is to find the controller in the devices list, stub/bind them and you should be good to go. In my case the controller is reported as


[1022:1487] 23:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller

BUT

this isn't the real audio controller that can be passed to a VM. I tried the 6.9 beta 1, tried different patched kernels for 6.8.3 with fixes for 3rd gen Ryzens (AMD onboard audio/usb controller flr patch, Vega-Navi-Patch etc.). During all my testings I completely missed, that this board provides 2 separate USB audio devices. There is no need for passing through the Starship/Matisse controller or using a patched kernel. It works straight out of the box with Unraid 6.8.3. No ACS patch or unsafe interrupts needed. Simply pick one of the 2 USB audio devices in the VM template and you're good to go. Windows will recognize it as "Realtek USB Audio" and on Linux (Zorin15.1) it also works straight out of the box. 👍

 

grafik.png.677d7a041b2034e9b966ea244972a8a5.png

 

The first device is for the front panel audio and the second one for the back panel 5.1/7.1 links. And guess what....

you can use them in 2 different VMs and run them simultaneously without issues or any extra tweaking needed. 😍

 

For reference my initial posting, trying to get help in the custom kernel thread:

 

Wow thank you so much for sharing this. I have an TRX40 Aorus Master and have been struggling through this issue.

 

I have been seeing this USB audio devices and questioning what they could possibly be, but like you, kept trying to find a way to get that "HD audio controller" which the VM Setup page allows you to choose in the drop-down menu for audio devices.

 

So are you consider isolating these sound cards and passing them through as "Other PCI Devices" they way you normally would a sound card, or just using the standard "USB Devices" check-box in the VM config to pass it through? I know there can be some benefits of passing through an entire USB controller instead of a USB device for example, but I'm not sure if there are any benefits in a sound card situation.

 

Now passing a USB controller has been extremely difficult, can you describe how you were able to pass through the USB controller? (or are you just passing through individual USB devices). I can pass through individual devices, but not the controller which would be greatly preferred for hotswaping etc.

Link to comment
21 hours ago, Vaggeto said:

So are you consider isolating these sound cards and passing them through as "Other PCI Devices" they way you normally would a sound card, or just using the standard "USB Devices" check-box in the VM config to pass it through?

I only have set the checkbox for one of the USB Audio devices for my VM. The "Starship/Matisse HD Audio Controller" itself I don't use at all but I have a entry for it in the syslinux config to prevent Unraid to use it, just in case. I have no issues with this so far. Everything is working fine.

 

21 hours ago, Vaggeto said:

Now passing a USB controller has been extremely difficult, can you describe how you were able to pass through the USB controller? (or are you just passing through individual USB devices). I can pass through individual devices, but not the controller which would be greatly preferred for hotswaping etc.

For my main VM I have also an entry for one of the USB controllers in my syslinux config. In my case it's a "ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller" on the Aorus Xtreme. A multi USB hub from Anker with card reader and a couple USB ports are sitting on my desk and is connected to ASMedia controller.

 

The following entry is from my xml for that controller. Bus address might be different for you. Keep in mind, it only works if the controller is in it's own IOMMU group, separated from other devices.

 

grafik.png.247cb1b22ae323bba6c6df79c797d61a.png

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x46' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>

grafik.png.a81cb042693989f09c0754055c7f59db.png

Link to comment
2 hours ago, bastl said:

I only have set the checkbox for one of the USB Audio devices for my VM. The "Starship/Matisse HD Audio Controller" itself I don't use at all but I have a entry for it in the syslinux config to prevent Unraid to use it, just in case. I have no issues with this so far. Everything is working fine.

 

For my main VM I have also an entry for one of the USB controllers in my syslinux config. In my case it's a "ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller" on the Aorus Xtreme. A multi USB hub from Anker with card reader and a couple USB ports are sitting on my desk and is connected to ASMedia controller.

 

The following entry is from my xml for that controller. Bus address might be different for you. Keep in mind, it only works if the controller is in it's own IOMMU group, separated from other devices.

 

grafik.png.247cb1b22ae323bba6c6df79c797d61a.png


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x46' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>

grafik.png.a81cb042693989f09c0754055c7f59db.png

Thanks!

That makes sense on the audio. Do you mind sharing the code to prevent unRAID to use the audio controller that you are checking as a USB audio device? (or is it the xen-pciback.hide= code?)

 

For the USB controller, is that an on-board controller or PCIe? ("ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller" )

 

For whatever reason when I try to pass my controllers, the VM won't load. They are in their own IOMMU groups and everything. I posted here about it.

I do have one additional controller on my motherboard, but it is the only controller with USB2 ports so I don't want to pass it through even if I could since I'd prefer for my unRAID USB flash drive to be on a USB2 port.

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.