ASRock Fatal1ty X399 Professional Gaming IOMMU Groupings


Recommended Posts

All,

 

Just built up a new Threadripper server. Having some reliability issues with the M.2 ports and a pair of Intel 660p flash drives, but otherwise, this has been a great board for my needs and the M.2 issues may be related to the fact that I purchased this board used and it was used for overclocking previously.

In any case, I wanted to post the IOMMU groupings for this board as they appear to be quite good for some use cases. There is one group that lumps together quite a few devices, but everything I cared to break out and pass through has worked well. I have not applied any ACS overrides, this is just the default groupings on BIOS 3.30:

IOMMU group 0:	[1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 1:	[1022:1453] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
IOMMU group 2:	[1022:1453] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
IOMMU group 3:	[1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 4:	[1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 5:	[1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
IOMMU group 6:	[1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 7:	[1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 8:	[1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
IOMMU group 9:	[1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 10:	[1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
IOMMU group 11:	[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
		[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 12:	[1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0
		[1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1
		[1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2
		[1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3
		[1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4
		[1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5
		[1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6
		[1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7
IOMMU group 13:	[1022:1460] 00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0
		[1022:1461] 00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1
		[1022:1462] 00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2
		[1022:1463] 00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3
		[1022:1464] 00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4
		[1022:1465] 00:19.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5
		[1022:1466] 00:19.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6
		[1022:1467] 00:19.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7
IOMMU group 14:	[1022:43ba] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset USB 3.1 xHCI Controller (rev 02)
		[1022:43b6] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset SATA Controller (rev 02)
		[1022:43b1] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset PCIe Bridge (rev 02)
		[1022:43b4] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
		[1022:43b4] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
		[1022:43b4] 02:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
		[1022:43b4] 02:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
		[1022:43b4] 02:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
		[1d6a:d107] 03:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
		[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
		[8086:24fb] 05:00.0 Network controller: Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] (rev 10)
		[8086:1539] 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
IOMMU group 15:	[8086:f1a8] 08:00.0 Non-Volatile memory controller: Intel Corporation Device f1a8 (rev 03)
IOMMU group 16:	[1022:1470] 09:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1470 (rev c3)
IOMMU group 17:	[1022:1471] 0a:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1471
IOMMU group 18:	[1002:687f] 0b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] (rev c3)
IOMMU group 19:	[1002:aaf8] 0b:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aaf8
IOMMU group 20:	[1022:145a] 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
IOMMU group 21:	[1022:1456] 0c:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor
IOMMU group 22:	[1022:145c] 0c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller
IOMMU group 23:	[1022:1455] 0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
IOMMU group 24:	[1022:7901] 0d:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 25:	[1022:1457] 0d:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller
IOMMU group 26:	[1022:1452] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 27:	[1022:1453] 40:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
IOMMU group 28:	[1022:1452] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 29:	[1022:1452] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 30:	[1022:1453] 40:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
IOMMU group 31:	[1022:1452] 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 32:	[1022:1452] 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 33:	[1022:1454] 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
IOMMU group 34:	[1022:1452] 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 35:	[1022:1454] 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
IOMMU group 36:	[8086:f1a8] 41:00.0 Non-Volatile memory controller: Intel Corporation Device f1a8 (rev 03)
IOMMU group 37:	[10de:128b] 42:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1)
		[10de:0e0f] 42:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
IOMMU group 38:	[1022:145a] 43:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
IOMMU group 39:	[1022:1456] 43:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor
IOMMU group 40:	[1022:145c] 43:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller
IOMMU group 41:	[1022:1455] 44:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
IOMMU group 42:	[1022:7901] 44:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

As you can see, group 14 has all the networking in it, but otherwise most everything else breaks out pretty cleanly. I am able to reliably pass through the rear USB 3.1 Gen 1 slots to the VM, along with the onboard audio and the Vega card that is installed. Not passing through the audio portion of the Vega card at this point since it seems to mess up the reset for now.

 

Anyway, IOMMU Groupings without any overrides applied is what I wished I had for all of the motherboards I was considering during my shopping phase. Thought I would post this in case anyone else was considering this board.

 

Due to the reliability issues I am experiencing with the M.2 slots, I will likely pick up another board or two to test with. If I do, I will post those groupings as well.

Edited by harshl
Link to comment

@harshl What exactly is your issue with the M.2 slots? I have the same board working for almost 1 year with the same 3.30 BIOS for a couble months now and 2 M.2 drives installed. 500GB Samsung 960 EVO as cache and another 960 PRO as a passthrough for my main Win10 VM. Last one first installed as only M.2 with a bare metal Windows 10 Install, later passed through. I never had any issues with these drives. 

Link to comment
On 12/20/2018 at 5:51 AM, bastl said:

@harshl What exactly is your issue with the M.2 slots? I have the same board working for almost 1 year with the same 3.30 BIOS for a couble months now and 2 M.2 drives installed. 500GB Samsung 960 EVO as cache and another 960 PRO as a passthrough for my main Win10 VM. Last one first installed as only M.2 with a bare metal Windows 10 Install, later passed through. I never had any issues with these drives. 

@bastl I have 2x Intel 660p 2TB drives on the board acting as my cache. Out of the blue, one will just disappear, UNRAID can't write to it, the BIOS doesn't recognize it anymore, it is just gone. If I let the system keep running on the second remaining SSD, eventually, it will disappear from the system as well in the same manner.

If I shut the system down and bring it back up, even after removing power from the system for a time, they are typically still not recognized. If I pull them and move them to a different slot, then power back on, they are recognized and will run fine again for a period of time, after a lot of BTRFS scrubbing anyway. :)

I have had it work for over a week of uptime and I have had it work for less than 24 hours of uptime before the issue occurs. So far haven't been able to identify any particular trigger. It does not seem to be based on load to the SSDs. The last time it happened they were mostly idle and some data was being written to the SATA spinner.

 

The 512GB version of the 660p is on the QVL, but the 2TB is not. Intel claims there is no difference other than capacity, but there could be something that this tech I am working with isn't aware of.

 

Not really sure what to do other than get another board and try it. If I do, I will likely get the Taichi version and hope that the passthrough is as friendly as it has been with this board since they are very similar.

 

Thanks!

-Landon

Edited by harshl
Link to comment

Minor unconfirmed update on this. I have abandoned one particular M.2 slot and I'm running both drives in the lower M.2 slots that are side by side. So far this issue has not repeated, but since I don't know what triggers it, it could be a false sense of security at this point. :)

 

I will let it run for another week or two and see where I end up and report back. Hopefully I can run for that period of time without needing to reboot the host for any other reasons.

Edited by harshl
Link to comment

Small info on my setup. I have one M.2 plugged in the top slot and one in the lower left. The right one is empty. If i remember correctly, if you're using a U.2 device, one of the M.2 slots won't work. If you use something on the U.2 port i guess, one of the M.2 won't never work and not like in your case, randomly fail. One case i can think of besides a faulty drive or faulty slot is if you occupy all your PCIe slots there might be some sharing of the lanes happen.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.