skois

I'll test in the week how it works. hopefully will go well.

I'm not chaning the host pci address, im changing the guest pci address! NIC is HP NC364T (the same SI is using) IOMMU groups IOMMU group 0: [1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 1: [1022:1453] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 2: [1022:1453] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 3: [1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 4: [1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 5: [1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 6: [1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 7: [1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 8: [1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 9: [1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 10: [1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 11: [1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 12: [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59) [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) IOMMU group 13: [1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 IOMMU group 14: [1022:1460] 00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1461] 00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1462] 00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1463] 00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1464] 00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1465] 00:19.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1466] 00:19.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1467] 00:19.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 IOMMU group 15: [1022:43ba] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset USB 3.1 xHCI Controller (rev 02) [1022:43b6] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset SATA Controller (rev 02) [1022:43b1] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset PCIe Bridge (rev 02) [1022:43b4] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [1022:43b4] 02:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [1022:43b4] 02:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [1022:43b4] 02:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [1022:43b4] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) [8086:24fd] 05:00.0 Network controller: Intel Corporation Wireless 8265 / 8275 (rev 78) [8086:1539] 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) IOMMU group 16: [144d:a808] 08:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 IOMMU group 17: [111d:8018] 09:00.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e) IOMMU group 18: [111d:8018] 0a:02.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e) [8086:10bc] 0b:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06) [8086:10bc] 0b:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06) IOMMU group 19: [111d:8018] 0a:04.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e) [8086:10bc] 0c:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06) [8086:10bc] 0c:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06) IOMMU group 20: [10de:0640] 0d:00.0 VGA compatible controller: NVIDIA Corporation G96C [GeForce 9500 GT] (rev a1) IOMMU group 21: [1022:145a] 0e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function IOMMU group 22: [1022:1456] 0e:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor IOMMU group 23: [1022:145f] 0e:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller IOMMU group 24: [1022:1455] 0f:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function IOMMU group 25: [1022:7901] 0f:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 26: [1022:1457] 0f:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller IOMMU group 27: [1022:1452] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 28: [1022:1452] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 29: [1022:1452] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 30: [1022:1453] 40:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 31: [1022:1452] 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 32: [1022:1452] 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 33: [1022:1454] 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 34: [1022:1452] 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 35: [1022:1454] 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 36: [1002:1478] 41:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Device 1478 (rev c4) IOMMU group 37: [1002:1479] 42:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Device 1479 IOMMU group 38: [1002:731f] 43:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5700 / 5700 XT] (rev c4) IOMMU group 39: [1002:ab38] 43:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio IOMMU group 40: [1022:145a] 44:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function IOMMU group 41: [1022:1456] 44:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor IOMMU group 42: [1022:145f] 44:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller IOMMU group 43: [1022:1455] 45:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function IOMMU group 44: [1022:7901] 45:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

i would say yes. but dont take my word for it. i've been using it only for 24 hours!

Hello. i'm trying to passthrou my 4 port NIC to pfsense VM. I have followed SI's video I have stubbed the NIC and it appears on the VM editor. I'm selecting all 4 ports and boot the VM. pfSense only sees 1 port. (em0). The only way i managed to make pfSense to see all 4 ports was to manually change XML to this: By changing guest nic's addresses to: 01:00.0 01:00.1 01:00.2 01:00.3 from 01:00.0 03:00.0 04:00.0 05:00.0 Is this correct way to do it? Why with the default XML it only lists 1 port? Edited XML Part: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0c' slot='0x00' function='0x0'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0c' slot='0x00' function='0x1'/> </source> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/> </hostdev> auto generated XML. <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='15'> <name>pfSense</name> <uuid>333adf90-ea87-2912-7ae3-1e2915df94c0</uuid> <metadata> <vmtemplate xmlns="unraid" name="FreeBSD" icon="pfsense.png" os="freebsd"/> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='10'/> <vcpupin vcpu='1' cpuset='22'/> <vcpupin vcpu='2' cpuset='11'/> <vcpupin vcpu='3' cpuset='23'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/333adf90-ea87-2912-7ae3-1e2915df94c0_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='4' threads='1'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/Linux/pfSense-CE-2.4.4-RELEASE-p3-amd64.iso'/> <backingStore/> <target dev='hda' bus='sata'/> <readonly/> <boot order='2'/> <alias name='sata0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='writeback'/> <source file='/mnt/user/domains/pfSense/vdisk1.img'/> <backingStore/> <target dev='hdc' bus='sata'/> <boot order='1'/> <alias name='sata0-0-2'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x14'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x15'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x16'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0x17'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0x18'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x19'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller> <controller type='pci' index='11' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='11' port='0x1a'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/> </controller> <controller type='pci' index='12' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='12' port='0x1b'/> <alias name='pci.12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <serial type='pty'> <source path='/dev/pts/2'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-15-pfSense/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <graphics type='vnc' port='5900' autoport='yes' websocket='5700' listen='0.0.0.0' keymap='en-us'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0c' slot='0x00' function='0x0'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0c' slot='0x00' function='0x1'/> </source> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain>

F12 seems solid, didnt have any problems. Only one time Windows VM didnt want to wake up after couple of hours on sleep. But host was fine. force stopped and started again fine. With only ACS enabled from bios, i didnt need acs override from VM settings. everything is split in its own iommu. The only thing i have problem is with mb audio. i can pass it to vm, its working, its in its own iommu group. But vm logs says: Cannot reset device, depends on group xx which is not owned. Had this on previous bios version also tho

Just pressed boot after setting bios settings, Will leave array running and VM on all night. will report tomorrow. Btw the unstabilities i had, might be a docker problem a specific one.., don't know why. But after upgrading my machine. I tried running Handbrake, encoded a couple of clips but on 3rd the docker just restarted. Tried again. the same. i though it was the file. tried another. all good, tried the "bad" file again, encoded without problems. between those failed encodes i had server lockups or not responding on some things. after 3 days stopped handbrake from opening and is very stable for now on F12i. will let you know tomorrow for f12.

There it is! F12 is up (no letters after it, guessing its a stable.., will try tonight!) https://www.gigabyte.com/Motherboard/X399-DESIGNARE-EX-rev-10/support#support-dl-bios

Gigabyte just pulled down F12i BIOS from website. Either they are going to upload a new F12 revision. Or F12 have something bad in it! EDIT! F12 (without any letters after it is up, will try it after work)

Wait, i just checked again from my laptop and my mobile data, and Gigabyte might have pulled F12 down? I was able to see F12 but now i can see F11 as latest. What is going here?

Mine is also v1, if you dont mind please tell me if you using any kernel parameters between append and initrd! I'll be waiting Thanks!

Thanks for replying! I was thinking to go 6.8.0 stable. Maybe 4.19.88 kernel is more stable? But not sure if my hardware is even supported on this kernel Do you mind sharing some info about bios? Like what version you running and what is your bios settings, and what kernel parameters you have? Maybe that will give me a good headstart to start digging more. I though i could give as much info as i can to avoid things like, "hey send this, run this command, try this" if i have already done it But i agree this info was maybe too much lol!

First of all. I want to say a big THANKS to anyone that even bothers reading this. This post is to get help, not solving any problems. I'll be happy to post all the solutions on my problems at the end so next people looking for same problems can get the help also. I'm including a lot of info. Maybe too much? I don't except from only one person to solve all my problems. I was running Unraid with an FX8350 and Gigabyte 990FXA-UD3. All of this was working fine, except pfsense which i couldn't due to low cpu count and resources (never tried to set it up) The reason posting all of my problems here (in motherboard/cpu section) is because it started after changing the MB/CPU :). If you think is should break the problems and post to different subcategories. I'll do it. It was just felt silly to post all info so many times. More problems might appear down the road. I just build the server 2 days ago and still discovering! The Build. MB: Gigabyte X399 Ex Designare CPU: Threadripper 2920x RAM: Corsair Vengeange 4x8G 3200 NVME: Samsung Evo Plus 970 500G(UD* for VM) GPU: Sapphire Rx 5700 8G Pulse SSDs: Samsung 850 Evo 250G(Cache), Samsung 860 Evo 250G(UD* for VM) HDDs: 2xWD Red 2TB(Array), 1xWD Blue 2TB (Parity) Extra: 1x4Port Gigabit Nic *Unassigned Devices What i want to achieve. -Unraid as NAS -Dockers (Plex, Sonarr, Radarr, Jacket, Bazarr, Deluge, Handbrake, Nextcloud, Unifi Controller) -VMs (1 x Windows*for Games*,1 x Linux*as daily driver*, 1 x pfSense*as Router/firewall*) Note: Windows VM - Linux VM and Handbrake will be pinned on the same CPU's because they will never run together! Only one at a time! Unraid Version: 6.8.0 rc7 Booting in UEFI Currently running bios: F12 Bios Setup: M.I.T/Advanced CPU Core Settings/Core Performance Boost: Auto M.I.T/Advanced CPU Core Settings/AMD Cool&Quiet function: Enabled M.I.T/Advanced CPU Core Settings/SVM Mode: Enabled M.I.T/Advanced CPU Core Settings/Global C-state Control: Enabled Disabled(After comming home from work VM was paused and server had locked up,disabled for testing) Enabled again, disabling it made it worse. M.I.T/Advanced CPU Core Settings/Power Supply Idle Control: Typical Current Idle M.I.T/Advanced CPU Core Settings/Opcache Control: Auto M.I.T/Advanced CPU Core Settings/Downcore Control: Auto M.I.T/Advanced CPU Core Settings/SMT Mode: Auto M.I.T/Extreme Memory Profile(X.M.P): Profile1 M.I.T/Precision Boost Overdrive: Auto Peripherals/AMD CPU fTPM: Disabled Peripherals/XHCI Hand-off: Enabled Peripherals/EHCI Hand-off: Enabled Peripherals/SR-IOV Support: Enabled Peripherals/Trusted Computing/Security Device Support: Disable Peripherals/AMD CBS/Enable IBS: Disabled Peripherals/AMD CBS/Custom Pstates - Throttling: UNTOUCHED* Peripherals/AMD CBS/Prefetcher Settings/L1 Stream HW Prefetcher: Auto Peripherals/AMD CBS/Prefetcher Settings/L2 Stream HW Prefetcher: Auto Peripherals/AMD CBS/Memory Interleaving: Auto Peripherals/AMD CBS/DRAM Controller Configuration/Gear Down Mode: Auto Peripherals/AMD CBS/DRAM Controller Configuration/DRAM Power Options/Power Down Enable: Auto Peripherals/AMD CBS/Precision Boost Overdrive Configuration: UNTOUCHED* Peripherals/AMD CBS/ACS Enable: Enable Peripherals/AMD CBS/PCIe ARI Support: Enable Peripherals/AMD CBS/cTDP Conrol: Auto Chipset/IOMMU: Enabled Chipset/SB Clock Spread Spectrum: Enabled Chipset/SATA Mode: AHCI Chipset/Chipset SATA Port Enable: Enabled Power/S3 Save Mode: Disabled Power/High Precision Event Timer: Enabled IOMMU with ACS Override Disabled: IOMMU group 0: [1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 1: [1022:1453] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 2: [1022:1453] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 3: [1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 4: [1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 5: [1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 6: [1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 7: [1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 8: [1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 9: [1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 10: [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59) [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) IOMMU group 11: [1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 IOMMU group 12: [1022:1460] 00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1461] 00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1462] 00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1463] 00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1464] 00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1465] 00:19.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1466] 00:19.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1467] 00:19.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 IOMMU group 13: [1022:43ba] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset USB 3.1 xHCI Controller (rev 02) [1022:43b6] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset SATA Controller (rev 02) [1022:43b1] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset PCIe Bridge (rev 02) [1022:43b4] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [1022:43b4] 02:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [1022:43b4] 02:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [1022:43b4] 02:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [1022:43b4] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) [8086:24fd] 05:00.0 Network controller: Intel Corporation Wireless 8265 / 8275 (rev 78) [8086:1539] 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) [111d:8018] 07:00.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e) [111d:8018] 08:02.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e) [111d:8018] 08:04.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e) [8086:10bc] 09:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06) [8086:10bc] 09:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06) [8086:10bc] 0a:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06) [8086:10bc] 0a:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06) IOMMU group 14: [144d:a808] 0b:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 IOMMU group 15: [1022:145a] 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function IOMMU group 16: [1022:1456] 0c:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor IOMMU group 17: [1022:145f] 0c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller IOMMU group 18: [1022:1455] 0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function IOMMU group 19: [1022:7901] 0d:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 20: [1022:1457] 0d:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller IOMMU group 21: [1022:1452] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 22: [1022:1452] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 23: [1022:1452] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 24: [1022:1453] 40:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 25: [1022:1452] 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 26: [1022:1452] 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 27: [1022:1454] 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 28: [1022:1452] 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 29: [1022:1454] 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 30: [1002:1478] 41:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Device 1478 (rev c4) IOMMU group 31: [1002:1479] 42:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Device 1479 IOMMU group 32: [1002:731f] 43:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5700 / 5700 XT] (rev c4) IOMMU group 33: [1002:ab38] 43:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio IOMMU group 34: [1022:145a] 44:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function IOMMU group 35: [1022:1456] 44:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor IOMMU group 36: [1022:145f] 44:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller IOMMU group 37: [1022:1455] 45:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function IOMMU group 38: [1022:7901] 45:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU with ACS Override: Both New IOMMU after moving NIC to 4th slot with ACS Override: Both VM Settings/PCIe ACS override: Disabled VMSettings/VFIO allow unsafe interrupts: No Kernel Parameters: rcu_nocbs=0-23 vfio-pci.ids=8086:10bc (testing without rcu_nocbs after updating to F12 Bios) lstopo (stays tha same with acs disabled and both) image attached. After moving NIC to 4th slot the new topo is Problems i have so far 1) Parity Disk Log shows The Bold and Underlined lines are in Orange (Warnings) Also note that the other 2 Disks does not show: Prometheus root: /usr/sbin/wsdd 2) Navi Reset Bug (Would like a help rebuilding latest kernel with Gnif's V2 Patch https://forum.level1techs.com/t/navi-reset-kernel-patch/147547/47 But this can be done after fixing all the other problems 3) dmesg shows in red 70-persistent-net.rules contains Tried removing the "--" but after reboot the -- comes back. Run dmesg -c and rebooted. Attached dmesg with the error. 4) pfSense VM only see one of the 4 ports of the passed PCI NIC pfSense: pciconf -l gives: (only pasted the line that refers to the NIC) After changing the NIC slot it still does the same. Tried it on a bare metal machine and the NIC works fine. It recognized all ports. So its something with the Virtualisation causing problems. XML I made changes on XML from this to this i saw 2 ports: the 01:00:0 and 01:00:1 Then changed each port's address on XML to 01:00:0, 01:00:1, 01:00:2, 01:00:3 and i could see all ports in the pfSense VM. But i'm not sure if this correct and if i'm supposed to do it like that. 5) Windows and Linux VM having problems resetting onboard Audio, Weird thing is that when booted the sounds works. XML 6) Having this crash at random times EDIT 22/12/19 After disabled Handbrake container, unraid seems stable, no lockouts, no weird things, (except know bugs like navi reset) For some reason even if handbrake wasn't doing anything it was doing something bad. If it was encoding. the server would become unusable soon after the 1st failed encode. Nothing was showing on dmesg or system logs. Had a couple of lockups but at the same time i was doing stuff. like adding second cache drive where balance got stuck and tried to stop balance and reboot. At first day when windows vm goes to sleep cant wake (because of the navi bug), but it was also locking up host. Now it wont lock host. I don't know if it was just coincidence the 1st day. because i didnt do any changes on settings etc. The only thing i've done is to update to 6.8.0 stable and revert back to 6.8.0. rc7. Will report back in a couple of days to see how its going. dmesg

Is it possible to write down all your Bios settings and the kernel parameters, well the settings that affects the working state of Unraid - KVM. Not boot settings etc. Yesterday i did my build with Ex Designare and 2920x. Already had it die on me. One time it just crashed and giving me 500 internal error for GUI, could ping and ssh, but everything was unresponsive. even "htop" was blank. And today GUI was loading half pages, couldnt ssh and more. Trouble passing through MB Audio.. Before making a huge post with details. Your settings will help make a good base. To test and post my problems orginized. Thanks! And hopefully you will have sometime time to helm me, along with @SpaceInvaderOne to help me stabilize my new build! (As i saw on one of SI's videos he also have a 2950 *plus knowledge* :D) Just made a post about my build and loads of info/problems.

What problems do you have with F12i? Also i can't see on the website the F12e you mention. (I'm not someone that can help you solve anything) I will be upgrading my server today to a x399 Ex Designare + tr2920. Dont know what BIOS does it have preinstalled yet.

Hey! How is the patch acting? Does it work well? Thanks!

Hey, I believe that you already seen it.. But I thought I try. Gnif posted a v2 here before 20 days https://forum.level1techs.com/t/navi-reset-kernel-patch/147547/46 Fingers crossed it's good enough (not break other things) to make it on 6.9.0 rc1 *I'm using rx5700