ASR

Hi, to start I'm french, so sorry for my rusty english ! I did an unraid server for my work (environnemental restore) with : Asus X570 WS PRO ACE (for ECC and 3 PCIE 16X...) 2600 AMD (waiting 3900X stock) 32 GB ECC Ram Crucial 3 X AMD RX550 for VM 3 HDD RED 4To (1 parity / 2 DATA) 1 SSD 500 Go Cache 1 SSD NVME 500 to Passthrought specific VM My plan is to use this server with - 1 VM for a specific worker (W10 pro, 8 threads, 8G ram, 1 RX550 passtrought, 1 NVME idem...) ==> this is working without issue now in Q35-4.1 only, HyperV yes, OVMF, nvme is passthrough... I have error 127 but it's not a problem ; - NAS UNRAID to store some sharing data with my team and I'll do partition save/data save of their computer in autonomatic mode later ==> is working easily ; - At least 1 VM (actually passthrought but later in headless) on linux to working in SQL data base (fish data bases...) => and this is my issue, I can use Debian or another in VNC mode, but I can't passthrough another card. Unraid is starting on the first slot, I can't chose in the bios :(. The bottom card is use for the first Win10 VM, but It's impossible to use the middle card ! If I start this VM, it'll go in suspend mode (yellow icon) and I got this log on the VM : 2019-10-29T07:42:44.673194Z qemu-system-x86_64: vfio_err_notifier_handler(0000:0b:00.0) Unrecoverable error detected. Please collect any data possible and then kill the guest 2019-10-29T07:42:45.366243Z qemu-system-x86_64: vfio_err_notifier_handler(0000:0b:00.0) Unrecoverable error detected. Please collect any data possible and then kill the guest This log in general : Oct 29 08:42:44 Tower kernel: pcieport 0000:00:03.2: AER: Uncorrected (Non-Fatal) error received: 0000:00:00.0 Oct 29 08:42:44 Tower kernel: pcieport 0000:00:03.2: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Receiver ID) Oct 29 08:42:44 Tower kernel: pcieport 0000:00:03.2: AER: device [1022:1453] error status/mask=00200000/04400000 Oct 29 08:42:44 Tower kernel: pcieport 0000:00:03.2: AER: [21] ACSViol (First) Oct 29 08:42:44 Tower kernel: pcieport 0000:00:03.2: AER: Device recovery failed Oct 29 08:42:44 Tower kernel: AMD-Vi: Completion-Wait loop timed out Oct 29 08:42:44 Tower kernel: AMD-Vi: Completion-Wait loop timed out Oct 29 08:42:45 Tower kernel: AMD-Vi: Completion-Wait loop timed out Oct 29 08:42:45 Tower kernel: iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0b:00.0 address=0x83b073e80] Oct 29 08:42:45 Tower kernel: pcieport 0000:00:03.2: AER: Uncorrected (Non-Fatal) error received: 0000:00:00.0 Oct 29 08:42:45 Tower kernel: pcieport 0000:00:03.2: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Receiver ID) Oct 29 08:42:45 Tower kernel: pcieport 0000:00:03.2: AER: device [1022:1453] error status/mask=00200000/04400000 Oct 29 08:42:45 Tower kernel: pcieport 0000:00:03.2: AER: [21] ACSViol (First) Oct 29 08:42:45 Tower kernel: pcieport 0000:00:03.2: AER: Device recovery failed My system information : Model: N/A M/B: ASUSTeK COMPUTER INC. Pro WS X570-ACE Version Rev X.0x - s/n: 190552238900195 LAST BIOS BIOS: American Megatrends Inc. Version 1001. Dated: 09/09/2019 CPU: AMD Ryzen 5 2600 Six-Core @ 3400 MHz HVM: Enabled IOMMU: Enabled Cache: 576 KiB, 3072 KiB, 16384 KiB Memory: 32 GiB DDR4 Multi-bit ECC (max. installable capacity 128 GiB) Network: eth0: 1000 Mbps, full duplex, mtu 1500 Kernel: Linux 5.3.7-Unraid x86_64 OpenSSL: 1.1.1d I have VFIO allow unsafe interrupts : YES, no need PCIe ACS override because my IOMMU are this and it's ok for me : PCI Devices and IOMMU Groups IOMMU group 0: [1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 1: [1022:1453] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 2: [1022:1453] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 3: [1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 4: [1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 5: [1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 6: [1022:1453] 00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 7: [1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 8: [1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 9: [1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 10: [1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 11: [1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 12: [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59) [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) IOMMU group 13: [1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 IOMMU group 14: [8086:f1a8] 01:00.0 Non-Volatile memory controller: Intel Corporation SSD 660P Series (rev 03) IOMMU group 15: [1022:57ad] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57ad IOMMU group 16: [1022:57a3] 03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3 IOMMU group 17: [1022:57a3] 03:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3 IOMMU group 18: [1022:57a3] 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3 IOMMU group 19: [1022:57a4] 03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4 [1022:1485] 07:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:149c] 07:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 07:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller IOMMU group 20: [1022:57a4] 03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4 [1022:7901] 08:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 21: [1022:57a4] 03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4 [1022:7901] 09:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 22: [1002:67ff] 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] (rev ff) [1002:aae0] 04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] IOMMU group 23: [8086:1539] 05:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) IOMMU group 24: [10ec:816e] 06:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a) [10ec:8168] 06:00.1 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 1a) [10ec:816a] 06:00.2 Serial controller: Realtek Semiconductor Co., Ltd. Device 816a (rev 1a) [10ec:816c] 06:00.7 IPMI Interface: Realtek Semiconductor Co., Ltd. Device 816c (rev 1a) IOMMU group 25: [1002:67ff] 0a:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] (rev ff) [1002:aae0] 0a:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] IOMMU group 26: [1002:67ff] 0b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] (rev ff) [1002:aae0] 0b:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] (rev ff) IOMMU group 27: [1022:145a] 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function IOMMU group 28: [1022:1456] 0c:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor IOMMU group 29: [1022:145f] 0c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller IOMMU group 30: [1022:1455] 0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function IOMMU group 31: [1022:7901] 0d:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 32: [1022:1457] 0d:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller Thank you if you can help me ! really. 10 days remaning before I had to buy a licence. Have a nice day !