January 8, 20233 yr Hi guys, I tried guides here and tried to search for a solution, but all are not working. When I try to bind only the Mellanox card it auto selects the Realtek NIC also. I tried to edit the vfio-pci.cfg manually and delete the Realtek NIC but when unraid boots, no NIC available only loopback using 'ifconfig'. I also tried ACS settings but still the same cant passthrough the Mellanox card. Do i miss something or does it not work mit my setup? MOBO: MAG B460M MORTAR CPU: i5-10400 Thanks! tower-diagnostics-20230108-1836.zip
January 8, 20233 yr Community Expert 43 minutes ago, emersonicus said: Hi guys, I tried guides here and tried to search for a solution, but all are not working. When I try to bind only the Mellanox card it auto selects the Realtek NIC also. I tried to edit the vfio-pci.cfg manually and delete the Realtek NIC but when unraid boots, no NIC available only loopback using 'ifconfig'. I also tried ACS settings but still the same cant passthrough the Mellanox card. Do i miss something or does it not work mit my setup? MOBO: MAG B460M MORTAR CPU: i5-10400 Thanks! tower-diagnostics-20230108-1836.zip 128.5 kB · 0 downloads I think the issue you are going to have is that only one PCI slot is connected to CPU other two are via the PCH which also include the NIC and even with acs override is cannot break up the connections enough. Is the Mellanox is PCI_E3? So likely to be a motherboard hardware constraint. Edited January 8, 20233 yr by SimonF
January 8, 20233 yr Author Hi, Indeed, it was on PCI_E3. I switched it now to PCI_E1 and now im getting error when trying to start the VM with the card. Quote internal error: qemu unexpectedly closed the monitor: qxl_send_events: spice-server bug: guest stopped, ignoring 2023-01-08T14:39:06.860794Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:01:00.0","id":"hostdev0","bus":"pci.4","addr":"0x0"}: vfio 0000:01:00.0: Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy Thanks! Edited January 8, 20233 yr by emersonicus
January 8, 20233 yr Community Expert 1 hour ago, emersonicus said: Hi, Indeed, it was on PCI_E3. I switched it now to PCI_E1 and now im getting error when trying to start the VM with the card. Thanks! Is that card bound to VFIO and what is in 1:00.0
January 8, 20233 yr Author Yes, according to the logs it’s bound successfully and the 01:00.0 is the Mellanox dual nic card which I want to passthrough.
January 8, 20233 yr Community Expert 2 minutes ago, emersonicus said: Yes, according to the logs it’s bound successfully and the 01:00.0 is the Mellanox dual nic card which I want to passthrough. and to confirm you have rebooted since binding to vfio as likely unraid will have loaded driver for the card. What does lspci -vs 1:00.0 show for driver. root@computenode:/mnt/user/domains/UbuntuServer/snap# lspci -vs 9:00.0 09:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3] Subsystem: Mellanox Technologies MT27500 Family [ConnectX-3] Flags: bus master, fast devsel, latency 0, IRQ 17, IOMMU group 24 Memory at 51a00000 (64-bit, non-prefetchable) Memory at 6110800000 (64-bit, prefetchable) Expansion ROM at 51900000 [disabled] Capabilities: [40] Power Management version 3 Capabilities: [48] Vital Product Data Capabilities: [9c] MSI-X: Enable+ Count=128 Masked- Capabilities: [60] Express Endpoint, MSI 00 Capabilities: [c0] Vendor Specific Information: Len=18 <?> Capabilities: [100] Alternative Routing-ID Interpretation (ARI) Capabilities: [148] Device Serial Number f4-52-14-03-00-0e-b5-c0 Capabilities: [154] Advanced Error Reporting Capabilities: [18c] Secondary PCI Express Kernel driver in use: mlx4_core Kernel modules: mlx4_core
January 8, 20233 yr Author Output: root@NAS:~# lspci -vs 1:00.0 01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] Subsystem: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] Flags: fast devsel, IRQ 16, IOMMU group 10 Memory at 81200000 (64-bit, non-prefetchable) [disabled] Memory at 6016800000 (64-bit, prefetchable) [disabled] Expansion ROM at 81100000 [disabled] Capabilities: [40] Power Management version 3 Capabilities: [48] Vital Product Data Capabilities: [9c] MSI-X: Enable- Count=128 Masked- Capabilities: [60] Express Endpoint, MSI 00 Capabilities: [c0] Vendor Specific Information: Len=18 <?> Capabilities: [100] Alternative Routing-ID Interpretation (ARI) Capabilities: [148] Device Serial Number e4-1d-53-03-00-37-b6-bf Capabilities: [108] Single Root I/O Virtualization (SR-IOV) Capabilities: [154] Advanced Error Reporting Capabilities: [18c] Secondary PCI Express Kernel driver in use: vfio-pci Kernel modules: mlx4_core
January 8, 20233 yr Community Expert 5 minutes ago, emersonicus said: Output: root@NAS:~# lspci -vs 1:00.0 01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] Subsystem: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] Flags: fast devsel, IRQ 16, IOMMU group 10 Memory at 81200000 (64-bit, non-prefetchable) [disabled] Memory at 6016800000 (64-bit, prefetchable) [disabled] Expansion ROM at 81100000 [disabled] Capabilities: [40] Power Management version 3 Capabilities: [48] Vital Product Data Capabilities: [9c] MSI-X: Enable- Count=128 Masked- Capabilities: [60] Express Endpoint, MSI 00 Capabilities: [c0] Vendor Specific Information: Len=18 <?> Capabilities: [100] Alternative Routing-ID Interpretation (ARI) Capabilities: [148] Device Serial Number e4-1d-53-03-00-37-b6-bf Capabilities: [108] Single Root I/O Virtualization (SR-IOV) Capabilities: [154] Advanced Error Reporting Capabilities: [18c] Secondary PCI Express Kernel driver in use: vfio-pci Kernel modules: mlx4_core Looks ok.
January 9, 20233 yr Author I tried to update the FW and still the same problem prevails. Can i just use the NICs without passthrough? will there be any major differences?
February 5, 20233 yr Author Solution Apparently is has something to do with i801_smbus si there a possible to fix this by deactivating i801_smbus? root@NAS:/mnt/user/Downloads# dmesg | grep 'Flags mismatch' [ 153.614390] genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:01:00.0)) vs. 00000080 (i801_smbus) EDIT: I added this to Syslinux cfg intel_iommu=on modprobe.blacklist=i2c_i801,i2c_smbus nows its working. Edited February 5, 20233 yr by emersonicus
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.