s1oz Posted March 18, 2021 Share Posted March 18, 2021 I'm very sorry, my English is not good has been normally turned on SRIOV through the command, and the VF network card can also be seen in the system equipment, but it is not displayed in the virtual machine Unraid OS kernel /bzimage append intel_iommu=pt vifo-pci.ids=8086:10c9 initrd=/bzroot //[/boot/config/go] #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & sudo echo 7 > /sys/class/net/eth1/device/sriov_numvfs xml addition can be used normally <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x03' slot='0x10' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> Quote Link to comment
ljm42 Posted March 18, 2021 Share Posted March 18, 2021 I do not have any experience with SRIOV, but it is discussed over Here: https://forums.unraid.net/topic/103323-how-to-using-sr-iov-in-unraid-with-1gb10gb40gb-network-interface-cards-nics/ The problem is that Unraid binds devices to vfio-pci very early in the boot process, so by the time you enable SRIOV it is already done. Try adjusting your go script like this: #!/bin/bash # enable SRIOV and process vfio-pci binding again echo 7 > /sys/class/net/eth1/device/sriov_numvfs /usr/local/sbin/vfio-pci >>/var/log/vfio-pci # Start the Management Utility /usr/local/sbin/emhttp & With this change, Unraid will re-process the vfio-pci bindings after running your SRIOV command. I recommend doing both of these BEFORE starting emhttp so everything will be in place before the array starts. Also, I'd recommend removing the `vfio-pci.ids-808:10c9` param from your syslinux. It is better to manage that from System Devices rather than in syslinux. Please post your diagnostics (Tools -> Diagnostics) after doing this Note: it is probably a good idea to stop the array and disable array autostart (Settings -> Disk Settings -> Enable auto start = no) before making these changes, that way if the system has problems you can power it down and edit the files again without worrying about an unclean shutdown. Quote Link to comment
s1oz Posted March 19, 2021 Author Share Posted March 19, 2021 (edited) @ljm42thank you the network.cfg file is like this, sriov cannot be used normally /boot/config/network.cfg # Generated settings: IFNAME[0]="br0" DHCP_KEEPRESOLV="no" DHCP6_KEEPRESOLV="no" BRNAME[0]="br0" BRNICS[0]="eth0" BRSTP[0]="no" BRFD[0]="0" DESCRIPTION[0]="unraid" PROTOCOL[0]="ipv4+ipv6" USE_DHCP[0]="yes" USE_DHCP6[0]="yes" METRIC6[0]="0" PRIVACY6[0]="0" IFNAME[1]="eth1" PROTOCOL[1]="ipv4+ipv6" USE_DHCP[1]="yes" USE_DHCP6[1]="yes" PRIVACY6[1]="0" SYSNICS="2" Only show one network card If the network.cfg file is like this, # Generated settings: IFNAME[0]="br0" DHCP_KEEPRESOLV="no" DHCP6_KEEPRESOLV="no" BRNAME[0]="br0" BRNICS[0]="eth0" BRSTP[0]="no" BRFD[0]="0" DESCRIPTION[0]="unraid" PROTOCOL[0]="ipv4+ipv6" USE_DHCP[0]="yes" USE_DHCP6[0]="yes" METRIC6[0]="0" PRIVACY6[0]="0" SYSNICS="1" After adjusting the go script, the network card is displayed, but other problems have occurred #!/bin/bash # enable SRIOV and process vfio-pci binding again echo 7 > /sys/class/net/eth1/device/sriov_numvfs /usr/local/sbin/vfio-pci >>/var/log/vfio-pci # Start the Management Utility /usr/local/sbin/emhttp & Network Settings-eth1 prot down The default port is closed after booting, you need to click to open it, the virtual network card can be used normally Can I add a user script to open the port? After manually clicking, the virtual machine can't get the network normally. You can only get the network normally by restarting the virtual machine or disabling the network card, and then opening it Adding user script can solve this problem. Is there a better way? Or can it be added to the go script? #!/bin/bash ifconfig eth1 up tower-diagnostics-20210319-1023.zip Edited March 19, 2021 by s1oz Quote Link to comment
ljm42 Posted March 19, 2021 Share Posted March 19, 2021 Just to be clear, I have never worked on something like this before But if this solves the problem: ifconfig eth1 up then I would add it to the go script after vfio-pci but before emhttp. i.e. you want to call this after the network cards are setup but before plugins are installed, the array starts up, etc. And all that is started by emhttp. User scripts are for things that need to be run after the array has started. It looks like you have added some more to your go script: ip link set eth1 vf 0 mac 00:25:8b:03:10:01 ip link set eth1 vf 1 mac 00:25:8b:03:10:03 ... Is that related to the NICs that you bound to vfio-pci? If so I'd think these lines should come before the call to vfio-pci. But I don't really know. Quote Link to comment
s1oz Posted March 19, 2021 Author Share Posted March 19, 2021 (edited) 填写完毕后，所有问题均已解决。感谢您的耐心配合和指导 ###VFIO-PCI Log ErrorWarningSystem Loading config from /boot/config/vfio-pci.cfg BIND=0000:01:00.0|10de:0fc8 0000:01:00.1|10de:0e1b 0000:03:00.0|8086:10c9 0000:05:00.0|10de:2184 0000:05:00.1|10de:1aeb 0000:05:00.2|10de:1aec 0000:05:00.3|10de:1aed 0000:03:10.1|8086:10ca 0000:03:10.3|8086:10ca 0000:03:10.5|8086:10ca 0000:03:10.7|8086:10ca 0000:03:11.1|8086:10ca 0000:03:11.3|8086:10ca 0000:03:11.5|8086:10ca --- Processing 0000:01:00.0 10de:0fc8 Vendor:Device 10de:0fc8 found at 0000:01:00.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/0000:01:00.0 /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/0000:01:00.1 Binding... Successfully bound the device 10de:0fc8 at 0000:01:00.0 to vfio-pci --- Processing 0000:01:00.1 10de:0e1b Vendor:Device 10de:0e1b found at 0000:01:00.1 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:01:00.1/iommu_group/devices/0000:01:00.0 /sys/bus/pci/devices/0000:01:00.1/iommu_group/devices/0000:01:00.1 Binding... 0000:01:00.0 already bound to vfio-pci 0000:01:00.1 already bound to vfio-pci Successfully bound the device 10de:0e1b at 0000:01:00.1 to vfio-pci --- Processing 0000:03:00.0 8086:10c9 Vendor:Device 8086:10c9 found at 0000:03:00.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:03:00.0/iommu_group/devices/0000:03:00.0 Binding... Successfully bound the device 8086:10c9 at 0000:03:00.0 to vfio-pci --- Processing 0000:05:00.0 10de:2184 Vendor:Device 10de:2184 found at 0000:05:00.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:05:00.0/iommu_group/devices/0000:05:00.0 /sys/bus/pci/devices/0000:05:00.0/iommu_group/devices/0000:05:00.1 /sys/bus/pci/devices/0000:05:00.0/iommu_group/devices/0000:05:00.2 /sys/bus/pci/devices/0000:05:00.0/iommu_group/devices/0000:05:00.3 Binding... Unbound 0000:05:00.2 from xhci_hcd Successfully bound the device 10de:2184 at 0000:05:00.0 to vfio-pci --- Processing 0000:05:00.1 10de:1aeb Vendor:Device 10de:1aeb found at 0000:05:00.1 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:05:00.1/iommu_group/devices/0000:05:00.0 /sys/bus/pci/devices/0000:05:00.1/iommu_group/devices/0000:05:00.1 /sys/bus/pci/devices/0000:05:00.1/iommu_group/devices/0000:05:00.2 /sys/bus/pci/devices/0000:05:00.1/iommu_group/devices/0000:05:00.3 Binding... 0000:05:00.0 already bound to vfio-pci 0000:05:00.1 already bound to vfio-pci 0000:05:00.2 already bound to vfio-pci 0000:05:00.3 already bound to vfio-pci Successfully bound the device 10de:1aeb at 0000:05:00.1 to vfio-pci --- Processing 0000:05:00.2 10de:1aec Vendor:Device 10de:1aec found at 0000:05:00.2 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:05:00.2/iommu_group/devices/0000:05:00.0 /sys/bus/pci/devices/0000:05:00.2/iommu_group/devices/0000:05:00.1 /sys/bus/pci/devices/0000:05:00.2/iommu_group/devices/0000:05:00.2 /sys/bus/pci/devices/0000:05:00.2/iommu_group/devices/0000:05:00.3 Binding... 0000:05:00.0 already bound to vfio-pci 0000:05:00.1 already bound to vfio-pci 0000:05:00.2 already bound to vfio-pci 0000:05:00.3 already bound to vfio-pci Successfully bound the device 10de:1aec at 0000:05:00.2 to vfio-pci --- Processing 0000:05:00.3 10de:1aed Vendor:Device 10de:1aed found at 0000:05:00.3 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:05:00.3/iommu_group/devices/0000:05:00.0 /sys/bus/pci/devices/0000:05:00.3/iommu_group/devices/0000:05:00.1 /sys/bus/pci/devices/0000:05:00.3/iommu_group/devices/0000:05:00.2 /sys/bus/pci/devices/0000:05:00.3/iommu_group/devices/0000:05:00.3 Binding... 0000:05:00.0 already bound to vfio-pci 0000:05:00.1 already bound to vfio-pci 0000:05:00.2 already bound to vfio-pci 0000:05:00.3 already bound to vfio-pci Successfully bound the device 10de:1aed at 0000:05:00.3 to vfio-pci --- Processing 0000:03:10.1 8086:10ca Error: Device 0000:03:10.1 does not exist, unable to bind device --- Processing 0000:03:10.3 8086:10ca Error: Device 0000:03:10.3 does not exist, unable to bind device --- Processing 0000:03:10.5 8086:10ca Error: Device 0000:03:10.5 does not exist, unable to bind device --- Processing 0000:03:10.7 8086:10ca Error: Device 0000:03:10.7 does not exist, unable to bind device --- Processing 0000:03:11.1 8086:10ca Error: Device 0000:03:11.1 does not exist, unable to bind device --- Processing 0000:03:11.3 8086:10ca Error: Device 0000:03:11.3 does not exist, unable to bind device --- Processing 0000:03:11.5 8086:10ca Error: Device 0000:03:11.5 does not exist, unable to bind device --- vfio-pci binding complete Devices listed in /sys/bus/pci/drivers/vfio-pci: lrwxrwxrwx 1 root root 0 Mar 18 22:15 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 Mar 18 22:15 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 lrwxrwxrwx 1 root root 0 Mar 18 22:15 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1b.3/0000:03:00.0 lrwxrwxrwx 1 root root 0 Mar 18 22:15 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.0 lrwxrwxrwx 1 root root 0 Mar 18 22:15 0000:05:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.1 lrwxrwxrwx 1 root root 0 Mar 18 22:15 0000:05:00.2 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.2 lrwxrwxrwx 1 root root 0 Mar 18 22:15 0000:05:00.3 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.3 ls -l /dev/vfio/ total 0 crw------- 1 root root 249, 0 Mar 18 22:15 1 crw------- 1 root root 249, 1 Mar 18 22:15 12 crw------- 1 root root 249, 2 Mar 18 22:15 15 crw-rw-rw- 1 root root 10, 196 Mar 18 22:15 vfio Loading config from /boot/config/vfio-pci.cfg BIND=0000:01:00.0|10de:0fc8 0000:01:00.1|10de:0e1b 0000:03:00.0|8086:10c9 0000:05:00.0|10de:2184 0000:05:00.1|10de:1aeb 0000:05:00.2|10de:1aec 0000:05:00.3|10de:1aed 0000:03:10.1|8086:10ca 0000:03:10.3|8086:10ca 0000:03:10.5|8086:10ca 0000:03:10.7|8086:10ca 0000:03:11.1|8086:10ca 0000:03:11.3|8086:10ca 0000:03:11.5|8086:10ca --- Processing 0000:01:00.0 10de:0fc8 Vendor:Device 10de:0fc8 found at 0000:01:00.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/0000:01:00.0 /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/0000:01:00.1 Binding... 0000:01:00.0 already bound to vfio-pci 0000:01:00.1 already bound to vfio-pci Successfully bound the device 10de:0fc8 at 0000:01:00.0 to vfio-pci --- Processing 0000:01:00.1 10de:0e1b Vendor:Device 10de:0e1b found at 0000:01:00.1 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:01:00.1/iommu_group/devices/0000:01:00.0 /sys/bus/pci/devices/0000:01:00.1/iommu_group/devices/0000:01:00.1 Binding... 0000:01:00.0 already bound to vfio-pci 0000:01:00.1 already bound to vfio-pci Successfully bound the device 10de:0e1b at 0000:01:00.1 to vfio-pci --- Processing 0000:03:00.0 8086:10c9 Vendor:Device 8086:10c9 found at 0000:03:00.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:03:00.0/iommu_group/devices/0000:03:00.0 Binding... 0000:03:00.0 already bound to vfio-pci Successfully bound the device 8086:10c9 at 0000:03:00.0 to vfio-pci --- Processing 0000:05:00.0 10de:2184 Vendor:Device 10de:2184 found at 0000:05:00.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:05:00.0/iommu_group/devices/0000:05:00.0 /sys/bus/pci/devices/0000:05:00.0/iommu_group/devices/0000:05:00.1 /sys/bus/pci/devices/0000:05:00.0/iommu_group/devices/0000:05:00.2 /sys/bus/pci/devices/0000:05:00.0/iommu_group/devices/0000:05:00.3 Binding... 0000:05:00.0 already bound to vfio-pci 0000:05:00.1 already bound to vfio-pci 0000:05:00.2 already bound to vfio-pci 0000:05:00.3 already bound to vfio-pci Successfully bound the device 10de:2184 at 0000:05:00.0 to vfio-pci --- Processing 0000:05:00.1 10de:1aeb Vendor:Device 10de:1aeb found at 0000:05:00.1 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:05:00.1/iommu_group/devices/0000:05:00.0 /sys/bus/pci/devices/0000:05:00.1/iommu_group/devices/0000:05:00.1 /sys/bus/pci/devices/0000:05:00.1/iommu_group/devices/0000:05:00.2 /sys/bus/pci/devices/0000:05:00.1/iommu_group/devices/0000:05:00.3 Binding... 0000:05:00.0 already bound to vfio-pci 0000:05:00.1 already bound to vfio-pci 0000:05:00.2 already bound to vfio-pci 0000:05:00.3 already bound to vfio-pci Successfully bound the device 10de:1aeb at 0000:05:00.1 to vfio-pci --- Processing 0000:05:00.2 10de:1aec Vendor:Device 10de:1aec found at 0000:05:00.2 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:05:00.2/iommu_group/devices/0000:05:00.0 /sys/bus/pci/devices/0000:05:00.2/iommu_group/devices/0000:05:00.1 /sys/bus/pci/devices/0000:05:00.2/iommu_group/devices/0000:05:00.2 /sys/bus/pci/devices/0000:05:00.2/iommu_group/devices/0000:05:00.3 Binding... 0000:05:00.0 already bound to vfio-pci 0000:05:00.1 already bound to vfio-pci 0000:05:00.2 already bound to vfio-pci 0000:05:00.3 already bound to vfio-pci Successfully bound the device 10de:1aec at 0000:05:00.2 to vfio-pci --- Processing 0000:05:00.3 10de:1aed Vendor:Device 10de:1aed found at 0000:05:00.3 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:05:00.3/iommu_group/devices/0000:05:00.0 /sys/bus/pci/devices/0000:05:00.3/iommu_group/devices/0000:05:00.1 /sys/bus/pci/devices/0000:05:00.3/iommu_group/devices/0000:05:00.2 /sys/bus/pci/devices/0000:05:00.3/iommu_group/devices/0000:05:00.3 Binding... 0000:05:00.0 already bound to vfio-pci 0000:05:00.1 already bound to vfio-pci 0000:05:00.2 already bound to vfio-pci 0000:05:00.3 already bound to vfio-pci Successfully bound the device 10de:1aed at 0000:05:00.3 to vfio-pci --- Processing 0000:03:10.1 8086:10ca Vendor:Device 8086:10ca found at 0000:03:10.1 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:03:10.1/iommu_group/devices/0000:03:10.1 Binding... Unbound 0000:03:10.1 from igbvf Successfully bound the device 8086:10ca at 0000:03:10.1 to vfio-pci --- Processing 0000:03:10.3 8086:10ca Vendor:Device 8086:10ca found at 0000:03:10.3 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:03:10.3/iommu_group/devices/0000:03:10.3 Binding... Unbound 0000:03:10.3 from igbvf Successfully bound the device 8086:10ca at 0000:03:10.3 to vfio-pci --- Processing 0000:03:10.5 8086:10ca Vendor:Device 8086:10ca found at 0000:03:10.5 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:03:10.5/iommu_group/devices/0000:03:10.5 Binding... Unbound 0000:03:10.5 from igbvf Successfully bound the device 8086:10ca at 0000:03:10.5 to vfio-pci --- Processing 0000:03:10.7 8086:10ca Vendor:Device 8086:10ca found at 0000:03:10.7 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:03:10.7/iommu_group/devices/0000:03:10.7 Binding... Unbound 0000:03:10.7 from igbvf Successfully bound the device 8086:10ca at 0000:03:10.7 to vfio-pci --- Processing 0000:03:11.1 8086:10ca Vendor:Device 8086:10ca found at 0000:03:11.1 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:03:11.1/iommu_group/devices/0000:03:11.1 Binding... Unbound 0000:03:11.1 from igbvf Successfully bound the device 8086:10ca at 0000:03:11.1 to vfio-pci --- Processing 0000:03:11.3 8086:10ca Vendor:Device 8086:10ca found at 0000:03:11.3 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:03:11.3/iommu_group/devices/0000:03:11.3 Binding... Unbound 0000:03:11.3 from igbvf Successfully bound the device 8086:10ca at 0000:03:11.3 to vfio-pci --- Processing 0000:03:11.5 8086:10ca Vendor:Device 8086:10ca found at 0000:03:11.5 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:03:11.5/iommu_group/devices/0000:03:11.5 Binding... Unbound 0000:03:11.5 from igbvf Successfully bound the device 8086:10ca at 0000:03:11.5 to vfio-pci --- vfio-pci binding complete Devices listed in /sys/bus/pci/drivers/vfio-pci: lrwxrwxrwx 1 root root 0 Mar 19 11:15 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 Mar 19 11:15 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 lrwxrwxrwx 1 root root 0 Mar 19 11:15 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1b.3/0000:03:00.0 lrwxrwxrwx 1 root root 0 Mar 19 11:15 0000:03:10.1 -> ../../../../devices/pci0000:00/0000:00:1b.3/0000:03:10.1 lrwxrwxrwx 1 root root 0 Mar 19 11:15 0000:03:10.3 -> ../../../../devices/pci0000:00/0000:00:1b.3/0000:03:10.3 lrwxrwxrwx 1 root root 0 Mar 19 11:15 0000:03:10.5 -> ../../../../devices/pci0000:00/0000:00:1b.3/0000:03:10.5 lrwxrwxrwx 1 root root 0 Mar 19 11:15 0000:03:10.7 -> ../../../../devices/pci0000:00/0000:00:1b.3/0000:03:10.7 lrwxrwxrwx 1 root root 0 Mar 19 11:15 0000:03:11.1 -> ../../../../devices/pci0000:00/0000:00:1b.3/0000:03:11.1 lrwxrwxrwx 1 root root 0 Mar 19 11:15 0000:03:11.3 -> ../../../../devices/pci0000:00/0000:00:1b.3/0000:03:11.3 lrwxrwxrwx 1 root root 0 Mar 19 11:15 0000:03:11.5 -> ../../../../devices/pci0000:00/0000:00:1b.3/0000:03:11.5 lrwxrwxrwx 1 root root 0 Mar 19 11:15 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.0 lrwxrwxrwx 1 root root 0 Mar 19 11:15 0000:05:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.1 lrwxrwxrwx 1 root root 0 Mar 19 11:15 0000:05:00.2 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.2 lrwxrwxrwx 1 root root 0 Mar 19 11:15 0000:05:00.3 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.3 ls -l /dev/vfio/ total 0 crw------- 1 root root 249, 0 Mar 19 11:15 1 crw------- 1 root root 249, 1 Mar 19 11:15 12 crw------- 1 root root 249, 2 Mar 19 11:15 15 crw------- 1 root root 249, 3 Mar 19 11:15 16 crw------- 1 root root 249, 4 Mar 19 11:15 17 crw------- 1 root root 249, 5 Mar 19 11:15 18 crw------- 1 root root 249, 6 Mar 19 11:15 19 crw------- 1 root root 249, 7 Mar 19 11:15 20 crw------- 1 root root 249, 8 Mar 19 11:15 21 crw------- 1 root root 249, 9 Mar 19 11:15 22 crw-rw-rw- 1 root root 10, 196 Mar 19 11:15 vfio Edited March 20, 2021 by s1oz 2 Quote Link to comment
ljm42 Posted March 19, 2021 Share Posted March 19, 2021 Hey that is great to hear! Thanks for posting the final go script, to help document this for other folks would you mind adding screenshots of System Devices and maybe copy/paste the vfio-pci log too? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.