doug.holcombe Posted February 25, 2017 Share Posted February 25, 2017 Hello, I have been using unRaid for the past year half. Its been the best thing ever! THANK YOU LIME Technology and others! So I am needing some help now. I know that this has been hashed bashed and trashed over a bit. I have done all my searching. I have tried a number of different findings and still no luck with this. Here is my current setup System Information Model: Custom M/B: Supermicro - X10SL7-F CPU: Intel® Xeon® CPU E3-1241 v3 @ 3.50GHz HVM: Enabled IOMMU: Enabled Cache: 256 kB, 1024 kB, 8192 kB Memory: 32 GB (max. installable capacity 32 GB) Network: eth0: 1000 Mb/s, full duplex, mtu 1500 eth1: not connected Kernel: Linux 4.9.10-unRAID x86_64 OpenSSL: 1.0.2k Uptime: PCI Devices and IOMMU Groups IOMMU group 11 [8086:1533] 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) IOMMU group 12 [8086:1533] 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) IOMMU group 13 [8086:1521] 07:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection IOMMU group 14 [8086:1521] 07:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection IOMMU group 15 [8086:1521] 07:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection IOMMU group 16 [8086:1521] 07:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection Here is what I am trying to accomplish: I want to pass-through the last two individual ports of my Intel I350-T4 NIC to one of my VM that is running Debian. So that would be [8086:1521] 07:00.2 and [8086:1521] 07:00.3 that I want to use on that Quad port NIC. BUT from my reading and self education on this, I cannot pick out these ports separately and leave the others for unRaid. And that is fine as I would like to use the other two for two other VMs. So I would need to stub the whole card so that unRaid doesn't see it and want to use it. Here is what I have done so far: I have added this to the syslinux.cfg: append pci-stub.ids=8086:1521 initrd=/bzroot I have even tried append vfio-pci.ids=8086:1521 initrd=/bzroot I personally do not know what the difference is. But they both seem to do the same thing and allowing the WHOLE NIC card to be stubbed from the unRaid server. After a reboot of the unRaid server. I see that this is successful in this part of the steps. Now onto the addition of the two ports that I want a Debian VM to use. Here is what I can see and then what two ports I have selected. And here is the XML for this VM image PBX xml.txt Here is the most important part of the whole thing. <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/> </hostdev> All of this should be right from all the research that I have done. But the VM does not see the two cards. Here are all the Hidden different posts that I have found outside of the sticky main posts on this subject. These sure did help me get to this point! and I want to acknowledge the person behind each one and say Thank you! I feel that there are more posts that I have read on this subject. Its not that hard! but its not wanting to work. What am I missing? Why will this not work? Thank you Doug Quote Link to comment
saarg Posted February 25, 2017 Share Posted February 25, 2017 You can use the pcie number to stub instead of the whole card. This is what I have in my syslinux.cfg for 3 of my 4 ports. xen-pciback.hide=(04:00.3)(04:00.2)(04:00.1) If you have made the VM previous to adding the network cards, you might have to add them manually. At least I had to in my Ubuntu server VM. What is the output of the below command in the VM. lspci Quote Link to comment
doug.holcombe Posted February 25, 2017 Author Share Posted February 25, 2017 39 minutes ago, saarg said: You can use the pcie number to stub instead of the whole card. This is what I have in my syslinux.cfg for 3 of my 4 ports. xen-pciback.hide=(04:00.3)(04:00.2)(04:00.1) If you have made the VM previous to adding the network cards, you might have to add them manually. At least I had to in my Ubuntu server VM. What is the output of the below command in the VM. lspci root@pbx:~# lspci 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller 00:01.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 04) 00:07.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03) 00:07.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03) 00:07.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03) 00:07.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02) 01:00.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge 02:01.0 Ethernet controller: Red Hat, Inc Virtio network device 02:02.0 Communication controller: Red Hat, Inc Virtio console 02:03.0 SCSI storage controller: Red Hat, Inc Virtio block device 02:04.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection 02:05.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection 02:06.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon root@pbx:~# Oh SNAP! They have been pass through! So off to figure out how to add them in to the OS so that it can see them. Do you have any starting points for me? Thank you! Doug Quote Link to comment
saarg Posted February 25, 2017 Share Posted February 25, 2017 31 minutes ago, doug.holcombe said: root@pbx:~# lspci 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller 00:01.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 04) 00:07.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03) 00:07.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03) 00:07.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03) 00:07.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02) 01:00.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge 02:01.0 Ethernet controller: Red Hat, Inc Virtio network device 02:02.0 Communication controller: Red Hat, Inc Virtio console 02:03.0 SCSI storage controller: Red Hat, Inc Virtio block device 02:04.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection 02:05.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection 02:06.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon root@pbx:~# Oh SNAP! They have been pass through! So off to figure out how to add them in to the OS so that it can see them. Do you have any starting points for me? Thank you! Doug You need to add them in /etc/network/interfaces. Looks like below. auto eth0 iface eth0 inet dhcp Most likely, your new interfaces will be eth1 and eth2. So just add two of the above and replace with the correct ethx. For more info you can read more here Quote Link to comment
Repooc Posted January 2, 2019 Share Posted January 2, 2019 On 2/25/2017 at 2:42 PM, saarg said: You can use the pcie number to stub instead of the whole card. This is what I have in my syslinux.cfg for 3 of my 4 ports. xen-pciback.hide=(04:00.3)(04:00.2)(04:00.1) If you have made the VM previous to adding the network cards, you might have to add them manually. At least I had to in my Ubuntu server VM. What is the output of the below command in the VM. lspci So does this mean I can pass 2 of my 4 ports on a nic to my vm's and let unraid have the other 2 this way or do i still need a seperate nic to do this with and pass the whole care even though in my device list, the card has 2 ports in each of their own groups but with same vendor id. Quote Link to comment
saarg Posted January 2, 2019 Share Posted January 2, 2019 If they are in separate groups, you can pass one through and let unraid use the others. If you use the method I mentioned, you use the PCI ID and not the vendor ID. This way you can pass hide which part of the card you want to pass through. Only thing different to the normal stubbing, is that you have to manually add the device tag in the xml. You can't choose it in the other devices list. 1 Quote Link to comment
Repooc Posted January 2, 2019 Share Posted January 2, 2019 1 minute ago, saarg said: If they are in separate groups, you can pass one through and let unraid use the others. If you use the method I mentioned, you use the PCI ID and not the vendor ID. This way you can pass hide which part of the card you want to pass through. Only thing different to the normal stubbing, is that you have to manually add the device tag in the xml. You can't choose it in the other devices list. Sounds like this is what I am aiming for... This is the IOMMU group info for my nic, not sure if there is a way to figure out which corresponds to which port... IOMMU group 13: [14e4:1639] 01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) [14e4:1639] 01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) IOMMU group 14: [14e4:1639] 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) [14e4:1639] 02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) Would this be my append line then? append xen-pciback.hide=(02:00.0)(02:00.1) initrd=/bzroot Would this be what I need in my VM xml then??? <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> </hostdev> Quote Link to comment
SpaceInvaderOne Posted January 2, 2019 Share Posted January 2, 2019 43 minutes ago, Repooc said: Sounds like this is what I am aiming for... This is the IOMMU group info for my nic, not sure if there is a way to figure out which corresponds to which port... IOMMU group 13: [14e4:1639] 01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) [14e4:1639] 01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) IOMMU group 14: [14e4:1639] 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) [14e4:1639] 02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) Would this be my append line then? append xen-pciback.hide=(02:00.0)(02:00.1) initrd=/bzroot Would this be what I need in my VM xml then??? <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> </hostdev> Your XML looks good and will pass through the iommu group 14. Your append line it looks fine too. But just as a side note many people (myself included until jonp told me a while ago) you can just add extra parameters to the end of the line. They don't have to be between the append and the initrd so the line can look like this as you have it append xen-pciback.hide=(02:00.0)(02:00.1) initrd=/bzroot or just at the end append initrd=/bzroot xen-pciback.hide=(02:00.0)(02:00.1) they just have to be on the same line. I think it is less likely to make a typo have it at the end. 1 Quote Link to comment
Repooc Posted January 2, 2019 Share Posted January 2, 2019 (edited) Thanks for clarification, much appreciated. Will test it out when plex isn't in use. Edit: Seems to work as intended... woohoo!! Edited January 2, 2019 by Repooc Edit with testing results Quote Link to comment
saarg Posted January 2, 2019 Share Posted January 2, 2019 No easy way to tell which port is which, but usually the first PCI number is one of the ports on one of the ends. Then you just have to check with a cable which one work in the vm. The second port will be the one beside it. Quote Link to comment
Repooc Posted January 3, 2019 Share Posted January 3, 2019 I had found a post where someone posted a handy bash script to list your nic ports with link status in which i had plugged in to the router/switch and was able to see which was connected. #!/bin/bash for ETH in $(find /sys/devices/ -type d -iname "eth[0-9]"); do port=$(basename $ETH); export $(cat $ETH/device/uevent); link_status=$(cat $ETH/operstate); desc=$(lspci -d $PCI_ID| cut -d ':' -f3) pci_bus=$(echo $PCI_SLOT_NAME|sed -e 's#0000:##') echo -e "Found NIC: $port\n Name: $desc\n PCI id: $PCI_ID\n PCI bus: $pci_bus\n Status: $link_status\n" done Can't recall the original author that posted this but credit goes to them. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.