Jump to content
jonp

***GUIDE*** Passing Through Network Controllers to unRAID 6 Virtual Machines

142 posts in this topic Last Reply

Recommended Posts

I have followed the 3rd method in this video to bypass one of my nics. I have a pcie network card with 2 nics i350

 

This is the iommu group

IOMMU group 6:

 

[1022:43bb] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset USB 3.1 xHCI Controller (rev 02)

  [1022:43b7] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset SATA Controller (rev 02)
  [1022:43b2] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b2 (rev 02)
  [1022:43b4] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
  [1022:43b4] 02:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
  [1022:43b4] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
  [10ec:8168] 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
  [8086:1521] 05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
  [8086:1521] 05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

 

I want to bypass only the last nic, or in the worst case both of them (the 2 last) which belong to the same pcie card.

 

I have edited the syslinux config accordingly

default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append iommu=pt vfio-pci.ids=8086:1521 initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

This is the pfsense VM

 

<domain type='kvm'>
  <name>pfSense</name>
  <uuid>2769b456-605b-c3ca-fbf0-29c5f3799322</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/>
  </metadata>
  <memory unit='KiB'>2621440</memory>
  <currentMemory unit='KiB'>2621440</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='5'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.6'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/VMDisks/pfsense/pfSense.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:5a:66:6f'/>
      <source bridge='br1'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:f6:52:6a'/>
      <source bridge='br2'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='es'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

And I get this error when I try to start the VM

 

Execution error

internal error: qemu unexpectedly closed the monitor: 2018-03-14T21:32:13.226454Z qemu-system-x86_64: -device vfio-pci,host=05:00.1,id=hostdev0,bus=pci.5,addr=0x0: vfio error: 0000:05:00.1: group 6 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

 

 
What I'm doing wrong?

 

Captura.PNG

Share this post


Link to post

After adding this to the syslinux confi now it works

 

label unRAID OS
  menu default
  kernel /bzimage
  append pcie_acs_override=downstream,multifunction iommu=pt vfio-pci.ids=8086:1521 initrd=/bzroot

 

IOMMU group 14: [8086:1521] 05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
 
IOMMU group 15: [8086:1521] 05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

 

But both NICS are being blacklisted and passthrough to the VM and I only want the second, it is possible? any other workaround to try?

Share this post


Link to post
On 3/12/2018 at 5:36 AM, Osiris said:

I want to pass it through to a VM. I've reviewed my own post (don't laugh) and don't really see where I might have been confusing about this.

I spoke about network card isolation as to be possibly offered via the unraid GUI (in order to pass it through to a VM).

 

I was just asking if the tutorial video was indeed not really applicable to my situation, as both my NICs are part of the same 'id', namely 8086:1533

So thanks for confirming that.

 

Did you figure this out? I'm in the same boat as you with 4 onboard nics all reporting the same info.

I was able to use xen-pciback.hide=(02:00.1) on one of mine and it now is blacklisted from unRaid, but my VM still doesnt see it listed as an available pci device

Share this post


Link to post
8 hours ago, amp3d said:

 

Did you figure this out? I'm in the same boat as you with 4 onboard nics all reporting the same info.

I was able to use xen-pciback.hide=(02:00.1) on one of mine and it now is blacklisted from unRaid, but my VM still doesnt see it listed as an available pci device

 

Then you need to add it manually to the vm XML. I didn't remember that it doesn't show up in that list. 

I did make a request (I think I did) to add devices hidden with xen-pciback.hide to the list, but never got an answer. 

Share this post


Link to post
On 3/14/2018 at 5:42 PM, L0rdRaiden said:

After adding this to the syslinux confi now it works

 

label unRAID OS
  menu default
  kernel /bzimage
  append pcie_acs_override=downstream,multifunction iommu=pt vfio-pci.ids=8086:1521 initrd=/bzroot

 

IOMMU group 14: [8086:1521] 05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
 
IOMMU group 15: [8086:1521] 05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

 

But both NICS are being blacklisted and passthrough to the VM and I only want the second, it is possible? any other workaround to try?

 

Perfect!  This worked for my quad Intel Nic too!   Thank you for sharing your solution!  

Share this post


Link to post

Sorry I havent kept up to date on this as I have been super busy at work.

IMHO to get the best results and use out of pfsense running on a VM you really want to have a separate dual NIC if you can spare a x4 slot on your board that is.

 

My setup uses a billion modem router which is setup as a modem only and because pfsense uses dual NIC it can afford me to use one port to handle my WAN connection and one port to handle my gateway. 

From there I have a 8 port switch that is connected to the gateway port. It may sound like a funny way to do it, but doing it like this I set a static ip for unraid and because all pc's are connected to the switch I can communicate with them at all time... even when pfsense isnt running.

 

If you want more clarification just let me know.

Share this post


Link to post

If i try this i get a big problem, because its an server with a 4 port nic onboard al have the same vendor en dev ids so they al get passed trough even if i do 

append  pcie_acs_override=downstream,multifunction iommu=pt vfio-pci.ids=14e4:1639 initrd=/bzroot,/bzroot-gui
  IOMMU group 14:    [14e4:1639] 01:00.0 Ethernet controller: BCM5709 Gigabit Ethernet
  IOMMU group 15:    [14e4:1639] 01:00.1 Ethernet controller: BCM5709 Gigabit Ethernet
  IOMMU group 16:    [14e4:1639] 02:00.0 Ethernet controller: BCM5709 Gigabit Ethernet
  IOMMU group 16:    [14e4:1639] 02:00.1 Ethernet controller: BCM5709 Gigabit Ethernet
what i want to acomplish is that nic eth0 1:00.0 is left alone because that is the port that is staticly configured so it can be coneected to from outside for the web interface.

nic eth1 1:00.1 need to be passed trough to vms (sophos/pfsence) 

nic eth2 02:00.0 and eth3 02:00.1 where a failover bond/bridge br2 for the other vms and docker to use for internet connection.

 

but if i remove the iommu groups and the code between the append and initrd=/bzroot i cant pass the cards trough. 

 

can some one point out if this is even poosible or not ?

 

regard 
Michel

Edited by sojab0on
Capital ETH changed to eth and added pci ids for eth2 and eth3

Share this post


Link to post

Hey guys this is my first post, after doing allot of googling and reading this thread, I have one question. I only have 1 nic with 4 10g ports, I want to assign this physical card to my pfsense vm, I know this would temporarily cut off my network, but once I have pfsense setup, could I setup a virtual connection (ideally 10g) to the unRaid machine and get on my network and internet from there? The 10g card will be going to a 10g switch to a bunch of ipcams. I want the server to have full 10g connectivity with the cams.

01:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)

 

 

Share this post


Link to post
On 11/13/2017 at 11:08 AM, johnnie.black said:

 

Update on this, as it may be useful info for other users, without ACS override the X11SSM-F groups together the two top PCIe slots, so I've been using ACS override for more than year because I wanted to pass-trough an addon NIC on the 2nd slot, server has been pretty much 100% stable, until recently when I added a SATA controller to the other PCIe slot, since then my VMs started crashing regularly, in hindsight, mostly when both the NIC and SATA controller were being used at the same time, so I disabled ACS override, swapped the NIC to a now free bottom slot that is on its own group and no more issues, so ACS override may appear to work correctly but not be completely stable in some cases.

I'm assuming the ones grouped together are the x16 slot and adjacent? All others are isolated?

Share this post


Link to post
6 minutes ago, aberg83 said:

I'm assuming the ones grouped together are the x16 slot and adjacent? All others are isolated?

Yes, both CPU slots are in the same group, both PCH slots are alone on their own group.

Share this post


Link to post
5 minutes ago, johnnie.black said:

Yes, both CPU slots are in the same group, both PCH slots are alone on their own group.

By chance do you use a xeon with quicksync with this board and have you tried plex hw transcoding. Not sure how that would work with a board that also has an ipmi gpu.

Share this post


Link to post
By chance do you use a xeon with quicksync with this board and have you tried plex hw transcoding. Not sure how that would work with a board that also has an ipmi gpu.
Sorry, don't use plex.

Sent from my SM-T230 using Tapatalk

Share this post


Link to post

I wonder if anyone can help with a slightly odd issue I'm having. I can only pass half a 2 port NIC through.

 

I've successfully followed the steps to get unraid to allow me to pass my broadcom BCR5720 Dual ethernet card through to a VM, however, the VM OS can only see one of the ports. 

 

If i disable the 'stubbing' I can see both devices in unraid, and they appear to work (gets an IP over them at least). This is across multiple VM oses, although the particular one i'm testing currently is pfsense (built on BSD).

 

My IOMMU from SysDevs - Group 11 is the onboard dual port NIC, Group 13 is the PCI-e one I'm passing through

IOMMU group 0:	[8086:3403] 00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)
IOMMU group 1:	[8086:3408] 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
IOMMU group 2:	[8086:340a] 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
IOMMU group 3:	[8086:340e] 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
IOMMU group 4:	[8086:3410] 00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)
IOMMU group 5:	[8086:3411] 00:0a.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 13)
IOMMU group 6:	[8086:342e] 00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13)
[8086:3422] 00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
[8086:3423] 00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
IOMMU group 7:	[8086:3a37] 00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
[8086:3a38] 00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
[8086:3a3c] 00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
IOMMU group 8:	[8086:3a34] 00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
[8086:3a35] 00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
[8086:3a36] 00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
[8086:3a39] 00:1d.3 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
[8086:3a3a] 00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
IOMMU group 9:	[8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
[102b:0532] 06:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
IOMMU group 10:	[8086:3a16] 00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
IOMMU group 11:	[14e4:163b] 01:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
[14e4:163b] 01:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
IOMMU group 12:	[1000:0072] 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
IOMMU group 13:	[14e4:165f] 03:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
[14e4:165f] 03:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
IOMMU group 14:	[8086:2c70] fe:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02)
[8086:2d81] fe:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02)
IOMMU group 15:	[8086:2d90] fe:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02)
[8086:2d91] fe:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02)
[8086:2d92] fe:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02)
[8086:2d93] fe:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02)
[8086:2d94] fe:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02)
[8086:2d95] fe:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02)
IOMMU group 16:	[8086:2d98] fe:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02)
[8086:2d99] fe:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02)
[8086:2d9a] fe:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02)
[8086:2d9c] fe:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02)
IOMMU group 17:	[8086:2da0] fe:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02)
[8086:2da1] fe:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02)
[8086:2da2] fe:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02)
[8086:2da3] fe:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02)
IOMMU group 18:	[8086:2da8] fe:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02)
[8086:2da9] fe:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02)
[8086:2daa] fe:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02)
[8086:2dab] fe:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02)
IOMMU group 19:	[8086:2db0] fe:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02)
[8086:2db1] fe:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02)
[8086:2db2] fe:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02)
[8086:2db3] fe:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02)
IOMMU group 20:	[8086:2c70] ff:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02)
[8086:2d81] ff:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02)
IOMMU group 21:	[8086:2d90] ff:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02)
[8086:2d91] ff:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02)
[8086:2d92] ff:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02)
[8086:2d93] ff:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02)
[8086:2d94] ff:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02)
[8086:2d95] ff:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02)
IOMMU group 22:	[8086:2d98] ff:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02)
[8086:2d99] ff:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02)
[8086:2d9a] ff:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02)
[8086:2d9c] ff:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02)
IOMMU group 23:	[8086:2da0] ff:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02)
[8086:2da1] ff:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02)
[8086:2da2] ff:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02)
[8086:2da3] ff:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02)
IOMMU group 24:	[8086:2da8] ff:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02)
[8086:2da9] ff:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02)
[8086:2daa] ff:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02)
[8086:2dab] ff:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02)
IOMMU group 25:	[8086:2db0] ff:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02)
[8086:2db1] ff:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02)
[8086:2db2] ff:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02)
[8086:2db3] ff:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02)

and my syslinux config:

append pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 vfio-pci.ids=14e4:165f initrd=/bzroot

If anyone has any suggestions what to try I'd appreciate it. 

Share this post


Link to post
Posted (edited)

I followed the steps in @SpaceInvaderOneto passthrough my NIC to a pfsense vm, but I'm getting an error at vm start. Please see my post here: 

Does anyone have any ideas to help me get past this? 

 

UPDATE: this fixed my issue:

https://forums.unraid.net/topic/37959-guide-passing-through-network-controllers-to-unraid-6-virtual-machines/?do=findComment&comment=592661

Brb dinner

Edited by guruleenyc

Share this post


Link to post
On 9/26/2017 at 11:10 PM, Azyx said:

 

OMG! Thank you so much Saarg, I've been trying the whole day to passthrough my dual interface Intel NIC.
Tried with and without the "pci-stub" and "vfio-pci" in the syslinux config. Also tried to pass just one of the interfaces.... Tried so many things until i stumbled upon your post with the link for the "vfio_iommu_type1.allow_unsafe_interrupts=1" command. This should be added to the OP's post, I think.

 

AWESOME! This resolved the same error for me as well on an IBM x3200 M3

Thank you! 

Share this post


Link to post
On 5/1/2015 at 8:36 AM, jonp said:

As an example, in mounting an SMB share to my SSD-based cache pool from inside my Windows VM, I was able to see IO throughput to the share exceed 250MB/s (that's megabytes, not bits).

I've got this working but I'd like to reverse the example to share a VM filesystem and then mount on my Unraid host using VirtIO. Is this possible somehow?

Share this post


Link to post
On 3/14/2018 at 5:34 PM, L0rdRaiden said:

I have followed the 3rd method in this video to bypass one of my nics. I have a pcie network card with 2 nics i350

 

This is the iommu group

IOMMU group 6:

 

[1022:43bb] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset USB 3.1 xHCI Controller (rev 02)

  [1022:43b7] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset SATA Controller (rev 02)
  [1022:43b2] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b2 (rev 02)
  [1022:43b4] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
  [1022:43b4] 02:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
  [1022:43b4] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
  [10ec:8168] 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
  [8086:1521] 05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
  [8086:1521] 05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

 

I want to bypass only the last nic, or in the worst case both of them (the 2 last) which belong to the same pcie card.

 

I have edited the syslinux config accordingly


default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append iommu=pt vfio-pci.ids=8086:1521 initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

This is the pfsense VM

 


<domain type='kvm'>
  <name>pfSense</name>
  <uuid>2769b456-605b-c3ca-fbf0-29c5f3799322</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/>
  </metadata>
  <memory unit='KiB'>2621440</memory>
  <currentMemory unit='KiB'>2621440</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='5'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.6'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/VMDisks/pfsense/pfSense.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:5a:66:6f'/>
      <source bridge='br1'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:f6:52:6a'/>
      <source bridge='br2'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='es'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

And I get this error when I try to start the VM

 

Execution error

internal error: qemu unexpectedly closed the monitor: 2018-03-14T21:32:13.226454Z qemu-system-x86_64: -device vfio-pci,host=05:00.1,id=hostdev0,bus=pci.5,addr=0x0: vfio error: 0000:05:00.1: group 6 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

 

 
What I'm doing wrong?

 

Captura.PNG

What a life saver.   This video is must watch if trying to figure out why CentOS will not install with networking configured.

I came from the Qnap world and all of this done for you.

With this amazing video was finally able to install CentOS!

 

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.