***GUIDE*** Passing Through Network Controllers to unRAID 6 Virtual Machines


jonp

Recommended Posts

  • 3 weeks later...

Before knowing that they are not optimal for VMs I got a Skylake board and Xeon to upgrade one of my servers, I need to pass trough at least one NIC and I need to enable PCIe ACS override to be able to do it, I tested and it works, my question is:

 

Is this usually stable? With any issues be easily detected? I just don't want to replace the hardware and find out that the system is not stable after a few days or weeks of use.

Link to comment

Before knowing that they are not optimal for VMs I got a Skylake board and Xeon to upgrade one of my servers, I need to pass trough at least one NIC and I need to enable PCIe ACS override to be able to do it, I tested and it works, my question is:

 

Is this usually stable? With any issues be easily detected? I just don't want to replace the hardware and find out that the system is not stable after a few days or weeks of use.

 

I guess this question doesn't have an easy answer, so I'll try another, is anyone here using a Supermicro X11SSM-F with ACS override? Any issues?

Link to comment
  • 4 weeks later...

Hey there i need a bit of help.

I'm a noob so have a bit of patience please.

 

I have a unraid box that has 2 nic's on the motherboard and a pci express  x4 card with 4 1gb ports.

 

I want to install pfsense to be a firewall and put the ISP PPPoE connection in one NIC that is NOT available to unraid ( want the internet to go in pfsense and then be routed in my home switch and give internet to all devices (pc, xbox, tv, unraid etc.) and not have my unraid exposed)

 

can you help ?

PfSense is not absolutely necessary (can use different os) i'm just familiar with this one.

 

Thank you

Link to comment
  • 2 months later...
  • 1 month later...
  • 3 weeks later...

I'm struggling to passthrough my cards to a pfSense VM: 

At first the cards were shown in separate groups, now they are in the same group and the device ID changed (??).

 

I'm not sure if this is the problem though as when pfSense boots it says:

 

pci2: <network, ethernet> at device 5.0 (no driver attached)
pci2: <network, ethernet> at device 6.0 (no driver attached)

any ideas?   Thanks

Link to comment
  • 4 weeks later...

So after fighting and losing to using two bridges with pfsense (Physical machines could access the Internet but VMs on br0 couldn't - appeared to be an issue with inbound packets to the VMs), I passed through two of my Intel NICs (same vendor IDs).

 

I was fortunate that they were in different IOMMU groups, and all I did was add the hostdev rows to the pfsense VM XML. No syslinux.cfg stubs and no ACS override. Really simple in the end. Unraid no longer sees the cards in the network settings.

 

Did something change in 6.3.2 to make this so easy? Should I expect the world to end as I'm missing something?

 

My Windows VM can now use the Internet, I can RDP to it from the internet (although this did work from the LAN even though internet didn't).

 

I'm curious why twin bridges didn't work, I might search harder for an answer one day out of curiosity.

 

Thanks all, couldn't have done it without this thread. ESXi night night!

John

Link to comment
6 hours ago, johner said:

So after fighting and losing to using two bridges with pfsense (Physical machines could access the Internet but VMs on br0 couldn't - appeared to be an issue with inbound packets to the VMs), I passed through two of my Intel NICs (same vendor IDs).

 

I was fortunate that they were in different IOMMU groups, and all I did was add the hostdev rows to the pfsense VM XML. No syslinux.cfg stubs and no ACS override. Really simple in the end. Unraid no longer sees the cards in the network settings.

 

Did something change in 6.3.2 to make this so easy? Should I expect the world to end as I'm missing something?

 

My Windows VM can now use the Internet, I can RDP to it from the internet (although this did work from the LAN even though internet didn't).

 

I'm curious why twin bridges didn't work, I might search harder for an answer one day out of curiosity.

 

Thanks all, couldn't have done it without this thread. ESXi night night!

John

 

I would suggest you stub them. The reason is that then the drivers aren't loaded in unraid and less chances of something suddenly not working. The other reason is that you can choose the cards in the other PCI device list in the vm template. When adding stuff to the XML, it gets wiped if you do any edits in the vm template. 

Link to comment
  • 2 months later...
On 1.5.2015 at 5:36 PM, jonp said:

 


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
      </source>
    </hostdev>
Modify the address line entering in the two digit bus, slot, and function from your ID.  So 00:19.0 translates to what I have above.

 

My question:

How can I do this for four NIC´s? I have a quad Port NIC and want to use it in pfSense.

 

Link to comment
  • 2 months later...

I installed a TP-Link TG-3468 (Realtek) network card into my system, it sees it but its grouped with my motherboard Ethernet (Intel) IOMMU group 14. My ACS Override is ON. and I tried swapping the card to another PCIe Slot. Any other way to force it to another seperate IOMMU group?

 

EDIT:

Solution:

Have ACS patch ON, in Main>Flash>Syslinux configuration:

I changed pcie_acs_override=downstream  to  pcie_acs_override=multifunction

 

This separated the IOMMU groups even more. Ill leave this here for future People

 

Quote

IOMMU group 0
	[1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 1
	[1022:1453] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
IOMMU group 2
	[1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
IOMMU group 3
	[1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 4
	[1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 5
	[1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
IOMMU group 6
	[1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 7
	[1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 8
	[1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
IOMMU group 9
	[1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 10
	[1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
IOMMU group 11
	[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
	[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 12
	[1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460
	[1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461
	[1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462
	[1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463
	[1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464
	[1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465
	[1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466
	[1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467
IOMMU group 13
	[1987:5007] 01:00.0 Non-Volatile memory controller: Device 1987:5007 (rev 01)
IOMMU group 14
	[1022:43b9] 03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43b9 (rev 02)
	[1022:43b5] 03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b5 (rev 02)
	[1022:43b0] 03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b0 (rev 02)
	[1022:43b4] 1d:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[1022:43b4] 1d:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[1022:43b4] 1d:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[1022:43b4] 1d:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[1022:43b4] 1d:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[1022:43b4] 1d:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[10ec:8168] 24:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
	[1b21:1343] 25:00.0 USB controller: ASMedia Technology Inc. Device 1343
	[8086:1539] 26:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
IOMMU group 15
	[10de:1187] 28:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 760] (rev a1)
	[10de:0e0a] 28:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
IOMMU group 16
	[1022:145a] 29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
	[1022:1456] 29:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456
	[1022:145c] 29:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 145c
IOMMU group 17
	[1022:1455] 2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
	[1022:7901] 2a:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
	[1022:1457] 2a:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457

 

Edited by Boomer42
Link to comment
  • 1 month later...
On 13/04/2016 at 11:35 AM, saarg said:

 

You should try this from the unraid wiki

 

The extra line added is supposed to be added. It is the address inside the specific VM. So nothing to worry about.

 

 


 <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
 

 

 

OMG! Thank you so much Saarg, I've been trying the whole day to passthrough my dual interface Intel NIC.
Tried with and without the "pci-stub" and "vfio-pci" in the syslinux config. Also tried to pass just one of the interfaces.... Tried so many things until i stumbled upon your post with the link for the "vfio_iommu_type1.allow_unsafe_interrupts=1" command. This should be added to the OP's post, I think.

 

  • Upvote 1
Link to comment
On 19/10/2016 at 6:32 AM, white13wolf said:

Hey there i need a bit of help.

I'm a noob so have a bit of patience please.

 

I have a unraid box that has 2 nic's on the motherboard and a pci express  x4 card with 4 1gb ports.

 

I want to install pfsense to be a firewall and put the ISP PPPoE connection in one NIC that is NOT available to unraid ( want the internet to go in pfsense and then be routed in my home switch and give internet to all devices (pc, xbox, tv, unraid etc.) and not have my unraid exposed)

 

can you help ?

PfSense is not absolutely necessary (can use different os) i'm just familiar with this one.

 

Thank you

If you still need help with this let me know as I have spent some time getting the perfect pfsense vm running on unRAID with a fallback if unRAID craps itself for some reason.

  • Like 1
Link to comment
  • 1 month later...
On 25/09/2016 at 12:58 AM, johnnie.black said:
On 24/09/2016 at 5:27 PM, johnnie.black said:

Before knowing that they are not optimal for VMs I got a Skylake board and Xeon to upgrade one of my servers, I need to pass trough at least one NIC and I need to enable PCIe ACS override to be able to do it, I tested and it works, my question is:

 

Is this usually stable? With any issues be easily detected? I just don't want to replace the hardware and find out that the system is not stable after a few days or weeks of use.

 

 

I guess this question doesn't have an easy answer, so I'll try another, is anyone here using a Supermicro X11SSM-F with ACS override? Any issues?

 

Update on this, as it may be useful info for other users, without ACS override the X11SSM-F groups together the two top PCIe slots, so I've been using ACS override for more than year because I wanted to pass-trough an addon NIC on the 2nd slot, server has been pretty much 100% stable, until recently when I added a SATA controller to the other PCIe slot, since then my VMs started crashing regularly, in hindsight, mostly when both the NIC and SATA controller were being used at the same time, so I disabled ACS override, swapped the NIC to a now free bottom slot that is on its own group and no more issues, so ACS override may appear to work correctly but not be completely stable in some cases.

Link to comment
  • 1 month later...
On 28/09/2017 at 6:17 AM, JWMutant said:

If you still need help with this let me know as I have spent some time getting the perfect pfsense vm running on unRAID with a fallback if unRAID craps itself for some reason.

 

I dont know about him, but id love to know how you did that. A setup like that would suit perfectly for my use case

Link to comment
  • 1 month later...
On 18/12/2017 at 9:05 PM, JWMutant said:


I’ll write up a quick guide when I finish work.


Sent from my iPhone using Tapatalk

Have you prepared that guide? It's published somewhere? 

The only thing stopping me from getting unraid is that I am not sure I will be able to setup pfsense with all the network setup right so all my traffic goes through pfsense first and then unraid or the rest of my network. I have a hyperv config right now and the Linux word is a bit unkown for me.

 

 

Link to comment
4 hours ago, L0rdRaiden said:

Have you prepared that guide? It's published somewhere? 

The only thing stopping me from getting unraid is that I am not sure I will be able to setup pfsense with all the network setup right so all my traffic goes through pfsense first and then unraid or the rest of my network. I have a hyperv config right now and the Linux word is a bit unkown for me.

 

 

setting up a pfsense VM and passing through NICs is very easy.  I've just done a screenshot of the VM setup page for a new pfsense VM.  After start, you just setup pfsense as normal and treat it like a normal machine

 

5a87878aea22a_FireShotCapture57-Highlander_AddVM_-https___1d087a25aac48109ee9a15217a.thumb.png.096f263c83e83c428c73fc1ff09a81bf.png

Link to comment
10 hours ago, DZMM said:

setting up a pfsense VM and passing through NICs is very easy.  I've just done a screenshot of the VM setup page for a new pfsense VM.  After start, you just setup pfsense as normal and treat it like a normal machine

 

5a87878aea22a_FireShotCapture57-Highlander_AddVM_-https___1d087a25aac48109ee9a15217a.thumb.png.096f263c83e83c428c73fc1ff09a81bf.png

Thanks

With the network bridge you are giving exclusive access to pfsense to one of the nic s?

How do you configure, and where the virtual switches? Is there an user interface or you have to go CLI?

Link to comment
  • 3 weeks later...

Is the video on NIC isolation still viable in Unraid 6.4.1 ?

 

Also, I've got 2 NIC with the same 'ids'

IOMMU group 11: [8086:1533] 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 
IOMMU group 12: [8086:1533] 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 

different groups though. I fear to isolate them both if I do this.

Edited by Osiris
Link to comment
3 hours ago, Osiris said:

Is the video on NIC isolation still viable in Unraid 6.4.1 ?

 

Also, I've got 2 NIC with the same 'ids'


IOMMU group 11: [8086:1533] 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 
IOMMU group 12: [8086:1533] 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 

different groups though. I fear to isolate them both if I do this.

 

Use the one below instead. Change it to your id's of course. 

 

xen-pciback.hide=(04:00.3)

 

Link to comment
12 hours ago, saarg said:

 

Use the one below instead. Change it to your id's of course. 

 


xen-pciback.hide=(04:00.3)

 


Just to clarify

I want to isolate only the second NIC of those two with the same id.

IOMMU group 11: [8086:1533] 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 
IOMMU group 12: [8086:1533] 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 

as lspci gives me

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:13.0 Non-VGA unclassified device: Intel Corporation Sunrise Point-H Integrated Sensor Hub (rev 31)
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation Sunrise Point-H Thermal subsystem (rev 31)
00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31)
00:16.1 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #2 (rev 31)
00:17.0 SATA controller: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] (rev 31)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #1 (rev f1)
00:1c.1 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #2 (rev f1)
00:1c.4 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #5 (rev f1)
00:1c.6 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #7 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
01:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
03:00.0 Non-Volatile memory controller: Device 1987:5007 (rev 01)
04:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
05:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)

should I use 

xen-pciback.hide=(02:00.0)

?

Secondly, I was wondering, isn't there a way to offer isolation via the network settings in the GUI?
Something that's only configurable when there's more than one Ethernet controller in the lspci or system devices output?

Edited by Osiris
Link to comment

A little bit confused here about what you want to do. You want to pass through one of the NICs to a VM or separate the network using vlans? This thread is about passing stuff through to a VM. 

So if you want network separation, you should move along to another thread (search Google). 

If you want to pass through only one of the NICs, then play along here :)

 

Either you have changed PCI slot for your NICs or it wasn't from you the first time as the PCI ID changed. 

Your xen-pciback.hide looks good. 

Link to comment

I want to pass it through to a VM. I've reviewed my own post (don't laugh) and don't really see where I might have been confusing about this.

I spoke about network card isolation as to be possibly offered via the unraid GUI (in order to pass it through to a VM).

 

I was just asking if the tutorial video was indeed not really applicable to my situation, as both my NICs are part of the same 'id', namely 8086:1533

So thanks for confirming that.

Edited by Osiris
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.