Bridging with multiple ETH/nic interfaces


Recommended Posts

Well, I know that the answer to my question may very well be: "turnoff bridging at the UNRAID menu and then use the go script to setup bridging by yourself", but I want to give it a try to see if I can find a way to do it within the system.

 

Setup: I have a 3 nic box

Objective: I want 1 nic to be exclusive to UNRAID, 1 nic exclusive to a VM, 1 nic to be promiscous to several VMs.

 

Current situation: if you turn on the bridging option the rc.inet1 script actually binds br0 to all ETH/nic available:

root@Tower:~# brctl show                   
bridge name	bridge id		STP enabled	interfaces
br0		8000.001517a065b6	yes		eth0
						eth1
						eth2
						vnet0
docker0		8000.56847afe9799	no		

 

I can live with bridging all nics, but in this case I need a br0->eth0, br1->eth1, br2->eth2

 

Any idea on how to achieve it, other then using the go script?

Link to comment

UnRaid 6b7 and KVM – Multi Nic Sharing

 

Goal: in a multi NIC environment, use all nics independently in a predictable manner with the best performance possible.

 

There are 4 gottas that we have to look for in setting up the system:

  • UnRaid web bridging and bonding option
  • PCI Configuration
  • MACs
  • NIC Drivers and Emulation

 

The main issue that you'll face as soon as you start configuring your system is the web GUI of UnRaid: it is very simplistic and basically it binds and bridges ALL available nics together under the same bond and/or bridge. What this means is that if you activate it you get all ETHx assigned to BR0. This is NOT what you want. What you want is for a single bridge to be created and linked to each different physical nic so that you can independently assign what you need/want to different VMs.

 

The second issue is that you ideally need to tell KVM where that nic is located on you PCI bus. This makes the system just more robust and speedier, though is a “cloud” environment binds the guest to a single host.

 

The third issue to solve is making sure that you have a predictable MAC: if you want your VM's network configuration to work reliably between reboots, you need to make sure that you present to it the same MAC every time.

 

Last problem that you want to solve is linked to the capability of the nic. Left by itself KVM will propagate to the guest a simple 100mb/s virtual nic. If you have that nic connected to a 100mb/s network it won't be a problem of course. But if you have a gigabit switch with a gigabit nic you want to make sure that you can take full advantage of the capabilities offered to you.

 

So, let's say that you have a setup like this one:

 

Generic_Network_Diagram.png

 

You want to expose a VM to the outside (some web server, mail server or, God forbid, a firewall!) and some other VM to the inside. Maybe you want to dedicate a NIC to just UnRaid, so that you can make the most of your switch's backplane capability.

 

Standard Warning: this is NOT a secure setup, it is NOT a typical 3 bastion configuration and if someone gains access to the VM that we are exposing to the Internet, then he does have access to the inside network. There is even a possibility that the actual host (UnRaid) could be broken into via the exposed nic, without the need to compromise the actual exposed guest. So be careful, take care, and don't be overconfident.

 

Setting up Bridges

How would you go about it? Well, let's start by listing the available NICs.

 

root@Tower:~# ifconfig -a
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet X.X.X.X  netmask 255.255.255.0  broadcast X.X.X.X
        ether e8:39:35:xx:xx:xx  txqueuelen 1000  (Ethernet)
        RX packets 21176163  bytes 27861232863 (25.9 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9132593  bytes 6238202394 (5.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 18  

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:15:17:xx:xx:xx txqueuelen 1000  (Ethernet)
        RX packets 6467004  bytes 501074792 (477.8 MiB)
        RX errors 0  dropped 267  overruns 0  frame 0
        TX packets 14851045  bytes 18438882164 (17.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 18  memory 0xfe8e0000-fe900000  

eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:15:17:xx:xx:xx  txqueuelen 1000  (Ethernet)
        RX packets 12946775  bytes 17396615344 (16.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7387714  bytes 586589652 (559.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 19  memory 0xfe880000-fe8a0000  

 

So, I have a three NIC setup. ETH0 is on the motherboard, the other two are on an Intel dual gigabit card. What do I want to do with this? Well, I want:

 

  • ETH0 – carry only UnRaid traffic
  • ETH1 – go to the router and expose a firewall
  • ETH2 – go to the internal switch and be used by all internal facing VMs

 

So ETH0 and ETH2 will be physically linked to the internal switch. ETH1 will be physically linked to the external router. Now that we know what we have, let's create a few bridges.

 

ETH0

No Bridge needed, will be used only by UnRaid.

 

ETH1

It is our external interface and we need to give a name to the bridge:

root@Tower:~# brctl addbr out0
root@Tower:~# brctl stp out0 on
root@Tower:~# brctl addif out0 eth1

 

ETH2

It is our internal interface and we need to give a name to the bridge:

root@Tower:~# brctl addbr in0
root@Tower:~# brctl stp in0 on
root@Tower:~# brctl addif in0 eth2

 

Now let's bring up the interfaces:

root@Tower:~# ifconfig eth1 up
root@Tower:~# ifconfig eth2 up
root@Tower:~# ifconfig out0 up
root@Tower:~# ifconfig in0 up

 

Now we have the bridges set up and running. Next, set up our KVM xml configuration file.

 

KVM Configuration

We need to configure our VM. Starting from the default configuration found here http://lime-technology.com/forum/index.php?topic=33807.0 we need to modify our file. Erase the included “interface” config.

 

Then open a web browser and go to a MAC Generator: http://www.miniwebtool.com/mac-address-generator/

 

Use 54:52:00 as MAC Address Prefix and ask to generate an address in the format of XX:XX:XX:XX:XX:XX. Get a new mac address and use it as the KVM mac address:

 

ETH1

We insert this config:

 

    <interface type='bridge'>
      <mac address='54:52:00:xx:xx:xx'/>
      <source bridge='out0'/>
      <model type='e1000'/>
    </interface>

 

ETH2

We insert this config:

 

    <interface type='bridge'>
      <mac address='54:52:00:xx:xx:xx'/>
      <source bridge='in0'/>
      <model type='e1000'/>
    </interface>

Why are we specifying the mac address? Because this is the best way to make sure that once setup the guest OS applies the same configuration between reboots.

 

Another item of interest is the “Model Type”. This determines how KVM presents the device to the guest os. If you omit it then it plays safe and presents it as an RTL card capable of 100mb/s. If you have a gigabit card, then you need to specify it as virtio or e1000 to make use of the full bandwidth available.

 

PCI Configuration

We are missing the PCI address in our configuration. I could walk you through on how to recover that information, but the easiest way to do it is to use virsh. So let's start it up (in place of {vmname} you must put the name of the vm defined in the xmlfile.xml:

 

root@Tower:~# virsh define /path/to/xmlfile.xml
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # edit {vmname}

 

You can now see that you have a configuration much more complete then the one included in the original xml file. Basically libvirt and virsh are filling in all missing information for you by inferring the best setup given your hardware. This is important because here you can see that there are PCI addresses for all configurations.

 

Scroll down to the interface tag. You should get something like this:

 

<address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>

 

This is what you need. Copy all interface tags (2 if you have assigned both of them to the VM), close your editor and quit virsh. Now change your xml file by substituting the old interface tags with the new one. Or you could dump the XML from virsh, it is up to you.

 

We are almost there!

 

Go on and boot the VM:

 

root@Tower:~# virsh undefine {vmname}
root@Tower:~# virsh create /path/to/xmlfile.xml

 

There you go, your VM now uses the multiple nics you have setup.

 

All this will be probably be obsoleted once someone gets WebVirtMgr working in a docker container. The bridging part will still apply though.

 

Ah, I almost forgot. If you want the bridges to survive unraid reboots you would have to copy the configuration to the /boot/config/go file. Just add it all there:

 

brctl addbr out0
brctl stp out0 on
brctl addif out0 eth1
brctl addbr in0
brctl stp in0 on
brctl addif in0 eth2
ifconfig eth1 up
ifconfig eth2 up
ifconfig out0 up
ifconfig in0 up

Link to comment
  • 1 year later...

Ah! I never used the novnc setting, and certainly not during install: you must have access to the console somehow, unless you are doing a silent headless install, but that's something corporations do to install defined is images that are preconfigured, something we don't do at our level.

 

But the important thing is that you got it to work. :)

Link to comment

I am setting up the bridge in the GO file

ifconfig eth0 down

brctl addbr pub

brctl stp pub yes

brctl addif pub eth0

ifconfig eth0 0.0.0.0

ifconfig pub 192.168.20.58 netmask 255.255.255.0 up

 

ifconfig eth1 down

brctl addbr priv

brctl stp priv yes

brctl addif priv eth1

ifconfig eth1 0.0.0.0

ifconfig priv 192.168.0.199 netmask 255.255.255.0 up

 

Nothing is configured in network or bridge pages in the gui

 

I can see both networks from pcs but not the VMs

 

 

The VM log has this

2015-11-27 16:36:07.205+0000: starting up libvirt version: 1.2.18, qemu version: 2.3.0

Domain id=1 is tainted: high-privileges

Domain id=1 is tainted: host-cpu

char device redirected to /dev/pts/0 (label charserial0)

 

Link to comment

 

 

I am setting up the bridge in the GO file

ifconfig eth0 down

brctl addbr pub

brctl stp pub yes

brctl addif pub eth0

ifconfig eth0 0.0.0.0

ifconfig pub 192.168.20.58 netmask 255.255.255.0 up

 

ifconfig eth1 down

brctl addbr priv

brctl stp priv yes

brctl addif priv eth1

ifconfig eth1 0.0.0.0

ifconfig priv 192.168.0.199 netmask 255.255.255.0 up

 

Nothing is configured in network or bridge pages in the gui

 

I can see both networks from pcs but not the VMs

 

 

The VM log has this

2015-11-27 16:36:07.205+0000: starting up libvirt version: 1.2.18, qemu version: 2.3.0

Domain id=1 is tainted: high-privileges

Domain id=1 is tainted: host-cpu

char device redirected to /dev/pts/0 (label charserial0)

 

Under Settings/VM Manager you needed to point that to your bridge (in order for the native novnc to work) and in the vm setup. Edit the vm and pick your bridge also.

Link to comment

Update here

I got untangle to work along with using ultravnc

It sees both networks and routes after I fix all my fat fingers and brain farts.

I has have a windows 7 computer that is a VM it works but the network card does not see any of the bridges that on the system.

I have tried all 3 of them. it shows as disconnected.

 

I also have a 2003 windows server that does not boot at all. It is not seeing the drive.

it just says booting from drive.

 

These are vmdk files from virtualbox.

 

 

Link to comment
  • 3 months later...

If you write the relative commands in the go file, then yes.

 

I figured - thanks.  Another question regarding the same train of thought.  If I have a quad port NIC & I want to split it up.. 2 ports for an unraid bond (and bridge), 1 green port for pfSense, 1 red port for PfSense.  How can I tackle this using /boot/config/network.cfg to ensure it persists after a reboot? .. Beyond what you just informed me.

Link to comment

If you write the relative commands in the go file, then yes.

 

I figured - thanks.  Another question regarding the same train of thought.  If I have a quad port NIC & I want to split it up.. 2 ports for an unraid bond (and bridge), 1 green port for pfSense, 1 red port for PfSense.  How can I tackle this using /boot/config/network.cfg to ensure it persists after a reboot? .. Beyond what you just informed me.

 

Something you may want to look into is passing the NICs directly to pfSense for better performance. If they aren't in an IOMMU group with something else then it's probably the best bet as pfSense will handle all the link up/down states.

Link to comment

If you write the relative commands in the go file, then yes.

 

I figured - thanks.  Another question regarding the same train of thought.  If I have a quad port NIC & I want to split it up.. 2 ports for an unraid bond (and bridge), 1 green port for pfSense, 1 red port for PfSense.  How can I tackle this using /boot/config/network.cfg to ensure it persists after a reboot? .. Beyond what you just informed me.

 

Something you may want to look into is passing the NICs directly to pfSense for better performance. If they aren't in an IOMMU group with something else then it's probably the best bet as pfSense will handle all the link up/down states.

 

The problem with that idea is I would have to pass the entire quad-port for passthrough.  It's an Intel NIC.  Is there any way around that?  I was hoping to not use my Realtek onboard NIC for Unraid.

Link to comment

Reference -

 

root@unraid:/boot/config# lspci

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) (rev 02)

00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU)

00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B)

00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port A)

00:0b.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (NB-SB link)

00:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890S PCI Express bridge for GPP2 port 1

00:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port B)

00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)

00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller

00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller

00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller

00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller

00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42)

00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller (rev 40)

00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40)

00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)

00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40)

00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller

00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller

00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller

00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0

00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1

00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2

00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3

00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4

00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5

01:00.0 VGA compatible controller: NVIDIA Corporation GF104 [GeForce GTX 460] (rev a1)

01:00.1 Audio device: NVIDIA Corporation GF104 High Definition Audio Controller (rev a1)

02:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 12)

03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)

04:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)

05:00.0 PCI bridge: Integrated Device Technology, Inc. [iDT] PES12N3A PCI Express Switch (rev 04)

06:00.0 PCI bridge: Integrated Device Technology, Inc. [iDT] PES12N3A PCI Express Switch (rev 04)

06:01.0 PCI bridge: Integrated Device Technology, Inc. [iDT] PES12N3A PCI Express Switch (rev 04)

07:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)

07:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)

08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)

08:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)

root@unraid:/boot/config# lspci -n

00:00.0 0600: 1002:5a14 (rev 02)

00:00.2 0806: 1002:5a23

00:02.0 0604: 1002:5a16

00:0a.0 0604: 1002:5a1d

00:0b.0 0604: 1002:5a1f

00:0c.0 0604: 1002:5a20

00:0d.0 0604: 1002:5a1e

00:11.0 0106: 1002:4391 (rev 40)

00:12.0 0c03: 1002:4397

00:12.2 0c03: 1002:4396

00:13.0 0c03: 1002:4397

00:13.2 0c03: 1002:4396

00:14.0 0c05: 1002:4385 (rev 42)

00:14.1 0101: 1002:439c (rev 40)

00:14.2 0403: 1002:4383 (rev 40)

00:14.3 0601: 1002:439d (rev 40)

00:14.4 0604: 1002:4384 (rev 40)

00:14.5 0c03: 1002:4399

00:16.0 0c03: 1002:4397

00:16.2 0c03: 1002:4396

00:18.0 0600: 1022:1600

00:18.1 0600: 1022:1601

00:18.2 0600: 1022:1602

00:18.3 0600: 1022:1603

00:18.4 0600: 1022:1604

00:18.5 0600: 1022:1605

01:00.0 0300: 10de:0e22 (rev a1)

01:00.1 0403: 10de:0beb (rev a1)

02:00.0 0106: 1b4b:9172 (rev 12)

03:00.0 0107: 1000:0072 (rev 03)

04:00.0 0107: 1000:0072 (rev 03)

05:00.0 0604: 111d:8018 (rev 04)

06:00.0 0604: 111d:8018 (rev 04)

06:01.0 0604: 111d:8018 (rev 04)

07:00.0 0200: 8086:10a4 (rev 06)

07:00.1 0200: 8086:10a4 (rev 06)

08:00.0 0200: 8086:10a4 (rev 06)

08:00.1 0200: 8086:10a4 (rev 06)

Link to comment

Am I right in thinking you have an IBM branded Intel Quad NIC? I had one passed completely through to a pfSense VM and had uPnP issues with online gaming where it would disconnect me after five minutes or so, so if uPnP is important to your network you may want to consider bare metal instead. Granted, I was using all four ports (WAN, LAN, and two additional WiFi APs). When I was using an Intel dual NIC card in a VM it was fine.

 

These are the relevant bits for passthrough:

 

07:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
07:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
08:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)

 

Try passing just 08:00.0 & 08:00.1 to the VM using this sticky. It may work... it may not.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.