Garani Posted August 27, 2014 Share Posted August 27, 2014 Well, I know that the answer to my question may very well be: "turnoff bridging at the UNRAID menu and then use the go script to setup bridging by yourself", but I want to give it a try to see if I can find a way to do it within the system. Setup: I have a 3 nic box Objective: I want 1 nic to be exclusive to UNRAID, 1 nic exclusive to a VM, 1 nic to be promiscous to several VMs. Current situation: if you turn on the bridging option the rc.inet1 script actually binds br0 to all ETH/nic available: root@Tower:~# brctl show bridge name bridge id STP enabled interfaces br0 8000.001517a065b6 yes eth0 eth1 eth2 vnet0 docker0 8000.56847afe9799 no I can live with bridging all nics, but in this case I need a br0->eth0, br1->eth1, br2->eth2 Any idea on how to achieve it, other then using the go script? Quote Link to comment
Garani Posted September 1, 2014 Author Share Posted September 1, 2014 Well, since I got no reply, I went ahead and wrote my own bridging commands in go. Quote Link to comment
dmacias Posted September 1, 2014 Share Posted September 1, 2014 Well, since I got no reply, I went ahead and wrote my own bridging commands in go. Please share your go commands . Did you get it working like you wanted? I have 2 nic's and would like to do something similar. Quote Link to comment
Garani Posted September 1, 2014 Author Share Posted September 1, 2014 It works beautifully after getting a few quirks out of the way. I'll write a mini how-to step by step on making interfaces working with bridging, PCI bus, driver emulation and so on. Quote Link to comment
Garani Posted September 2, 2014 Author Share Posted September 2, 2014 UnRaid 6b7 and KVM – Multi Nic Sharing Goal: in a multi NIC environment, use all nics independently in a predictable manner with the best performance possible. There are 4 gottas that we have to look for in setting up the system: UnRaid web bridging and bonding option PCI Configuration MACs NIC Drivers and Emulation The main issue that you'll face as soon as you start configuring your system is the web GUI of UnRaid: it is very simplistic and basically it binds and bridges ALL available nics together under the same bond and/or bridge. What this means is that if you activate it you get all ETHx assigned to BR0. This is NOT what you want. What you want is for a single bridge to be created and linked to each different physical nic so that you can independently assign what you need/want to different VMs. The second issue is that you ideally need to tell KVM where that nic is located on you PCI bus. This makes the system just more robust and speedier, though is a “cloud” environment binds the guest to a single host. The third issue to solve is making sure that you have a predictable MAC: if you want your VM's network configuration to work reliably between reboots, you need to make sure that you present to it the same MAC every time. Last problem that you want to solve is linked to the capability of the nic. Left by itself KVM will propagate to the guest a simple 100mb/s virtual nic. If you have that nic connected to a 100mb/s network it won't be a problem of course. But if you have a gigabit switch with a gigabit nic you want to make sure that you can take full advantage of the capabilities offered to you. So, let's say that you have a setup like this one: You want to expose a VM to the outside (some web server, mail server or, God forbid, a firewall!) and some other VM to the inside. Maybe you want to dedicate a NIC to just UnRaid, so that you can make the most of your switch's backplane capability. Standard Warning: this is NOT a secure setup, it is NOT a typical 3 bastion configuration and if someone gains access to the VM that we are exposing to the Internet, then he does have access to the inside network. There is even a possibility that the actual host (UnRaid) could be broken into via the exposed nic, without the need to compromise the actual exposed guest. So be careful, take care, and don't be overconfident. Setting up Bridges How would you go about it? Well, let's start by listing the available NICs. root@Tower:~# ifconfig -a eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet X.X.X.X netmask 255.255.255.0 broadcast X.X.X.X ether e8:39:35:xx:xx:xx txqueuelen 1000 (Ethernet) RX packets 21176163 bytes 27861232863 (25.9 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 9132593 bytes 6238202394 (5.8 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 18 eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:15:17:xx:xx:xx txqueuelen 1000 (Ethernet) RX packets 6467004 bytes 501074792 (477.8 MiB) RX errors 0 dropped 267 overruns 0 frame 0 TX packets 14851045 bytes 18438882164 (17.1 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 18 memory 0xfe8e0000-fe900000 eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:15:17:xx:xx:xx txqueuelen 1000 (Ethernet) RX packets 12946775 bytes 17396615344 (16.2 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 7387714 bytes 586589652 (559.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 19 memory 0xfe880000-fe8a0000 So, I have a three NIC setup. ETH0 is on the motherboard, the other two are on an Intel dual gigabit card. What do I want to do with this? Well, I want: ETH0 – carry only UnRaid traffic ETH1 – go to the router and expose a firewall ETH2 – go to the internal switch and be used by all internal facing VMs So ETH0 and ETH2 will be physically linked to the internal switch. ETH1 will be physically linked to the external router. Now that we know what we have, let's create a few bridges. ETH0 No Bridge needed, will be used only by UnRaid. ETH1 It is our external interface and we need to give a name to the bridge: root@Tower:~# brctl addbr out0 root@Tower:~# brctl stp out0 on root@Tower:~# brctl addif out0 eth1 ETH2 It is our internal interface and we need to give a name to the bridge: root@Tower:~# brctl addbr in0 root@Tower:~# brctl stp in0 on root@Tower:~# brctl addif in0 eth2 Now let's bring up the interfaces: root@Tower:~# ifconfig eth1 up root@Tower:~# ifconfig eth2 up root@Tower:~# ifconfig out0 up root@Tower:~# ifconfig in0 up Now we have the bridges set up and running. Next, set up our KVM xml configuration file. KVM Configuration We need to configure our VM. Starting from the default configuration found here http://lime-technology.com/forum/index.php?topic=33807.0 we need to modify our file. Erase the included “interface” config. Then open a web browser and go to a MAC Generator: http://www.miniwebtool.com/mac-address-generator/ Use 54:52:00 as MAC Address Prefix and ask to generate an address in the format of XX:XX:XX:XX:XX:XX. Get a new mac address and use it as the KVM mac address: ETH1 We insert this config: <interface type='bridge'> <mac address='54:52:00:xx:xx:xx'/> <source bridge='out0'/> <model type='e1000'/> </interface> ETH2 We insert this config: <interface type='bridge'> <mac address='54:52:00:xx:xx:xx'/> <source bridge='in0'/> <model type='e1000'/> </interface> Why are we specifying the mac address? Because this is the best way to make sure that once setup the guest OS applies the same configuration between reboots. Another item of interest is the “Model Type”. This determines how KVM presents the device to the guest os. If you omit it then it plays safe and presents it as an RTL card capable of 100mb/s. If you have a gigabit card, then you need to specify it as virtio or e1000 to make use of the full bandwidth available. PCI Configuration We are missing the PCI address in our configuration. I could walk you through on how to recover that information, but the easiest way to do it is to use virsh. So let's start it up (in place of {vmname} you must put the name of the vm defined in the xmlfile.xml: root@Tower:~# virsh define /path/to/xmlfile.xml Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # edit {vmname} You can now see that you have a configuration much more complete then the one included in the original xml file. Basically libvirt and virsh are filling in all missing information for you by inferring the best setup given your hardware. This is important because here you can see that there are PCI addresses for all configurations. Scroll down to the interface tag. You should get something like this: <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> This is what you need. Copy all interface tags (2 if you have assigned both of them to the VM), close your editor and quit virsh. Now change your xml file by substituting the old interface tags with the new one. Or you could dump the XML from virsh, it is up to you. We are almost there! Go on and boot the VM: root@Tower:~# virsh undefine {vmname} root@Tower:~# virsh create /path/to/xmlfile.xml There you go, your VM now uses the multiple nics you have setup. All this will be probably be obsoleted once someone gets WebVirtMgr working in a docker container. The bridging part will still apply though. Ah, I almost forgot. If you want the bridges to survive unraid reboots you would have to copy the configuration to the /boot/config/go file. Just add it all there: brctl addbr out0 brctl stp out0 on brctl addif out0 eth1 brctl addbr in0 brctl stp in0 on brctl addif in0 eth2 ifconfig eth1 up ifconfig eth2 up ifconfig out0 up ifconfig in0 up Quote Link to comment
jpimlott Posted November 26, 2015 Share Posted November 26, 2015 I am trying the same thing and my interfaces no longer become accessible. I tried it the way is was showed there john Quote Link to comment
Garani Posted November 26, 2015 Author Share Posted November 26, 2015 A couple of things changed since 6.0 went live. 1) you do not have to fiddle with PCI, the current web interface does that for you. Do don't do anything there. 2) in the network settings of unraid, did you turn on or off the bridging directive? Quote Link to comment
jpimlott Posted November 26, 2015 Share Posted November 26, 2015 I have bridge turned off in unraid config. I followed a set from http://www.tldp.org/HOWTO/BRIDGE-STP-HOWTO/set-up-the-bridge.html Now the interfaces are up but I can access the VMs from vnc to finish the config for rdp. John Quote Link to comment
Garani Posted November 26, 2015 Author Share Posted November 26, 2015 I am sorry, but I didn't quite catch. Your interfaces are up and assigned to the vm? And yes, you are always supposed to use Vnc to install a vm. After you have configured it you can access it via the network with a terminal, Vnc, Xserver or rdp (if windows) Quote Link to comment
jpimlott Posted November 26, 2015 Share Posted November 26, 2015 The noVNC fails when launched. I can get the server from both interfaces with different pcs for both the web site or smb. Novnc worked before I made the changes to the bridge settings. I have not turned on the bridge in the unraid gui. Quote Link to comment
Garani Posted November 26, 2015 Author Share Posted November 26, 2015 Ah! I never used the novnc setting, and certainly not during install: you must have access to the console somehow, unless you are doing a silent headless install, but that's something corporations do to install defined is images that are preconfigured, something we don't do at our level. But the important thing is that you got it to work. Quote Link to comment
jpimlott Posted November 27, 2015 Share Posted November 27, 2015 NoVNC is the default viewer web based that comes with unraid. I have also tried vncviewer but no luck there either How else can I connect to console of the VMs? Quote Link to comment
Garani Posted November 27, 2015 Author Share Posted November 27, 2015 Well, I use the web Vnc console, but it is very sporadic that I use it, since I run Linux boxes on the hypervisor. Quote Link to comment
dmacias Posted November 27, 2015 Share Posted November 27, 2015 NoVNC is the default viewer web based that comes with unraid. I have also tried vncviewer but no luck there either How else can I connect to console of the VMs? What bridge do you use for your vm's? And did you set that bridge under Settings/VM Manager? Quote Link to comment
jpimlott Posted November 27, 2015 Share Posted November 27, 2015 I am setting up the bridge in the GO file ifconfig eth0 down brctl addbr pub brctl stp pub yes brctl addif pub eth0 ifconfig eth0 0.0.0.0 ifconfig pub 192.168.20.58 netmask 255.255.255.0 up ifconfig eth1 down brctl addbr priv brctl stp priv yes brctl addif priv eth1 ifconfig eth1 0.0.0.0 ifconfig priv 192.168.0.199 netmask 255.255.255.0 up Nothing is configured in network or bridge pages in the gui I can see both networks from pcs but not the VMs The VM log has this 2015-11-27 16:36:07.205+0000: starting up libvirt version: 1.2.18, qemu version: 2.3.0 Domain id=1 is tainted: high-privileges Domain id=1 is tainted: host-cpu char device redirected to /dev/pts/0 (label charserial0) Quote Link to comment
dmacias Posted November 27, 2015 Share Posted November 27, 2015 I am setting up the bridge in the GO file ifconfig eth0 down brctl addbr pub brctl stp pub yes brctl addif pub eth0 ifconfig eth0 0.0.0.0 ifconfig pub 192.168.20.58 netmask 255.255.255.0 up ifconfig eth1 down brctl addbr priv brctl stp priv yes brctl addif priv eth1 ifconfig eth1 0.0.0.0 ifconfig priv 192.168.0.199 netmask 255.255.255.0 up Nothing is configured in network or bridge pages in the gui I can see both networks from pcs but not the VMs The VM log has this 2015-11-27 16:36:07.205+0000: starting up libvirt version: 1.2.18, qemu version: 2.3.0 Domain id=1 is tainted: high-privileges Domain id=1 is tainted: host-cpu char device redirected to /dev/pts/0 (label charserial0) Under Settings/VM Manager you needed to point that to your bridge (in order for the native novnc to work) and in the vm setup. Edit the vm and pick your bridge also. Quote Link to comment
Garani Posted November 27, 2015 Author Share Posted November 27, 2015 Tonight i'll do a couple of snapshots. Hang on tight Quote Link to comment
jpimlott Posted November 28, 2015 Share Posted November 28, 2015 Update here I got untangle to work along with using ultravnc It sees both networks and routes after I fix all my fat fingers and brain farts. I has have a windows 7 computer that is a VM it works but the network card does not see any of the bridges that on the system. I have tried all 3 of them. it shows as disconnected. I also have a 2003 windows server that does not boot at all. It is not seeing the drive. it just says booting from drive. These are vmdk files from virtualbox. Quote Link to comment
jnheinz Posted February 29, 2016 Share Posted February 29, 2016 Do these bridges & configurations persist a reboot? Quote Link to comment
Garani Posted February 29, 2016 Author Share Posted February 29, 2016 If you write the relative commands in the go file, then yes. Quote Link to comment
jnheinz Posted February 29, 2016 Share Posted February 29, 2016 If you write the relative commands in the go file, then yes. I figured - thanks. Another question regarding the same train of thought. If I have a quad port NIC & I want to split it up.. 2 ports for an unraid bond (and bridge), 1 green port for pfSense, 1 red port for PfSense. How can I tackle this using /boot/config/network.cfg to ensure it persists after a reboot? .. Beyond what you just informed me. Quote Link to comment
METDeath Posted February 29, 2016 Share Posted February 29, 2016 If you write the relative commands in the go file, then yes. I figured - thanks. Another question regarding the same train of thought. If I have a quad port NIC & I want to split it up.. 2 ports for an unraid bond (and bridge), 1 green port for pfSense, 1 red port for PfSense. How can I tackle this using /boot/config/network.cfg to ensure it persists after a reboot? .. Beyond what you just informed me. Something you may want to look into is passing the NICs directly to pfSense for better performance. If they aren't in an IOMMU group with something else then it's probably the best bet as pfSense will handle all the link up/down states. Quote Link to comment
jnheinz Posted February 29, 2016 Share Posted February 29, 2016 If you write the relative commands in the go file, then yes. I figured - thanks. Another question regarding the same train of thought. If I have a quad port NIC & I want to split it up.. 2 ports for an unraid bond (and bridge), 1 green port for pfSense, 1 red port for PfSense. How can I tackle this using /boot/config/network.cfg to ensure it persists after a reboot? .. Beyond what you just informed me. Something you may want to look into is passing the NICs directly to pfSense for better performance. If they aren't in an IOMMU group with something else then it's probably the best bet as pfSense will handle all the link up/down states. The problem with that idea is I would have to pass the entire quad-port for passthrough. It's an Intel NIC. Is there any way around that? I was hoping to not use my Realtek onboard NIC for Unraid. Quote Link to comment
jnheinz Posted February 29, 2016 Share Posted February 29, 2016 Reference - root@unraid:/boot/config# lspci 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) (rev 02) 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU) 00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B) 00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port A) 00:0b.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (NB-SB link) 00:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890S PCI Express bridge for GPP2 port 1 00:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port B) 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40) 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42) 00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40) 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40) 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5 01:00.0 VGA compatible controller: NVIDIA Corporation GF104 [GeForce GTX 460] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GF104 High Definition Audio Controller (rev a1) 02:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 12) 03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) 04:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) 05:00.0 PCI bridge: Integrated Device Technology, Inc. [iDT] PES12N3A PCI Express Switch (rev 04) 06:00.0 PCI bridge: Integrated Device Technology, Inc. [iDT] PES12N3A PCI Express Switch (rev 04) 06:01.0 PCI bridge: Integrated Device Technology, Inc. [iDT] PES12N3A PCI Express Switch (rev 04) 07:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) 07:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) 08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) 08:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) root@unraid:/boot/config# lspci -n 00:00.0 0600: 1002:5a14 (rev 02) 00:00.2 0806: 1002:5a23 00:02.0 0604: 1002:5a16 00:0a.0 0604: 1002:5a1d 00:0b.0 0604: 1002:5a1f 00:0c.0 0604: 1002:5a20 00:0d.0 0604: 1002:5a1e 00:11.0 0106: 1002:4391 (rev 40) 00:12.0 0c03: 1002:4397 00:12.2 0c03: 1002:4396 00:13.0 0c03: 1002:4397 00:13.2 0c03: 1002:4396 00:14.0 0c05: 1002:4385 (rev 42) 00:14.1 0101: 1002:439c (rev 40) 00:14.2 0403: 1002:4383 (rev 40) 00:14.3 0601: 1002:439d (rev 40) 00:14.4 0604: 1002:4384 (rev 40) 00:14.5 0c03: 1002:4399 00:16.0 0c03: 1002:4397 00:16.2 0c03: 1002:4396 00:18.0 0600: 1022:1600 00:18.1 0600: 1022:1601 00:18.2 0600: 1022:1602 00:18.3 0600: 1022:1603 00:18.4 0600: 1022:1604 00:18.5 0600: 1022:1605 01:00.0 0300: 10de:0e22 (rev a1) 01:00.1 0403: 10de:0beb (rev a1) 02:00.0 0106: 1b4b:9172 (rev 12) 03:00.0 0107: 1000:0072 (rev 03) 04:00.0 0107: 1000:0072 (rev 03) 05:00.0 0604: 111d:8018 (rev 04) 06:00.0 0604: 111d:8018 (rev 04) 06:01.0 0604: 111d:8018 (rev 04) 07:00.0 0200: 8086:10a4 (rev 06) 07:00.1 0200: 8086:10a4 (rev 06) 08:00.0 0200: 8086:10a4 (rev 06) 08:00.1 0200: 8086:10a4 (rev 06) Quote Link to comment
METDeath Posted March 1, 2016 Share Posted March 1, 2016 Am I right in thinking you have an IBM branded Intel Quad NIC? I had one passed completely through to a pfSense VM and had uPnP issues with online gaming where it would disconnect me after five minutes or so, so if uPnP is important to your network you may want to consider bare metal instead. Granted, I was using all four ports (WAN, LAN, and two additional WiFi APs). When I was using an Intel dual NIC card in a VM it was fine. These are the relevant bits for passthrough: 07:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) 07:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) 08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) 08:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) Try passing just 08:00.0 & 08:00.1 to the VM using this sticky. It may work... it may not. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.