Networking question: how to run dual bridges / dual NICs properly?

[testdasi]

My motherboard has 2 NICs so I have this grand idea of using both.

• 1 for my main workstation VM and Unraid stuff that connects directly to the Internet
• 1 for everything else (that, if things work, will be behind a VPN)

 

At first I thought it's as simple as setting up 2 bridges. That sort of works with a massive problem.

When I try to do intra-bridge transfer (e.g. accessing an Unraid share from my VM), I'm limited to 1Gbps. That suggests perhaps things go to the router and back (hence limited to the physical gigabit connection instead of the emulated 10Gb connection.

 

The limit would only go away if I physically disconnect the 2nd connection. It's actually not just limited to having 2 bridges. If the 2nd NIC is connected (i.e. have an IP address), everything is limited to gigabit.

 

Does anyone know how to get around this?

I'm not married to the idea of running 2 bridges, just some ways to utilize both of my NICs.

[ken-ji]

Can you elaborate your two bridge setup? I don't quite get what you meant by intra bridge trasnfer between your VM and Unraid share.

I did some testing on my setup and it seems the max I can get out of a Linux VM (with VirtIO network) doing SCP against Unraid is 180MB/s

A windows VM via SMB shares maxes out at 110MB/s, but these are all on the same bridge (My containers all live on a different bridge and VLAN)

[testdasi]

1 hour ago, ken-ji said:

Can you elaborate your two bridge setup? I don't quite get what you meant by intra bridge trasnfer between your VM and Unraid share.

I did some testing on my setup and it seems the max I can get out of a Linux VM (with VirtIO network) doing SCP against Unraid is 180MB/s

A windows VM via SMB shares maxes out at 110MB/s, but these are all on the same bridge (My containers all live on a different bridge and VLAN)

So on the network page, there are 2 sections for eth0 and eth1. For each section, I select Bonding = No, Bridge = Yes and in the device list of eth0 I put eth0 and similar eth1 in eth1 list. IPs are all automatic DHCP for now (I set static IP on my router DHCP server).

So that creates br0 which has eth0 and br1 which has eth1.

 

Intra-bridge = 2 devices within the same bridge. My Unraid server and my VM are connected to the router using the same cable, out of the same eth0 port under the same br0 bridge. Hence the connection between them ought to be at 10Gb i.e. the speed of the virtual network device.

 

From what you said, it sounds like an SMB limitation.  Sounds like via Linux you can get full speed (in your case 180MB/s, my case 500MB/s due to SSD) but via Windows SMB, it's limited to max cable speed i.e. gigabit (115MB/s or so) if there are multiple bridges.  

[ken-ji]

Oh, then we do have a similar config, just that my 2nd Bridge does not have an IPv4 address to properly allow Containers on the same VLAN as Unraid to talk to Unraid. Interesting that we do get the 1Gbps limit then on a Windows VM talking to unRAID on the same bridge. But unplugging a network should not have any effect. Also my Linux test was actually RAM disk (Alpine Linux ISO) to RAM disk (Unrsaid /tmp).

[Alphahelix]

could it be (it is just a thought) that editing the XML file, under the NIC setting, to vmxnet3 ( 

 ) will do the trick? It made my pfSense go from 1gbit to 10gbit on the virtual interface.

 

/Alphehelix

Edited June 28, 2019 by Alphahelix