VM Network performance inconsistency on 2nd 10gb NIC, but host Unraid server is perfect


Recommended Posts

Hey there,

    I setup a 10gb network the other day and everything has been working great so far, except for my Windows VM. It has been giving rather confusing results.

 

The setup:

 

NAS - Qnap TVS-951x 

  • NIC 1gb - 192.168.50.200
  • NIC 10gb - 172.254.6.200 (255.255.0.0 / no gateway)

 

Desktop PC - Pop_OS 20.10 - Gigabye Aorus Elite x570

  • Onboard 1gb NIC - 192.168.50.225
  • Asus XG-C100C  - 172.254.6.201 (255.255.0.0 / no gateway)

 

Unraid Server - MSI Gaming Plus x470

  • Onboard 1gb - 192.168.50.230
    • Windows 10 VM via 1gb Virtio-Net - 192.168.50.233 (both VM NIC show 10gb in network settings)
  • Asus XG-C100C  - 172.254.6.202 (255.255.0.0 / no gateway)
    • Windows 10 VM via 10gb Virtio - 172.254.6.204 (255.255.0.0 / no gateway)

 

(Used Virtio-Net originally for 2nd NIC, but VM was even slower, so tried Virtio and it was a bit faster)

zC5agZU.png

 

MikroTik 5 port 10gb switch for the 172.x.x.x devices

  • Shows 10GB Fullduplex across the board

 

Now here is what I am seeing. From my desktop to the NAS via iperf3 : (about expected and consistent)

Connecting to host 172.254.6.200, port 5201
[  5] local 172.254.6.201 port 57692 connected to 172.254.6.200 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   771 MBytes  6.47 Gbits/sec  1625    494 KBytes
[  5]   1.00-2.00   sec   748 MBytes  6.27 Gbits/sec  1605    358 KBytes
[  5]   2.00-3.00   sec   766 MBytes  6.43 Gbits/sec  2144    255 KBytes
[  5]   3.00-4.00   sec   782 MBytes  6.56 Gbits/sec  1892    322 KBytes
[  5]   4.00-5.00   sec   786 MBytes  6.60 Gbits/sec  1654    296 KBytes
[  5]   5.00-6.00   sec   786 MBytes  6.60 Gbits/sec  1946    320 KBytes
[  5]   6.00-7.00   sec   759 MBytes  6.36 Gbits/sec  1821    274 KBytes
[  5]   7.00-8.00   sec   779 MBytes  6.53 Gbits/sec  1850    345 KBytes
[  5]   8.00-9.00   sec   781 MBytes  6.55 Gbits/sec  1795    399 KBytes
[  5]   9.00-10.00  sec   722 MBytes  6.06 Gbits/sec  1880    264 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.50 GBytes  6.44 Gbits/sec  18212             sender
[  5]   0.00-10.00  sec  7.50 GBytes  6.44 Gbits/sec                  receiver

 

My Desktop to the Unraid server: (Beautiful, love it, consistent)

Connecting to host 172.254.6.202, port 5201
[  5] local 172.254.6.201 port 41536 connected to 172.254.6.202 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.10 GBytes  9.43 Gbits/sec    0   1.77 MBytes
[  5]   1.00-2.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.05 MBytes
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.15 MBytes
[  5]   3.00-4.00   sec  1.09 GBytes  9.41 Gbits/sec    0   2.32 MBytes
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.43 MBytes
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.43 MBytes
[  5]   6.00-7.00   sec  1.10 GBytes  9.42 Gbits/sec    5   1.07 MBytes
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.40 MBytes
[  5]   8.00-9.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.45 MBytes
[  5]   9.00-10.00  sec  1.10 GBytes  9.42 Gbits/sec    0   1.49 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.42 Gbits/sec    5             sender
[  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec                  receiver

 

--------------------------------------------------

Here is where the issues start:

 

My Desktop to the VM:

Run 1:

Desktop Side iperf client

Connecting to host 172.254.6.204, port 5201
[  5] local 172.254.6.201 port 38202 connected to 172.254.6.204 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   232 MBytes  1.95 Gbits/sec    0    220 KBytes
[  5]   1.00-2.00   sec   250 MBytes  2.10 Gbits/sec    0    220 KBytes
[  5]   2.00-3.00   sec   170 MBytes  1.43 Gbits/sec    0    220 KBytes
[  5]   3.00-4.00   sec   229 MBytes  1.92 Gbits/sec    0    220 KBytes
[  5]   4.00-5.00   sec  54.4 MBytes   456 Mbits/sec    0    220 KBytes
[  5]   5.00-6.00   sec  54.8 MBytes   460 Mbits/sec    0    220 KBytes
[  5]   6.00-7.00   sec   172 MBytes  1.45 Gbits/sec    0    220 KBytes
[  5]   7.00-8.00   sec   211 MBytes  1.77 Gbits/sec    0    220 KBytes
[  5]   8.00-9.00   sec  54.9 MBytes   460 Mbits/sec    0    220 KBytes
[  5]   9.00-10.00  sec  55.3 MBytes   464 Mbits/sec    0    220 KBytes

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.45 GBytes  1.25 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  1.45 GBytes  1.24 Gbits/sec                  receiver

 

VM running as iperf server :

Accepted connection from 172.254.6.201, port 38200
[  5] local 172.254.6.204 port 5201 connected to 172.254.6.201 port 38202
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   231 MBytes  1.94 Gbits/sec
[  5]   1.00-2.00   sec   250 MBytes  2.10 Gbits/sec
[  5]   2.00-3.00   sec   170 MBytes  1.43 Gbits/sec
[  5]   3.00-4.00   sec   229 MBytes  1.92 Gbits/sec
[  5]   4.00-5.00   sec  54.3 MBytes   455 Mbits/sec
[  5]   5.00-6.00   sec  54.5 MBytes   457 Mbits/sec
[  5]   6.00-7.00   sec   172 MBytes  1.45 Gbits/sec
[  5]   7.00-8.00   sec   211 MBytes  1.77 Gbits/sec
[  5]   8.00-9.00   sec  54.9 MBytes   460 Mbits/sec
[  5]   9.00-10.00  sec  55.0 MBytes   461 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.00  sec  1.45 GBytes  1.24 Gbits/sec                  receiver

 

------------------

Run 2:

Desktop to VM

Connecting to host 172.254.6.204, port 5201
[  5] local 172.254.6.201 port 43478 connected to 172.254.6.204 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   126 MBytes  1.05 Gbits/sec    0    217 KBytes
[  5]   1.00-2.00   sec  57.8 MBytes   485 Mbits/sec    0    217 KBytes
[  5]   2.00-3.00   sec  57.3 MBytes   481 Mbits/sec    0    217 KBytes
[  5]   3.00-4.00   sec  58.3 MBytes   489 Mbits/sec    0    217 KBytes
[  5]   4.00-5.00   sec  57.8 MBytes   485 Mbits/sec    0    217 KBytes
[  5]   5.00-6.00   sec  57.8 MBytes   485 Mbits/sec    0    217 KBytes
[  5]   6.00-7.00   sec  57.8 MBytes   485 Mbits/sec    0    217 KBytes
[  5]   7.00-8.00   sec  82.2 MBytes   690 Mbits/sec    0    217 KBytes
[  5]   8.00-9.00   sec  65.8 MBytes   552 Mbits/sec    0    217 KBytes
[  5]   9.00-10.00  sec  58.3 MBytes   489 Mbits/sec    0    217 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   679 MBytes   569 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   678 MBytes   568 Mbits/sec                  receiver

 

VM Side:

Accepted connection from 172.254.6.201, port 43476
[  5] local 172.254.6.204 port 5201 connected to 172.254.6.201 port 43478
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   125 MBytes  1.04 Gbits/sec
[  5]   1.00-2.00   sec  57.5 MBytes   483 Mbits/sec
[  5]   2.00-3.00   sec  57.7 MBytes   484 Mbits/sec
[  5]   3.00-4.00   sec  57.9 MBytes   486 Mbits/sec
[  5]   4.00-5.00   sec  58.2 MBytes   488 Mbits/sec
[  5]   5.00-6.00   sec  57.7 MBytes   484 Mbits/sec
[  5]   6.00-7.00   sec  57.9 MBytes   486 Mbits/sec
[  5]   7.00-8.00   sec  81.9 MBytes   687 Mbits/sec
[  5]   8.00-9.00   sec  65.9 MBytes   553 Mbits/sec
[  5]   9.00-10.00  sec  58.3 MBytes   489 Mbits/sec
[  5]  10.00-10.00  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.00  sec   678 MBytes   568 Mbits/sec                  receiver

 

--------------------------------------------

Now, the other way around, initiating the client from the VM to my Desktop PC:

Connecting to host 172.254.6.201, port 5201
[  4] local 172.254.6.204 port 60680 connected to 172.254.6.201 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   112 MBytes   935 Mbits/sec
[  4]   1.00-2.00   sec   113 MBytes   947 Mbits/sec
[  4]   2.00-3.00   sec   113 MBytes   945 Mbits/sec
[  4]   3.00-4.00   sec   113 MBytes   949 Mbits/sec
[  4]   4.00-5.00   sec   113 MBytes   948 Mbits/sec
[  4]   5.00-6.00   sec   113 MBytes   945 Mbits/sec
[  4]   6.00-7.00   sec   112 MBytes   944 Mbits/sec
[  4]   7.00-8.00   sec   113 MBytes   945 Mbits/sec
[  4]   8.00-9.00   sec   113 MBytes   948 Mbits/sec
[  4]   9.00-10.00  sec   113 MBytes   948 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.10 GBytes   946 Mbits/sec                  sender
[  4]   0.00-10.00  sec  1.10 GBytes   946 Mbits/sec                  receiver

 

Then from the VM to NAS:

Connecting to host 172.254.6.200, port 5201
[  4] local 172.254.6.204 port 60640 connected to 172.254.6.200 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   317 MBytes  2.66 Gbits/sec
[  4]   1.00-2.00   sec   358 MBytes  3.01 Gbits/sec
[  4]   2.00-3.00   sec   359 MBytes  3.01 Gbits/sec
[  4]   3.00-4.00   sec   371 MBytes  3.12 Gbits/sec
[  4]   4.00-5.00   sec   378 MBytes  3.17 Gbits/sec
[  4]   5.00-6.00   sec   381 MBytes  3.19 Gbits/sec
[  4]   6.00-7.00   sec   385 MBytes  3.23 Gbits/sec
[  4]   7.00-8.00   sec   393 MBytes  3.29 Gbits/sec
[  4]   8.00-9.00   sec   387 MBytes  3.25 Gbits/sec
[  4]   9.00-10.00  sec   389 MBytes  3.27 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  3.63 GBytes  3.12 Gbits/sec                  sender
[  4]   0.00-10.00  sec  3.63 GBytes  3.12 Gbits/sec                  receiver

 

 

Then the Unraid Server itself to the NAS:

Connecting to host 172.254.6.200, port 5201
[  5] local 172.254.6.202 port 51496 connected to 172.254.6.200 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   741 MBytes  6.22 Gbits/sec  244    305 KBytes
[  5]   1.00-2.00   sec   774 MBytes  6.49 Gbits/sec    0    297 KBytes
[  5]   2.00-3.00   sec   772 MBytes  6.48 Gbits/sec    0    308 KBytes
[  5]   3.00-4.00   sec   726 MBytes  6.09 Gbits/sec  505    342 KBytes
[  5]   4.00-5.00   sec   770 MBytes  6.46 Gbits/sec   14    328 KBytes
[  5]   5.00-6.00   sec   712 MBytes  5.98 Gbits/sec  4140    342 KBytes
[  5]   6.00-7.00   sec   702 MBytes  5.89 Gbits/sec  1359    308 KBytes
[  5]   7.00-8.00   sec   742 MBytes  6.23 Gbits/sec   57    305 KBytes
[  5]   8.00-9.00   sec   781 MBytes  6.55 Gbits/sec    0    303 KBytes
[  5]   9.00-10.00  sec   785 MBytes  6.58 Gbits/sec    0    303 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.33 GBytes  6.30 Gbits/sec  6319             sender
[  5]   0.00-10.00  sec  7.32 GBytes  6.29 Gbits/sec                  receiver

 

 

----

 

So, all in all, it just seems like the VM is having issues, does anyone have any recommendations of things I might look into or try?

Edited by MostHated
Link to comment
Posted (edited)

The 1gb network is used for internet access. The 10gb network has no outside physical connection. The switch is only connected to the three devices, which are using static IP's with no gateway or dns.

 

8IZDoWZ.png

Edited by MostHated
Link to comment

I know that, but some how traffic may route thr between both subnet in Unraid or NAS ....

 

And you said Windows attach br0 or br1 but all test under 172.254.x.x.

 

You don't need change any setting, just before start iperf test, temporary disconnect all 1G link, there are no hurt.

Edited by Vr2Io
Link to comment
Posted (edited)

I am not quite sure I follow. The VM does have both br0 (the 192.x.x.x) and br1  (the 172.x.x.x) added. All tests were performed using 172.x.x.x, but that is because that is what I typed in, in order to run the test through the 10gb connection.

 

(200-204 whichever I wanted to test)

iperf3 -c 172.254.6.200 

 

That said, I can't remove the 1gb, otherwise there would be no internet connection. I don't have any static routes setup within the switch that would allow traffic from the 192.x.x.x subnet. Is there some sort of network segregation that I might need to perform within Unraid itself?

Edited by MostHated
Link to comment

This is temporary for test, not disconnect forever.

 

Best should be use one subnet only, one solution is use a 10G sfp+ to 1G RJ45 to connect both 10G and 1G switch, then all client use 1 NIC only.

Edited by Vr2Io
Link to comment

I understand, I can give that a try. My hope was to keep the 10gb for connection between the three devices only, but, I suppose it would not be the end of the world if it only had the one connection. The switch has 4 SPF rj45 (currently using 3) but also has 1 built-in 1gb rj45 which can be used to connect it to the network.

 

If nothing else, as you mentioned, it would be worth it just to see if it helps and narrow things down.

Link to comment
15 minutes ago, MostHated said:

narrow things down.

Yes

 

I misunderstand 5port 10G switch was five 10G sfp+ port, it is four 10G + one 1G.

 

Why don't design in 1 subnet 1 NIC ?

Edited by Vr2Io
Link to comment

The original idea was primarily for access control and maximum throughput. The 172 network has my NAS and all of my development projects, and also my home automation server and devices, it was just how I envisioned the setup being when I started putting it together. I suppose though, that giving a Windows VM access to both 192.x and 172.x at the same time negates any security in the first place.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.