Jump to content
joshstrange

10x Network speed drop in VM after initial surge

7 posts in this topic Last Reply

Recommended Posts

Posted (edited)

I have been thinking for years that my German host was throttling me or it was my ISP but due to a completely separate issue I found something pretty damning. I ran iperf3 out on my German host (iperf3 -s) and then ran the client from my MacBook, my unraid machine, and the VM in my unraid machine. My MBP and Unraid host performed the same so I'll just post one.

 

Here is the command I used: 

iperf3 -c <IP-OF-GERMAN-SERVER> -ub 1G -R

 

Host:

Connecting to host <REMOVED>, port 5201
Reverse mode, remote host <REMOVED> is sending
[  4] local <REMOVED> port 38896 connected to <REMOVED> port 5201
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-1.00   sec  46.6 MBytes   391 Mbits/sec  0.027 ms  3655/37428 (9.8%)
[  4]   1.00-2.00   sec  54.0 MBytes   453 Mbits/sec  0.027 ms  2758/41888 (6.6%)
[  4]   2.00-3.00   sec  58.2 MBytes   488 Mbits/sec  0.031 ms  1/42115 (0.0024%)
[  4]   3.00-4.00   sec  54.3 MBytes   456 Mbits/sec  0.035 ms  2323/41664 (5.6%)
[  4]   4.00-5.00   sec  57.2 MBytes   480 Mbits/sec  0.035 ms  15769/57181 (28%)
[  4]   5.00-6.00   sec  55.6 MBytes   467 Mbits/sec  0.043 ms  40686/80979 (50%)
[  4]   6.00-7.00   sec  56.9 MBytes   477 Mbits/sec  0.044 ms  40272/81477 (49%)
[  4]   7.00-8.00   sec  54.4 MBytes   456 Mbits/sec  0.044 ms  43713/83089 (53%)
[  4]   8.00-9.00   sec  56.8 MBytes   476 Mbits/sec  0.032 ms  41376/82486 (50%)
[  4]   9.00-10.00  sec  54.9 MBytes   460 Mbits/sec  0.035 ms  43178/82930 (52%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec  1.12 GBytes   960 Mbits/sec  0.042 ms  333027/828960 (40%)
[  4] Sent 828960 datagrams

iperf Done.

 

VM: 

 

Connecting to host <REMOVED>, port 5201
Reverse mode, remote host <REMOVED> is sending
[  4] local <REMOVED> port 52061 connected to <REMOVED> port 5201
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-1.00   sec  37.2 MBytes   312 Mbits/sec  0.141 ms  4022/8789 (46%)
[  4]   1.00-2.00   sec  40.8 MBytes   342 Mbits/sec  0.130 ms  3944/9163 (43%)
[  4]   2.00-3.00   sec  41.1 MBytes   345 Mbits/sec  0.217 ms  4276/9539 (45%)
[  4]   3.00-4.00   sec  40.1 MBytes   336 Mbits/sec  0.109 ms  4021/9154 (44%)
[  4]   4.00-5.00   sec  44.5 MBytes   373 Mbits/sec  0.122 ms  3001/8693 (35%)
[  4]   5.00-6.00   sec  28.4 MBytes   238 Mbits/sec  0.225 ms  6518/10157 (64%)
[  4]   6.00-7.00   sec  3.80 MBytes  31.9 Mbits/sec  0.389 ms  13215/13701 (96%)
[  4]   7.00-8.00   sec  2.62 MBytes  22.0 Mbits/sec  0.271 ms  14396/14731 (98%)
[  4]   8.00-9.00   sec  3.51 MBytes  29.4 Mbits/sec  0.184 ms  14265/14714 (97%)
[  4]   9.00-10.00  sec  3.35 MBytes  28.1 Mbits/sec  0.304 ms  13905/14334 (97%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec  1.13 GBytes   969 Mbits/sec  0.253 ms  115659/147876 (78%)
[  4] Sent 147876 datagrams

iperf Done.

 

 

As you can see they both start out strong (the host is stronger but still 300Mb/s is not terrible) but then the VM takes a 10x nosedive. I've seen similar things using rsync on the VM countless times and wrote it off.

 

What could cause this? I setup this VM back when I first installed Unraid years ago and its network bridge in the VM settings is set to br0.

Edited by joshstrange
it's -> its

Share this post


Link to post

Might help to know what kind of os the vm is running and what virtual nic you are using.

Share this post


Link to post

I am running Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-23-generic x86_64) in the VM and I'm using Br0 network bridge.

Share this post


Link to post

still didn't list the virtual Nic, only the bridge it uses for network access. I don't use virtualized ubuntu (but have on bare metal) anyways,  here's a test you can try to rule out if it's the virtual Nic: in your xml, it will tell you what virtual Nic you're using ( probably virtio)

 

<interface type='bridge'>
      <mac address='52:52:01:4j:66:74'/>
      <source bridge='br0'/>
      <model type='virtio'/>

 

try changing to e1000-82545em or vmxnet3 and retest. I don't know off the top of my head if ubuntu has drivers for those preloaded, so it may not work. OR you may need to add another network interface with one of those virtual nics while keeping the "sort of working" one, then do a hardware update for the drivers to download.

Share this post


Link to post

I'm sorry, yes I was using virtio. 

 

Here are my results with the two you mentioned:

 

e1000-82545em:

Connecting to host <removed>, port 5201
Reverse mode, remote host <removed> is sending
[  4] local <removed> port 43176 connected to <removed> port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.47 MBytes  12.3 Mbits/sec
[  4]   1.00-2.00   sec  4.02 MBytes  33.7 Mbits/sec
[  4]   2.00-3.00   sec  4.59 MBytes  38.5 Mbits/sec
[  4]   3.00-4.00   sec  4.80 MBytes  40.3 Mbits/sec
[  4]   4.00-5.00   sec  5.08 MBytes  42.6 Mbits/sec
[  4]   5.00-6.00   sec  4.39 MBytes  36.8 Mbits/sec
[  4]   6.00-7.00   sec  4.15 MBytes  34.8 Mbits/sec
[  4]   7.00-8.00   sec  4.47 MBytes  37.5 Mbits/sec
[  4]   8.00-9.00   sec  4.35 MBytes  36.5 Mbits/sec
[  4]   9.00-10.00  sec  4.45 MBytes  37.4 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  42.7 MBytes  35.8 Mbits/sec  102             sender
[  4]   0.00-10.00  sec  42.5 MBytes  35.7 Mbits/sec                  receiver

iperf Done.

 

vmxnet3:

Connecting to host <removed>, port 5201
Reverse mode, remote host <removed> is sending
[  4] local <removed> port 37254 connected to <removed> port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.66 MBytes  13.9 Mbits/sec
[  4]   1.00-2.00   sec  6.58 MBytes  55.2 Mbits/sec
[  4]   2.00-3.00   sec  6.60 MBytes  55.3 Mbits/sec
[  4]   3.00-4.00   sec  7.07 MBytes  59.3 Mbits/sec
[  4]   4.00-5.00   sec  7.50 MBytes  62.9 Mbits/sec
[  4]   5.00-6.00   sec  7.59 MBytes  63.7 Mbits/sec
[  4]   6.00-7.00   sec  7.16 MBytes  60.1 Mbits/sec
[  4]   7.00-8.00   sec  8.00 MBytes  67.1 Mbits/sec
[  4]   8.00-9.00   sec  5.04 MBytes  42.3 Mbits/sec
[  4]   9.00-10.00  sec  5.68 MBytes  47.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  63.8 MBytes  53.5 Mbits/sec   59             sender
[  4]   0.00-10.00  sec  63.8 MBytes  53.5 Mbits/sec                  receiver

iperf Done.

 

 

So I'll stick with the vmxnet3 for now. Is there a list of these types somewhere so I can research/try some more? Thank you for all your help and I'm sorry I didn't understand your original question.

Share this post


Link to post
1 hour ago, joshstrange said:

I'm sorry, yes I was using virtio. 

 

Here are my results with the two you mentioned:

 

e1000-82545em:


Connecting to host <removed>, port 5201
Reverse mode, remote host <removed> is sending
[  4] local <removed> port 43176 connected to <removed> port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.47 MBytes  12.3 Mbits/sec
[  4]   1.00-2.00   sec  4.02 MBytes  33.7 Mbits/sec
[  4]   2.00-3.00   sec  4.59 MBytes  38.5 Mbits/sec
[  4]   3.00-4.00   sec  4.80 MBytes  40.3 Mbits/sec
[  4]   4.00-5.00   sec  5.08 MBytes  42.6 Mbits/sec
[  4]   5.00-6.00   sec  4.39 MBytes  36.8 Mbits/sec
[  4]   6.00-7.00   sec  4.15 MBytes  34.8 Mbits/sec
[  4]   7.00-8.00   sec  4.47 MBytes  37.5 Mbits/sec
[  4]   8.00-9.00   sec  4.35 MBytes  36.5 Mbits/sec
[  4]   9.00-10.00  sec  4.45 MBytes  37.4 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  42.7 MBytes  35.8 Mbits/sec  102             sender
[  4]   0.00-10.00  sec  42.5 MBytes  35.7 Mbits/sec                  receiver

iperf Done.

 

vmxnet3:


Connecting to host <removed>, port 5201
Reverse mode, remote host <removed> is sending
[  4] local <removed> port 37254 connected to <removed> port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.66 MBytes  13.9 Mbits/sec
[  4]   1.00-2.00   sec  6.58 MBytes  55.2 Mbits/sec
[  4]   2.00-3.00   sec  6.60 MBytes  55.3 Mbits/sec
[  4]   3.00-4.00   sec  7.07 MBytes  59.3 Mbits/sec
[  4]   4.00-5.00   sec  7.50 MBytes  62.9 Mbits/sec
[  4]   5.00-6.00   sec  7.59 MBytes  63.7 Mbits/sec
[  4]   6.00-7.00   sec  7.16 MBytes  60.1 Mbits/sec
[  4]   7.00-8.00   sec  8.00 MBytes  67.1 Mbits/sec
[  4]   8.00-9.00   sec  5.04 MBytes  42.3 Mbits/sec
[  4]   9.00-10.00  sec  5.68 MBytes  47.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  63.8 MBytes  53.5 Mbits/sec   59             sender
[  4]   0.00-10.00  sec  63.8 MBytes  53.5 Mbits/sec                  receiver

iperf Done.

 

 

So I'll stick with the vmxnet3 for now. Is there a list of these types somewhere so I can research/try some more? Thank you for all your help and I'm sorry I didn't understand your original question.

no worries about misunderstanding.

 

I use to have a link but couldn't find it regarding virtualized nics. I think the only other is a Realtek one. You could search "KVM virtualized NIC e1000 virtue vmxnet3" and see if anything else pops up. But those 3 are the most used.

Share this post


Link to post

Out of interest I made a test between my Ubuntu 19.04 server VM (10.0.104.11) and my Ubuntu 19.04 desktop VM (10.0.104.12).

 

I use parallel TCP sessions to maximize the throughput. This is the command used at the desktop side.

iperf3 -c 10.0.104.11 -i0 -t20 -P10 -w400k -R

image.png.e8ef18cb98cb81d5d638e670565aa7bd.png

 

Since both VMs are on the same host, there is no true network limitation and performance is more determined by the VMs themselves.

The Ubuntu server maxes out at 28.4 Gb/s on my server and shows the driver is capable of handling very high speeds.

  • Like 1
  • Upvote 1

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.