Bizarre iperf results

Stupifier

By Stupifier
in VM Engine (KVM)

Recommended Posts

Stupifier Contributor

Stupifier

Posted
Posted (edited)

UPDATE WITH SOLUTION:
Changed Network Adapter in VM XML from virtio to e1000-82545em. Now iperf3 reports no packet loss and solid speed. I want vmxnet3 network adapter, but I do not know how to get that to install correctly on the VM.....I keep trying and failing. So e1000 fallback is what I'm using.
Why is this the solution? Isn't virtio supposed to be the best? Ya, I have NO IDEA....just reporting what I see in the iperf3 results.

------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------

Someone please help me on this.....

SUMMARY:
iperf between a regular Win10 Computer and Unraid Win10 VM observes HIGH packet loss (in one direction only).....but run same thing between Unraid itself and regular Win10 Computer.....no packet loss. Everything Ethernet connected. What is going on here!?


IPERF3 RESULTS BELOW. I confirm results on both ends of terminal console.

------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------

Win10 VM is iperf3 server, a regular Win10 computer is iperf3 client. Results are shown in one direction, then using the -R flag to reverse the direction.
  

PS C:\Windows\system32> iperf3 -c 192.168.1.119 -u -b 500M
Connecting to host 192.168.1.119, port 5201
[  4] local 192.168.1.162 port 62238 connected to 192.168.1.119 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.01   sec  58.8 MBytes   490 Mbits/sec  7526
[  4]   1.01-2.02   sec  60.2 MBytes   501 Mbits/sec  7709
[  4]   2.02-3.01   sec  54.8 MBytes   462 Mbits/sec  7011
[  4]   3.01-4.00   sec  59.1 MBytes   499 Mbits/sec  7571
[  4]   4.00-5.00   sec  59.5 MBytes   499 Mbits/sec  7611
[  4]   5.00-6.01   sec  60.4 MBytes   501 Mbits/sec  7725
[  4]   6.01-7.01   sec  59.5 MBytes   502 Mbits/sec  7618
[  4]   7.01-8.01   sec  59.0 MBytes   496 Mbits/sec  7558
[  4]   8.01-9.01   sec  60.2 MBytes   505 Mbits/sec  7706
[  4]   9.01-10.00  sec  59.3 MBytes   502 Mbits/sec  7595
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec   591 MBytes   496 Mbits/sec  0.119 ms  0/75629 (0%)
[  4] Sent 75629 datagrams

iperf Done.

and now the -R flag to reverse the direction. This means my Win10 VM is sending the packets instead of receiving the packets. 

PS C:\Windows\system32> iperf3 -c 192.168.1.119 -u -b 500M -R
Connecting to host 192.168.1.119, port 5201
Reverse mode, remote host 192.168.1.119 is sending
[  4] local 192.168.1.162 port 49610 connected to 192.168.1.119 port 5201
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-1.01   sec  26.0 MBytes   216 Mbits/sec  0.035 ms  4644/7968 (58%)
[  4]   1.01-2.01   sec  22.1 MBytes   186 Mbits/sec  0.037 ms  4779/7613 (63%)
[  4]   2.01-3.00   sec  23.5 MBytes   198 Mbits/sec  0.037 ms  4610/7615 (61%)
[  4]   3.00-4.01   sec  23.7 MBytes   197 Mbits/sec  0.035 ms  4525/7554 (60%)
[  4]   4.01-5.01   sec  23.8 MBytes   199 Mbits/sec  0.038 ms  4570/7617 (60%)
[  4]   5.01-6.01   sec  23.5 MBytes   198 Mbits/sec  0.034 ms  4592/7597 (60%)
[  4]   6.01-7.00   sec  24.4 MBytes   207 Mbits/sec  0.039 ms  4604/7730 (60%)
[  4]   7.00-8.00   sec  23.5 MBytes   197 Mbits/sec  0.039 ms  4546/7554 (60%)
iperf3: OUT OF ORDER - incoming packet = 10703 and received packet = 76218 AND SP = 4
iperf3: OUT OF ORDER - incoming packet = 10708 and received packet = 76221 AND SP = 4
iperf3: OUT OF ORDER - incoming packet = 10714 and received packet = 76230 AND SP = 4
----LOTS MORE OF THESE OUT OF ORDER PACKETS----
[  4]   9.01-10.00  sec  20.6 MBytes   175 Mbits/sec  5113.262 ms  5071/7298 (69%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec   598 MBytes   501 Mbits/sec  4256.378 ms  46955/76236 (62%)
[  4] Sent 76236 datagrams
[SUM]  0.0-10.0 sec  639 datagrams received out-of-order

iperf Done.


SO, that is between the Win10 VM and a regular Win10 Computer...all on same network. Very high packet loss observed in the reverse direction!

 

NOW, I will do the same test except between Unraid iteslf and the regular Win10 Computer....and NO packet loss observed in either direction! Results below.


Unraid is iperf3 server, regular Win10 computer is iperf3 client. Results are shown in one direction, then using the -R flag to reverse the direction.

  

PS C:\Windows\system32> iperf3 -c 192.168.1.111 -u -b 500M
Connecting to host 192.168.1.111, port 5201
[  4] local 192.168.1.162 port 49200 connected to 192.168.1.111 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec  59.3 MBytes   498 Mbits/sec  7596
[  4]   1.00-2.01   sec  58.9 MBytes   490 Mbits/sec  7542
[  4]   2.01-3.00   sec  59.7 MBytes   505 Mbits/sec  7636
[  4]   3.00-4.00   sec  60.1 MBytes   504 Mbits/sec  7688
[  4]   4.00-5.00   sec  59.9 MBytes   503 Mbits/sec  7672
[  4]   5.00-6.00   sec  58.9 MBytes   494 Mbits/sec  7541
[  4]   6.00-7.01   sec  57.4 MBytes   479 Mbits/sec  7349
[  4]   7.01-8.00   sec  62.5 MBytes   527 Mbits/sec  7996
[  4]   8.00-9.00   sec  58.5 MBytes   491 Mbits/sec  7493
[  4]   9.00-10.01  sec  57.7 MBytes   479 Mbits/sec  7388
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.01  sec   593 MBytes   497 Mbits/sec  0.017 ms  20/75900 (0.026%)
[  4] Sent 75900 datagrams

iperf Done.

and now the -R flag to reverse the direction. This means Unraid is sending the packets instead of receiving the packets.

  

PS C:\Windows\system32> iperf3 -c 192.168.1.111 -u -b 500M -R
Connecting to host 192.168.1.111, port 5201
Reverse mode, remote host 192.168.1.111 is sending
[  4] local 192.168.1.162 port 51055 connected to 192.168.1.111 port 5201
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-1.00   sec  59.8 MBytes   502 Mbits/sec  0.242 ms  321/7976 (4%)
[  4]   1.00-2.00   sec  59.6 MBytes   500 Mbits/sec  0.233 ms  0/7630 (0%)
[  4]   2.00-3.00   sec  59.6 MBytes   500 Mbits/sec  0.245 ms  0/7628 (0%)
[  4]   3.00-4.00   sec  59.6 MBytes   500 Mbits/sec  0.235 ms  0/7631 (0%)
[  4]   4.00-5.00   sec  59.6 MBytes   500 Mbits/sec  0.241 ms  0/7629 (0%)
[  4]   5.00-6.00   sec  59.6 MBytes   500 Mbits/sec  0.243 ms  0/7629 (0%)
[  4]   6.00-7.00   sec  59.6 MBytes   500 Mbits/sec  0.236 ms  0/7630 (0%)
[  4]   7.00-8.00   sec  59.6 MBytes   500 Mbits/sec  0.236 ms  0/7629 (0%)
[  4]   8.00-9.00   sec  59.6 MBytes   500 Mbits/sec  0.224 ms  0/7631 (0%)
[  4]   9.00-10.00  sec  59.6 MBytes   500 Mbits/sec  0.252 ms  0/7627 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec   599 MBytes   502 Mbits/sec  0.209 ms  321/76669 (0.42%)
[  4] Sent 76669 datagrams

iperf Done.

And just in case anyone is curious....I did in fact run these iperf tests between the Win10 VM and Unraid itself....as expected, no packet loss. Just a sanity check

I also went ahead and fired up another VM...just a standard Ubuntu Server (CLI only)....and I get the packet loss HERE as well (just as described above)

Using latest available virtio for the network adapter on these VMs....which should be the best there is

Edited by Stupifier
Link to comment
ghost82 Proficient

ghost82

Posted
Posted

You could try to increase the buffer size, by setting tx_queue_size and rx_queue_size in your xml, maximize them should decrease packet loss.

I can't try because vmxnet3 breaks login to icloud web on my vm.

  

<interface type='bridge'>
  <mac address='11:22:33:44:55:66'/>
  <source bridge='br0'/>
  <driver rx_queue_size='1024' tx_queue_size='1024'/>
  <model type='vmxnet3'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</interface>

 

Link to comment
Stupifier Contributor

Stupifier

Posted
  • Author
Posted (edited)
3 hours ago, ghost82 said:

You could try to increase the buffer size, by setting tx_queue_size and rx_queue_size in your xml, maximize them should decrease packet loss.

I can't try because vmxnet3 breaks login to icloud web on my vm.

  


<interface type='bridge'>
  <mac address='11:22:33:44:55:66'/>
  <source bridge='br0'/>
  <driver rx_queue_size='1024' tx_queue_size='1024'/>
  <model type='vmxnet3'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</interface>

 

But I can't even get vmxnet3 to work at all....Windows never finds a driver for it so it is DOA. And I searched like to hell to figure out how to install vmxnet3 driver and failed to find anything useful. Everyone mentions VMWare Tools which isn't useful at all.

Or are you just talking about increasing the rx/tx queue size regardless of if I'm virtio, e1000, or vmxnet3?

UPDATE: Just tried setting the tx/rx buffer size to 1024 on virtio.....No improvement to the iperf3 tests (same as above).

Edited by Stupifier
Link to comment
ghost82 Proficient

ghost82

Posted
Posted (edited)

Sorry, I understood you performed tests with vmxnet3 and the issue was packet loss.

vmxnet3 drivers are in vmware tools (free download after registration on vmware website), so you can install the whole package or extract from that package the drivers you need.

Edited by ghost82
Link to comment
Stupifier Contributor

Stupifier

Posted
  • Author
Posted
7 hours ago, ghost82 said:

Sorry, I understood you performed tests with vmxnet3 and the issue was packet loss.

vmxnet3 drivers are in vmware tools (free download after registration on vmware website), so you can install the whole package or extract from that package the drivers you need.

Right...Thing is....I actually grabbed the VMWare Tools iso file, mounted it inside my Win10 VM, then tried running the setup executable....nothing happens. No popup prompts to install, nothing... 

Then I added vmxnet3 as a 2nd NIC device in the Win10VM....I'm able to to boot....it shows an unknown network device. I try to update the driver on it and point it to the VMWare Tool iso mount location......nothing...doesn't find any drivers for it.

 

For some reason, I'm not able to get vmxnet3 drivers on this thing.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing