What is the difference between virtio and virtio-net ?


Recommended Posts

I currently get about 20Gbps between vm and unraid using virtio(test by iperf3), but at the same time I noticed a lot of logs in my syslog.

Jan 12 22:20:12 Unraid kernel: tun: unexpected GSO type: 0x0, gso_size 1192, hdr_len 1258
Jan 12 22:20:12 Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 12 22:20:12 Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 12 22:20:12 Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 12 22:20:12 Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 12 22:20:12 Unraid kernel: tun: unexpected GSO type: 0x0, gso_size 1192, hdr_len 1258
Jan 12 22:20:12 Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 12 22:20:12 Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 12 22:20:12 Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jan 12 22:20:12 Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

 

After looking through the forums I found that I need to change virtio to virtio-net to fix this problem. However, after testing with virtio-net the transfer speed between vm and unraid was reduced to 1.2Gbps, accompanied by high CPU usage.

Is this a bug of virtio-net?

Link to comment

Other than the event in your logs, do you notice anything causing issues?  If not, I'm not sure what you're asking us to "fix" here other than to make those log messages go away which seems like a kernel problem (probably driver related).

Link to comment
  • 5 months later...
  • 1 month later...

This is really confusing. If you start a VM with the "virtio" network adapter it starts QEMU with the "virtio-net-pci" NIC:

-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:99:b8:93,bus=pci.0,addr=0x3 \

 

By using "virtio-net" it starts QEMU with the "virtio-net" NIC:

-device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:99:b8:93,bus=pci.0,addr=0x3 \

 

But "virtio-net" is not part of the supported NICs?!

qemu -nic model=help
Supported NIC models:
e1000
e1000-82544gc
e1000-82545em
e1000e
i82550
i82551
i82557a
i82557b
i82557c
i82558a
i82558b
i82559a
i82559b
i82559c
i82559er
i82562
i82801
ne2k_pci
pcnet
rtl8139
tulip
virtio-net-pci
virtio-net-pci-non-transitional
virtio-net-pci-transitional
vmxnet3

 

And now the super confusing part:

qemu -device help
...
Network devices:
name "e1000", bus PCI, alias "e1000-82540em", desc "Intel Gigabit Ethernet"
name "e1000-82544gc", bus PCI, desc "Intel Gigabit Ethernet"
name "e1000-82545em", bus PCI, desc "Intel Gigabit Ethernet"
name "e1000e", bus PCI, desc "Intel 82574L GbE Controller"
name "i82550", bus PCI, desc "Intel i82550 Ethernet"
name "i82551", bus PCI, desc "Intel i82551 Ethernet"
name "i82557a", bus PCI, desc "Intel i82557A Ethernet"
name "i82557b", bus PCI, desc "Intel i82557B Ethernet"
name "i82557c", bus PCI, desc "Intel i82557C Ethernet"
name "i82558a", bus PCI, desc "Intel i82558A Ethernet"
name "i82558b", bus PCI, desc "Intel i82558B Ethernet"
name "i82559a", bus PCI, desc "Intel i82559A Ethernet"
name "i82559b", bus PCI, desc "Intel i82559B Ethernet"
name "i82559c", bus PCI, desc "Intel i82559C Ethernet"
name "i82559er", bus PCI, desc "Intel i82559ER Ethernet"
name "i82562", bus PCI, desc "Intel i82562 Ethernet"
name "i82801", bus PCI, desc "Intel i82801 Ethernet"
name "ne2k_isa", bus ISA
name "ne2k_pci", bus PCI
name "pcnet", bus PCI
name "rocker", bus PCI, desc "Rocker Switch"
name "rtl8139", bus PCI
name "tulip", bus PCI
name "usb-net", bus usb-bus
name "virtio-net-device", bus virtio-bus
name "virtio-net-pci", bus PCI, alias "virtio-net"
name "virtio-net-pci-non-transitional", bus PCI
name "virtio-net-pci-transitional", bus PCI
name "vmxnet3", bus PCI, desc "VMWare Paravirtualized Ethernet v3"

 

I highlight it:

name "virtio-net-pci", bus PCI, alias "virtio-net"

 

Only an alias?! Doesn't make sense at all.

 

EDIT: Ok, I finally understood the magic behind "virtio-net". It is really only an alias of "virtio-net-pci"! No joke.

 

The only difference is that "virtio-net" removes the "vhost=on" flag:

 

virtio-net

-netdev tap,fd=33,id=hostnet0 \
-device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:99:b8:93,bus=pci.0,addr=0x3 \

 

virtio

-netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=34 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:99:b8:93,bus=pci.0,addr=0x3 \

 

And it's absolutelly logic that this causes a bad performance as QEMU then creates an internal "virtio-net device":

https://www.usenix.org/sites/default/files/conference/protected-files/srecon20americas_slides_krosnov.pdf

image.png.37d1ad18c9d35c2794ff31b27f2df9c1.png

 

Instead of sharing the memory with the host:

image.png.fe341fc7233dacf2e56bf41f2cfe8255.png

 

A good write-up can be found here:

https://insujang.github.io/2021-03-15/virtio-and-vhost-architecture-part-2/

 

And now we understand the help text as well:

Quote

Default and recommended is 'virtio-net', which gives improved stability. To improve performance 'virtio' can be selected, but this may lead to stability issues.

 

 

Not sure about the stability thing, but if the Guest supports it, I would use "virtio", which enables vhost. As I think the names of the adapters are confusing I opened a bug report:

 

  • Like 3
  • Thanks 2
Link to comment
On 8/2/2021 at 8:06 PM, mgutt said:

This is really confusing. If you start a VM with the "virtio" network adapter it starts QEMU with the "virtio-net-pci" NIC:


-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:99:b8:93,bus=pci.0,addr=0x3 \

 

By using "virtio-net" it starts QEMU with the "virtio-net" NIC:


-device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:99:b8:93,bus=pci.0,addr=0x3 \

 

But "virtio-net" is not part of the supported NICs?!


qemu -nic model=help
Supported NIC models:
e1000
e1000-82544gc
e1000-82545em
e1000e
i82550
i82551
i82557a
i82557b
i82557c
i82558a
i82558b
i82559a
i82559b
i82559c
i82559er
i82562
i82801
ne2k_pci
pcnet
rtl8139
tulip
virtio-net-pci
virtio-net-pci-non-transitional
virtio-net-pci-transitional
vmxnet3

 

And now the super confusing part:


qemu -device help
...
Network devices:
name "e1000", bus PCI, alias "e1000-82540em", desc "Intel Gigabit Ethernet"
name "e1000-82544gc", bus PCI, desc "Intel Gigabit Ethernet"
name "e1000-82545em", bus PCI, desc "Intel Gigabit Ethernet"
name "e1000e", bus PCI, desc "Intel 82574L GbE Controller"
name "i82550", bus PCI, desc "Intel i82550 Ethernet"
name "i82551", bus PCI, desc "Intel i82551 Ethernet"
name "i82557a", bus PCI, desc "Intel i82557A Ethernet"
name "i82557b", bus PCI, desc "Intel i82557B Ethernet"
name "i82557c", bus PCI, desc "Intel i82557C Ethernet"
name "i82558a", bus PCI, desc "Intel i82558A Ethernet"
name "i82558b", bus PCI, desc "Intel i82558B Ethernet"
name "i82559a", bus PCI, desc "Intel i82559A Ethernet"
name "i82559b", bus PCI, desc "Intel i82559B Ethernet"
name "i82559c", bus PCI, desc "Intel i82559C Ethernet"
name "i82559er", bus PCI, desc "Intel i82559ER Ethernet"
name "i82562", bus PCI, desc "Intel i82562 Ethernet"
name "i82801", bus PCI, desc "Intel i82801 Ethernet"
name "ne2k_isa", bus ISA
name "ne2k_pci", bus PCI
name "pcnet", bus PCI
name "rocker", bus PCI, desc "Rocker Switch"
name "rtl8139", bus PCI
name "tulip", bus PCI
name "usb-net", bus usb-bus
name "virtio-net-device", bus virtio-bus
name "virtio-net-pci", bus PCI, alias "virtio-net"
name "virtio-net-pci-non-transitional", bus PCI
name "virtio-net-pci-transitional", bus PCI
name "vmxnet3", bus PCI, desc "VMWare Paravirtualized Ethernet v3"

 

I highlight it:


name "virtio-net-pci", bus PCI, alias "virtio-net"

 

Only an alias?! Doesn't make sense at all.

 

EDIT: Ok, I finally understood the magic behind "virtio-net". It is really only an alias of "virtio-net-pci"! No joke.

 

The only difference is that "virtio-net" removes the "vhost=on" flag:

 

virtio-net


-netdev tap,fd=33,id=hostnet0 \
-device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:99:b8:93,bus=pci.0,addr=0x3 \

 

virtio


-netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=34 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:99:b8:93,bus=pci.0,addr=0x3 \

 

And it's absolutelly logic that this causes a bad performance as QEMU then creates an internal "virtio-net device":

https://www.usenix.org/sites/default/files/conference/protected-files/srecon20americas_slides_krosnov.pdf

image.png.37d1ad18c9d35c2794ff31b27f2df9c1.png

 

Instead of sharing the memory with the host:

image.png.fe341fc7233dacf2e56bf41f2cfe8255.png

 

A good write-up can be found here:

https://insujang.github.io/2021-03-15/virtio-and-vhost-architecture-part-2/

 

And now we understand the help text as well:

 

 

Not sure about the stability thing, but if the Guest supports it, I would use "virtio", which enables vhost. As I think the names of the adapters are confusing I opened a bug report:

 

Great research, but I'm also super confused that why virtio-net act like a 1G network, but virtio is 10G network. Just same feeling as in the first post.

Why is there so big difference for the speed since it is just a alias?  Can we make it work like a 10G network?

Link to comment
8 hours ago, jinlife said:

Why is there so big difference for the speed since it is just a alias?

You should read my complete post. There is a difference. It's the vhost access flag. If it's disabled, QEMU needs to emulate a virtio-net device which produces huge CPU load (and limits the bandwidth).

 

Or in other words: There is really no reason to use the "virtio-net" adapter (with disabled vhost access). Regarding the unRAID help text it's only the default because it's more "stable" without any further explanation which OS could be "unstable". Maybe someone finds the reason why it became the default.

Link to comment
21 minutes ago, mgutt said:

There is really no reason to use the "virtio-net" adapter

virtio-net was introduced as a solution for people using both VMs and docker custom networks on the same interface, e.g. br0.

The virtio driver would cause system crashes when docker containers are active at the same time. Hence the help text talking about stability when using virtio-net.

 

Stability was chosen as default above performance.

 

  • Like 3
  • Thanks 1
Link to comment
On 8/5/2021 at 12:27 PM, bonienl said:

The virtio driver would cause system crashes when docker containers are active at the same time

Thank you for this information. If this is a general problem, why isn't "virtio" greyed out if a docker uses the br0 network?

Link to comment
22 hours ago, mgutt said:

why isn't "virtio" greyed out if a docker uses the br0 network?

 

First of all, thank you very much for these information, without reading qemu code I noticed performance issue and changed from virtio-net to virtio, but knowing the reason is always good.

I think it's not so simple, because what if one enable a docker after the vm?

Instead, I would explain that in the "info/help" box with some more info, instead of "stability".

Link to comment
5 minutes ago, ghost82 said:

I think it's not so simple, because what if one enable a docker after the vm?

The other way around would be possible, too (grey out br0 for dockers if already used by a vm)

 

Or only show a warning like I showed in a screenshot of this idea:

https://forums.unraid.net/topic/102826-add-warning-to-use-cache-share-settings-and-disallow-change-for-appdata-share/

 

5 minutes ago, ghost82 said:

I would explain that in the "info/help" box

I think its better if the OS is "self-explaining".

  • Like 1
Link to comment

...interesting finds.

Two/three more scenarios, that will probably add to the headache:

 

- what about br0.xx, when using VLANs? Which combination of possible allocation/attachments of VMS and Dockers will be "safe" or will all be unsafe, too?

- what if a VM tries to switch the NIC into promiscuous mode, like when starting a VPN gateway SW inside the VM? (maybe this kind of behavior is what kills the NIC in the first place?)

- what if you enable a second bridge on a second NIC in unraid? Is only this single bridge affected or do all bridges inherit this "bug"?

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.