VirtioFS Support Page

[kadajawi]

According to that Github page, "--inode-file-handles=mandatory" should work. I'd like to try, but I just don't know how to enter it in the XML...

[SimonF]

6 hours ago, kadajawi said:

According to that Github page, "--inode-file-handles=mandatory" should work. I'd like to try, but I just don't know how to enter it in the XML...

Is this from the new rust version, the rust options are not available via libvirt xml at present. You will have to create a script and change add the option. You would need to update the path to be your script. <binary path='/usr/libexec/virtiofsd' xattr='on'>

[mackid1993]

Just wondering if 6.12-RC4 has any changes that may impact this or if there is anything new that I can test? I'm eager to get this working!

[SimonF]

19 minutes ago, mackid1993 said:

Just wondering if 6.12-RC4 has any changes that may impact this or if there is anything new that I can test? I'm eager to get this working!

Nothing changed within this release. I hope qemu and libvirt will be updated in 6.13 I did read a patch that around not requirimg shared in mem backing but not sure of context yet. And likely to be qemu 8.1 released is 8.0 at present.

[mackid1993]

I was able to run Virtiofsd with the --inode-file-handles=mandatory switch. I created a script:

 

#!/bin/sh
exec /usr/libexec/virtiofsd --inode-file-handles=mandatory  -o sandbox=chroot "$@"

 

And called it virtiofsd.sh and placed it in /usr/libexec. In my xml I just changed /usr/libexec/virtiofsd to virtiofsd.sh so it runs with the argument. I am testing this with the latest rust version.

 

To make this persistent I created a folder in /boot called virtiofsd and placed the rust version in there along with my shell script. I added the following to /boot/config/go

 

mv /usr/libexec/virtiofsd /usr/libexec/virtiofsd.old

cp /boot/virtiofsd/virtiofsd /usr/libexec/virtiofsd

chmod +x /usr/libexec/virtiofsd

cp /boot/virtiofsd/virtiofsd.sh /usr/libexec/virtiofsd.sh

chmod +x /usr/libexec/virtiofsd.sh

 

Now to test with Backblaze Personal Backup on Win 11 and hopefully no crashing now. @SimonF does this sound like a good test case?

 

EDIT: Prior to running with --inode-file-handles=mandatory I noticed that in task manager processes were reporting unusually low memory usage but the overall memory usage percentage was extremely high and it just didn't make any sense. Now I am seeing the opposite where processes report memory usage consistent with what I would expect. I'm hopeful that this may work now.

Edited April 28, 2023 by mackid1993

[mackid1993]

So I have about 4 hours of uptime now. Memory usage is pretty high and I noticed it's all in the non paged pool. The non paged pool is using 4.7 GB. I'm wondering if it's that high as I'm running a backup and it's copying thousands of files. I wonder now if it will continue to grow or eventually stop before the machine hangs.

[mackid1993]

I just wanted to update. While It didn't lock up over night the non paged pool kept growing and growing and reached over 5 GB. I did tinker with a few other options with Virtiofsd but nothing seemed to help. I think this may be a driver issue in Windows.

[mackid1993]

I also want to draw attention to this Reddit post. I observed the exact same issue described with the non paged pool leak. I used poolmon to track it down and it pointed to refs.sys. The non paged pool will just slowly grow and grow until the VM runs out of memory.

[djmulder]

Is there a way to fully upgrade qemu/libvirt in unraid? Like with the above method to use some go script to replace the links? 

I know too little of linux to do this comfortably. Some step by step guide would be appreciated. (or some install script?)

 

I'm having random win11 crashes myself I'm assuming related to this (even tho I've disabled the use of virtiofs). Would be nice to be update on the go and see if it fixes the issues.

[mackid1993]

5 hours ago, djmulder said:

Is there a way to fully upgrade qemu/libvirt in unraid? Like with the above method to use some go script to replace the links? 

I know too little of linux to do this comfortably. Some step by step guide would be appreciated. (or some install script?)

 

I'm having random win11 crashes myself I'm assuming related to this (even tho I've disabled the use of virtiofs). Would be nice to be update on the go and see if it fixes the issues.

I have a feeling this is the virtio drivers and not QEMU. Keep an eye on your non paged pool in task manager go to performance and then memory. I found that the non paged pool slowly grows. Usually that is caused by a buggy kernel mode driver. I've tracked it down to refs.sys which is used for Resilient File System (ReFS). I have a feeling VirtioFS is piggybacking off of that somehow and the driver for Windows is still buggy.

[johnsanc]

3 hours ago, mackid1993 said:

I have a feeling this is the virtio drivers and not QEMU.

I have mentioned this several times before, but the lockups most people are having that peg all CPU cores is directly related to the presence of the memory backing config:

 

  <memoryBacking>
    <source type='memfd'/>
    <access mode='shared'/>
  </memoryBacking>

 

Please try to reproduce with JUST this config in place and nothing related to virtioFS.

[djmulder]

On 5/5/2023 at 12:25 AM, johnsanc said:

I have mentioned this several times before, but the lockups most people are having that peg all CPU cores is directly related to the presence of the memory backing config:

 

  <memoryBacking>
    <source type='memfd'/>
    <access mode='shared'/>
  </memoryBacking>

 

Please try to reproduce with JUST this config in place and nothing related to virtioFS.

I can't officially say I'm having the exact same problems (they sound very similar, I had virtiofs and just removed the mount, the drivers and winfsp are still present) and fairly new in joining this conversation. But do you mean by "peg all CPU cores" that I don't pin them? I have 8 cores pinned. It crashes fairly frequent, but infrequent enough to really pinpoint when. It can be mid minecraft session that I just have to force close the VM or I can play for hours.  

 

Here's my config: (note I've made quite a few changes already, so it's not 100% accurate to when the crashing started. So chances are you see things that I already messed with)

Spoiler

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='4'>
  <name>GamePC 1060</name>
  <uuid>aa576d9b-171a-229d-fe2b-685172eb50f9</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <source type='memfd'/>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='6'/>
    <vcpupin vcpu='1' cpuset='22'/>
    <vcpupin vcpu='2' cpuset='7'/>
    <vcpupin vcpu='3' cpuset='23'/>
    <vcpupin vcpu='4' cpuset='8'/>
    <vcpupin vcpu='5' cpuset='24'/>
    <vcpupin vcpu='6' cpuset='9'/>
    <vcpupin vcpu='7' cpuset='25'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-7.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/aa576d9b-171a-229d-fe2b-685172eb50f9_VARS-pure-efi-tpm.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='4' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/mnt/user/domains/GamePC 1060/vdisk1-snapshot-clean-install.img' index='3'/>
      <backingStore type='file' index='4'>
        <format type='qcow2'/>
        <source file='/mnt/user/domains/GamePC 1060/vdisk1.img'/>
        <backingStore/>
      </backingStore>
      <target dev='hdc' bus='virtio'/>
      <serial>Boot</serial>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/mnt/user/Virtual Machines Data/GamePC 1060/vdisk2.img' index='2'/>
      <backingStore/>
      <target dev='hdd' bus='virtio'/>
      <serial>Data1</serial>
      <alias name='virtio-disk3'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/mnt/user/Virtual Machines Data/GamePC 1060/vdisk3.img' index='1'/>
      <backingStore/>
      <target dev='hde' bus='virtio'/>
      <serial>Games1</serial>
      <alias name='virtio-disk4'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x8'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x9'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='8' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0xa'/>
      <alias name='pci.9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:91:83:f0'/>
      <source bridge='br0'/>
      <target dev='vnet3'/>
      <model type='virtio-net'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-4-GamePC 1060/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <tpm model='tpm-tis'>
      <backend type='emulator' version='2.0' persistent_state='yes'/>
      <alias name='tpm0'/>
    </tpm>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+0</label>
    <imagelabel>+0:+0</imagelabel>
  </seclabel>
</domain>

 

I don't like posting such a big XML, suggestions to snip it? Or should I just make a pastebin/file?

Edited May 7, 2023 by djmulder

[JonathanM]

9 hours ago, djmulder said:

I don't like posting such a big XML, suggestions to snip it?

[djmulder]

Small update, I removed WinFSP and stopped the Virtio service from starting and the VM survived the night for a change. So knock on wood 

[NLS]

Since 6.12 is coming, is there some "brief" on the current state of this and any guide on how to PROPERLY set it up?

 

[djmulder]

21 minutes ago, NLS said:

Since 6.12 is coming, is there some "brief" on the current state of this and any guide on how to PROPERLY set it up?

 

I've tried the "next" release with 6.12 and it seemed to not be updated enough to fix the issue.

 

26 minutes ago, djmulder said:

Small update, I removed WinFSP and stopped the Virtio service from starting and the VM survived the night for a change. So knock on wood 

Officially crashed 5 minutes ago.. I wasn't even connected.. so yeh didn't fix it 

[johnsanc]

The issue is precisely this: https://gitlab.com/qemu-project/qemu/-/issues/1270

And it has not been corrected. Until there is an update to that issue, I'd say this entire feature should be avoided completely for most people.

[mackid1993]

Simon said in an earlier post that memfd/shared may not be needed eventually if I understood correctly. If/when that occurs hopefully this won't be an issue any longer. Until then I removed everything but </nosharepages> from my memory backing config and my VM is stable. Other than that the non-paged pool will slowly fill up until it crashes.

[djmulder]

2 hours ago, johnsanc said:

The issue is precisely this: https://gitlab.com/qemu-project/qemu/-/issues/1270

And it has not been corrected. Until there is an update to that issue, I'd say this entire feature should be avoided completely for most people.

Thx will remove it for now.. let's see what it does.

[mackid1993]

Any news on this? Have the latest 6.12 RCs made any changes that could fix this memory leak?

[SimonF]

8 hours ago, mackid1993 said:

Any news on this? Have the latest 6.12 RCs made any changes that could fix this memory leak?

No, report on git hub moving to QEMU 7.2 seems to fix the issue. Hopefully 6.13 will have QEMU updated from 7.1

[mackid1993]

11 hours ago, SimonF said:

No, report on git hub moving to QEMU 7.2 seems to fix the issue. Hopefully 6.13 will have QEMU updated from 7.1

Oh that would be awesome if Lime Tech can do that. Thanks for the update Simon!

[orangeisnon]

6.12 stable is out and the changelog includes this line:

"Update Memory Backup processing for Virtiofs."

Anyone willing to upgrade and test? It seems the consensus is that an update to QEMU is needed but maybe just maybe this does it?

[SimonF]

12 minutes ago, orangeisnon said:

6.12 stable is out and the changelog includes this line:

"Update Memory Backup processing for Virtiofs."

Anyone willing to upgrade and test? It seems the consensus is that an update to QEMU is needed but maybe just maybe this does it?

That is just changes to XML to automatically change rather than manual. It will not fix the issue.

[mackid1993]

On 6/13/2023 at 2:23 AM, SimonF said:

No, report on git hub moving to QEMU 7.2 seems to fix the issue. Hopefully 6.13 will have QEMU updated from 7.1

Is there any way to manually upgrade to QEMU 7.2 like how were were able to switch to the rust version of Virtiofsd?