Since I removed virtiofs, and stopped the service on my VM, it no longer hangs daily, and having to force reboot. 
Fingers crossed. 

qemu updated to 7.2 in 6.12.5 today!

There's still a non-paged pool memory leak in Windows. Hopefully new Virtio drivers or qemu 8 will help.

6 hours ago, mackid1993 said:

There's still a non-paged pool memory leak in Windows. Hopefully new Virtio drivers or qemu 8 will help.

Is there an issue for this memory leak problem in the virtio driver's repo?

59 minutes ago, ppggff said:

Is there an issue for this memory leak problem in the virtio driver's repo?

I haven't seen one. Not sure if it's the driver or qemu.

These are the open issues that I can see cannot see any for winfsp

 

10 hours ago, SimonF said:

These are the open issues that I can see cannot see any for winfsp

 

So maybe it's a qemu issue or something with Unraid.

49 minutes ago, mackid1993 said:

So maybe it's a qemu issue or something with Unraid.

 

It's not limited to unraid.  I have the issue with opensuse tumbleweed as the host.  I have not seen any change across multiple version upgrades.  Currently my qemu version is 8.1.2.

 

I'm also not sure how it could be caused by qemu, as that's running on the host.  The memory leak is inside of the windows guest OS, so it seems to me that it has to be caused by the virtio-win kernel driver, or winfsp.

Edited November 29, 20232 yr by christophocles

1 hour ago, mackid1993 said:

So maybe it's a qemu issue or something with Unraid.

If you can provide details I am happy to open an issue or you can?

On 11/15/2023 at 7:41 AM, christophocles said:

 

 

Oh cool, someone actually read my reddit post.  It would have been even cooler if one of you guys had left a comment linking back to this thread for further discussion 😀.  So here we are 7 months later, I have more data to share.

 

I am 99% sure this has nothing to do with refs.sys.  If you do a case-insensitive search for mmdi you will see the actual string it finds in refs.sys: MmDisableModifiedWriteOfSection.  So it's not really a match to that pool name.

 

There's a thread on superuser that describes more methods for hunting down these non-paged pool leaks.   Note where it says: 

 

 

Mmdi is found in pooltag.txt so you actually have to use xperf and wpa for further debug.  Following the method described there, I captured a snapshot of memory growth, opened it in wpa, loaded symbols, and expanded the Mmdi pool to find stack references to winfsp-x64.dll and virtiofs.exe.  So there's the smoking gun, one of these drivers is the culprit.

 

I upgraded to the latest versions of WinFSP (2.0) and Virtio-win guest tools (0.1.240) and the leak is still active.

 

 

 

@SimonF, this post explains the issue way better than I can. Maybe you can open an issue, I'm guessing with the Virtio win driver team?

Edited November 29, 20232 yr by mackid1993

1 hour ago, christophocles said:

 

It's not limited to unraid.  I have the issue with opensuse tumbleweed as the host.  I have not seen any change across multiple version upgrades.  Currently my qemu version is 8.1.2.

 

I'm also not sure how it could be caused by qemu, as that's running on the host.  The memory leak is inside of the windows guest OS, so it seems to me that it has to be caused by the virtio-win kernel driver, or winfsp.

@mackid1993

 

Created this issue, Christoph, please add additional comments as required.

 

https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1004

1 minute ago, SimonF said:

@mackid1993

 

Created this issue, Christoph, please add additional comments as required.

 

https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1004

Thanks Simon. I'll go on there and add a comment as well.

@SimonF He is asking for some more info and I'm not really sure how to provide what he is asking for. Maybe @christophocles can assist. 

On 11/30/2023 at 11:09 AM, mackid1993 said:

@SimonF He is asking for some more info and I'm not really sure how to provide what he is asking for. Maybe @christophocles can assist. 

 

I added more supporting info and I also submitted a bug report for WinFsp.

 

https://github.com/winfsp/winfsp/issues/534

5 hours ago, christophocles said:

 

I added more supporting info and I also submitted a bug report for WinFsp.

 

https://github.com/winfsp/winfsp/issues/534

Thanks for filling this out and explaining the issue so well. Hopefully one of these teams will patch it. I'm betting its Virtio-win's problem as WinFSP is used in a lot of projects and is pretty stable. I use it with Cryptomator with no issue.

On 12/2/2023 at 3:30 PM, christophocles said:

 

I added more supporting info and I also submitted a bug report for WinFsp.

 

https://github.com/winfsp/winfsp/issues/534

Seems like it's a Virtio driver issue. Winfsp closed their report and provided a pretty good explanation. Hopefully the virtio-win team can reproduce and fix this and hopefully in the next driver release we'll finally be able to use VirtioFS.

@SimonF @christophocles Seems like they can't reproduce, they also referenced Virtio drivers .242 which don't seem to be released.

Does anyone know what C:\Program Files (x86)\fio\fio\fio.exe. The guy who couldn't reproduce it ran that. Is that something new they haven't released yet?

27 minutes ago, mackid1993 said:

Does anyone know what C:\Program Files (x86)\fio\fio\fio.exe. The guy who couldn't reproduce it ran that. Is that something new they haven't released yet?

I am guessing it is just another name for viofs depending on how he created it.

58 minutes ago, SimonF said:

I am guessing it is just another name for viofs depending on how he created it.

Ah, what I don't like is they are testing with pre-release drivers. If it's fixed in unreleased drivers how would we know without compiling them?

@christophocles @SimonF The virtio-win team were able to repro this!

The new VirtoFS Windows driver can be downloaded here. To use it you have to enable driver test mode at an admin command prompt with

bcdedit /set testsigning on

 

I also went into certmgr and added viofs.cat to Trusted Root Certification Authorities and Trusted Publishers.

 

If the watermark bothers you, this little tool makes it go away: https://winaero.com/download-universal-watermark-disabler/

 

It's useful until this fix is released with the newest Virtio drivers per the Virtio-win team.

 

Once the new drivers are released uninstall the watermark removal tool and turn off testsigning:

 

bcdedit /set testsigning off

 

It seems that after further testing the Virto-win team fixed the leak: https://github.com/virtio-win/kvm-guest-drivers-windows/pull/1022

 

I made a copy of these drivers so if they disappear someone ping me and I can share a new link.

 

@SimonF and @christophocles thank you for all of your help with this!

Edited January 12, 20242 yr by mackid1993
more info

After testing there is still a memory leak, just a different memory leak now.

Edit: See discussion here: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1004

It appears that the test drivers may have indeed fixed the leak after all.

Edited January 12, 20242 yr by mackid1993

As @mackid1993 has been testing along with some others on the github issue it looks like they have fixed the issue.

 

Hopefully in the next week or so a new virtio cd will be released with the new driver.

 

@mackid1993 are you using the rust version for the host?

2 minutes ago, SimonF said:

As @mackid1993 has been testing along with some others on the github issue it looks like they have fixed the issue.

 

Hopefully in the next week or so a new virtio cd will be released with the new driver.

 

@mackid1993 are you using the rust version for the host?

I'm just using stock Unraid, no rust. I'm on 6.12.6 with the test driver and it's very stable.