Windows 10 VM completely freezing after 6.11.1 upgrade


Recommended Posts

I upgraded to UnRAID 6.11.1 yesterday and everything went smoothly. However my Windows 10 VM appears to have stability issues and just freezes without any apparent errors as far as I can tell. There is nothing suspicious that I can see in the Windows Event Viewer logs and I don't see anything different in my VM log compared to when it was working fine. 

 

I used to have issues after I upgraded to 6.11.0 where the network connection to the VM would drop, but I resolved this by upgrading the virtio drivers to the ones from the latest stable ISO. This issue with 6.11.1 is different though, and I can tell the VM is completely frozen. I usually use RDP for accessing the VM, but if I plug in a monitor the graphics card I have passed through to the VM I can see the time on the windows lock screen is frozen.

 

The only major change I did was enabled the virtiofs feature. I've attached my diagnostics in case anyone can take a peek to see what the issue might be. I am not really sure where to look.

tower-diagnostics-20221009-1805.zip

Edited by johnsanc
  • Upvote 1
Link to comment

I have the same issue. It seems to be an issue with Virtiofs. If i remove it from my VM and it no longer freezes. When the VM freezes, all the cores allocated to the VM are locked at 100%. If anyone has a fix for it,  it would be great.  Would love to keep using virtiofs. I'm running 6.11,1 with windows 11 vm.

Edited by VegChan
Added more information
Link to comment

Thanks for confirming I’m not the only one. I just checked and all my CPUs assigned to that VM are pegged as well once the freeze occurs.

 

i tried disabling the VirtIO-FS service in windows and it still freezes, so that sounds more like a host problem. Perhaps some configuration issue in the XML or bug?

 

I’m open to suggestions since Virtiofs is sooooooo much faster than SMB shares for my use cases. I would love if we could get this VM freeze issue resolved.

Link to comment
5 hours ago, johnsanc said:

Yes, same here. Unfortunately I cannot find anything useful in any logs that even gives a clue.

Perhaps @SimonF has a suggestion for where to look, or maybe he can reproduce the issue as well?

My testing has been on a simple VM 8G mem, no external GPU just VNC and as yet no lockups, but no heavy utilisation.

 

Also I have not setup a service I just run the exe in a cmd window for virtiofs.exe

 

What is the setup of your machine, as I can add K4000 to VM. my CPU is 12600K.

Edited by SimonF
Link to comment

Here's the basic info of the host machine:

M/B: ASRock X570 Creator Version
BIOS: American Megatrends Inc. Version P2.40. Dated: 04/13/2020
CPU: AMD Ryzen 9 3900X 12-Core @ 3800 MHz
HVM: Enabled
IOMMU: Enabled
Cache: 768 KiB, 6 MB, 64 MB
Memory: 64 GiB DDR4 Multi-bit ECC (max. installable capacity 128 GiB)
Network: eth0: 1000 Mbps, full duplex, mtu 1500
Kernel: Linux 5.19.14-Unraid x86_64

 

And I've attached my VM XML to this post. Note I changed my memory from 32gb down to 16gb because someone else I was chatting with recently mentioned they were having stability issues with anything 32gb and above. So far the VM has been running for about an hour and half with 16gb without any issues yet. I'll let you know how it goes.

johnsanc-win10vm.xml

  • Like 1
Link to comment

Just a quick update - So far so good with 16GB allocated to the VM (25% of my total RAM in the system). The VM has been up longer than it ever has been so far with using virtiofs.

 

@VegChan - Can you share your setup and findings as well so far? I'm curious if you were also using either 32GB+ and/or 50%+ of your system RAM allocated to VM.

  • Like 1
Link to comment

For me it freezes around the 2hr mark. Please find my details below,

M/B:	ASUSTeK COMPUTER INC. ROG CROSSHAIR VIII DARK HERO Version Rev X.0x
BIOS:	American Megatrends Inc. Version 4201. Dated: 04/26/2022
CPU:	AMD Ryzen 9 5900X 12-Core @ 3700 MHz
HVM:	Enabled
IOMMU:	Enabled
Cache:	768 KiB, 6 MB, 64 MB
Memory:	32 GiB DDR4 (max. installable capacity 128 GiB)
Network:	eth0: 1000 Mbps, full duplex, mtu 1500
Kernel:	Linux 5.19.14-Unraid x86_64

 

vegchan-win11vm.xml

Link to comment

Same here.
Reverted back to use Samba shares, reverted memorybacking setting in the xml and removed virtio-fs from VM.  but I still get a frozen Windows 11 VM every 2 hours also.

 

I should also note while virtio-fs io was setup, i do notice occational studder in terms of responsiveness.  not sure if that is related.

 

M/B:	ASUSTeK COMPUTER INC. ROG STRIX X570-I GAMING Version Rev X.0x
BIOS:	American Megatrends Inc. Version 4403. Dated: 04/27/2022
CPU:	AMD Ryzen 9 5900X 12-Core @ 3700 MHz
HVM:	Enabled
IOMMU:	Enabled
Cache:	768 KiB, 6 MB, 64 MB
Memory:	32 GiB DDR4 (max. installable capacity 128 GiB)
Network:	bond0: fault-tolerance (active-backup), mtu 1500
Kernel:	Linux 5.19.14-Unraid x86_64

node-diagnostics-20221013-1203.zip

Edited by LeoRX
Link to comment
3 hours ago, Taddeusz said:

Looks like I've been having this problem on my Windows 11 vm. Which sucks because I was looking forward to having access to my Unraid shares without needing to go through SMB. Has anyone yet filed an official bug report on this freezing issue?

What hardware do you have.

Link to comment
1 minute ago, SimonF said:

What hardware do you have.

M/B:	ASUSTeK COMPUTER INC. PRIME H370M-PLUS Version Rev 1.xx - s/n: 180323582100011
BIOS:	American Megatrends Inc. Version 3201. Dated: 09/08/2021
CPU:	Intel® Core™ i7-9700 CPU @ 3.00GHz
HVM:	Enabled
IOMMU:	Enabled
Cache:	512 KiB, 2 MB, 12 MB
Memory:	48 GiB DDR4 (max. installable capacity 64 GiB)
Network:	eth0: 1000 Mbps, full duplex, mtu 1500
Kernel:	Linux 5.19.14-Unraid x86_64

 

Link to comment

I don't think its load related either.  majority of the time when it freeze is either when I'm away from the desk and nothing is running or just playing a video on youtube.

I've left task manager running, at the time of freeze, the task manager shows CPU at 2%, Memory 40%, 0% Disk, 0% Network, 0% GPU.

 

I've also tried some troubleshooting of my own.  Initially I was using the Virtio-FS service that was already there,  but I've notice it does not include any dependencies so I created my own service with VirtioIO FS Driver and WinFsp.Launcher as dependencies.  Does not make any difference appearntly.

 

I've left task manager running, at the time of freeze, the task manager shows CPU at 2%, Memory 40%, 0% Disk, 0% Network, 0% GPU.

Edited by LeoRX
added taskmgr stat at the time of freeze.
Link to comment

I did the same, at the time of the freeze there is no indication anything is awry within Windows. Everything is simply completely frozen. No mouse or keyboard input, but the screen is still with the image of whatever was present at the time of the freeze.

 

Just for the hell of it Im trying to leave memory backing config in place but without any virtio-fs references in the XML.

Considering how many people have the exact same issue with this already there must be a pretty big issue somewhere.

Edited by johnsanc
Link to comment

In the attempt to try and generate more log.  I did the following step and the Windows guest have not freezed once in the past 6 hours.

 

1. Edited /etc/libvirt/libvirtd.conf and updated log_level = 3 (and remove the hash that was in front of it).

2. Set Enable VMs to "No", under VM Manager in the GUI and click "Apply".

3. Set Enable VMs back to "Yes" and click "Apply".  At this stage, the VM Manager won't start, so I rebooted the server.

4. Reboot Unraid.

 

That's it.  I'm not seeing any log when click on "View libvirt log" button before or after change this setting.  So I'm wondering if anyone that's having this problem have rebooted Unraid after setting up Virtio-FS.  Perhaps a reboot is all that's needed?  Can someone try and report back?

 

I shall report back if the Windows guest freeze again.

 

Edited by LeoRX
Link to comment

Maybe we should look a the options on the xml.

 

    <filesystem type='mount' accessmode='passthrough'>
      <driver type='virtiofs' queue='1024'/>
      <binary path='/usr/libexec/virtiofsd' xattr='on'>
        <cache mode='always'/>
        <sandbox mode='chroot'/>
        <lock posix='on' flock='on'/>
      </binary>
      <source dir='/mnt/cache/'/>
      <target dir='test'/>
      <alias name='fs0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

Are you able to remove cache and lock options to see if that makes a difference. Or you can change cache to =none or =auto

Link to comment

I haven't tested yet, seeing the issues, but during the weekend if I can help, tell me what to do.
(I have a Win11 VM that is not critical to always run)

 

Hardware is AMD based, X570 chipset, 32GB RAM (I give 12 to that VM),  10Gbit LAN.

 

So if I can help to try scenarios and see what works or freezes, just guide me.

Link to comment

Well my test seems to narrow the issue down to memory backing. I left this in without any references to the virtiofs and it still froze the exact same way in a similar timeframe.

Any ideas on why this would be the case? Or what to try next?

 

  <memoryBacking>
    <source type='memfd'/>
    <access mode='shared'/>
  </memoryBacking>

 

Edited by johnsanc
Link to comment
3 hours ago, johnsanc said:

Well my test seems to narrow the issue down to memory backing. I left this in without any references to the virtiofs and it still froze the exact same way in a similar timeframe.

Any ideas on why this would be the case? Or what to try next?

 

  <memoryBacking>
    <source type='memfd'/>
    <access mode='shared'/>
  </memoryBacking>

 

No but will see if there are any known issues.

  • Thanks 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.