October 9, 20223 yr I upgraded to UnRAID 6.11.1 yesterday and everything went smoothly. However my Windows 10 VM appears to have stability issues and just freezes without any apparent errors as far as I can tell. There is nothing suspicious that I can see in the Windows Event Viewer logs and I don't see anything different in my VM log compared to when it was working fine. I used to have issues after I upgraded to 6.11.0 where the network connection to the VM would drop, but I resolved this by upgrading the virtio drivers to the ones from the latest stable ISO. This issue with 6.11.1 is different though, and I can tell the VM is completely frozen. I usually use RDP for accessing the VM, but if I plug in a monitor the graphics card I have passed through to the VM I can see the time on the windows lock screen is frozen. The only major change I did was enabled the virtiofs feature. I've attached my diagnostics in case anyone can take a peek to see what the issue might be. I am not really sure where to look. tower-diagnostics-20221009-1805.zip Edited October 9, 20223 yr by johnsanc
October 11, 20223 yr I have the same issue. It seems to be an issue with Virtiofs. If i remove it from my VM and it no longer freezes. When the VM freezes, all the cores allocated to the VM are locked at 100%. If anyone has a fix for it, it would be great. Would love to keep using virtiofs. I'm running 6.11,1 with windows 11 vm. Edited October 11, 20223 yr by VegChan Added more information
October 11, 20223 yr Author Thanks for confirming I’m not the only one. I just checked and all my CPUs assigned to that VM are pegged as well once the freeze occurs. i tried disabling the VirtIO-FS service in windows and it still freezes, so that sounds more like a host problem. Perhaps some configuration issue in the XML or bug? I’m open to suggestions since Virtiofs is sooooooo much faster than SMB shares for my use cases. I would love if we could get this VM freeze issue resolved.
October 11, 20223 yr The only thing that stopped the freezing for me was to remove the virtiofs mount from the xml as well as the memoryBacking.
October 11, 20223 yr Author Yes, same here. Unfortunately I cannot find anything useful in any logs that even gives a clue. Perhaps @SimonF has a suggestion for where to look, or maybe he can reproduce the issue as well?
October 11, 20223 yr Community Expert 5 hours ago, johnsanc said: Yes, same here. Unfortunately I cannot find anything useful in any logs that even gives a clue. Perhaps @SimonF has a suggestion for where to look, or maybe he can reproduce the issue as well? My testing has been on a simple VM 8G mem, no external GPU just VNC and as yet no lockups, but no heavy utilisation. Also I have not setup a service I just run the exe in a cmd window for virtiofs.exe What is the setup of your machine, as I can add K4000 to VM. my CPU is 12600K. Edited October 11, 20223 yr by SimonF
October 11, 20223 yr Author Here's the basic info of the host machine: M/B: ASRock X570 Creator Version BIOS: American Megatrends Inc. Version P2.40. Dated: 04/13/2020 CPU: AMD Ryzen 9 3900X 12-Core @ 3800 MHz HVM: Enabled IOMMU: Enabled Cache: 768 KiB, 6 MB, 64 MB Memory: 64 GiB DDR4 Multi-bit ECC (max. installable capacity 128 GiB) Network: eth0: 1000 Mbps, full duplex, mtu 1500 Kernel: Linux 5.19.14-Unraid x86_64 And I've attached my VM XML to this post. Note I changed my memory from 32gb down to 16gb because someone else I was chatting with recently mentioned they were having stability issues with anything 32gb and above. So far the VM has been running for about an hour and half with 16gb without any issues yet. I'll let you know how it goes. johnsanc-win10vm.xml
October 11, 20223 yr Author Just a quick update - So far so good with 16GB allocated to the VM (25% of my total RAM in the system). The VM has been up longer than it ever has been so far with using virtiofs. @VegChan - Can you share your setup and findings as well so far? I'm curious if you were also using either 32GB+ and/or 50%+ of your system RAM allocated to VM.
October 11, 20223 yr Author I spoke too soon. Froze again and CPUs pegged. Also the 0/12 CPU is not allocated to the VM... Edited October 11, 20223 yr by johnsanc
October 11, 20223 yr For me it freezes around the 2hr mark. Please find my details below, M/B: ASUSTeK COMPUTER INC. ROG CROSSHAIR VIII DARK HERO Version Rev X.0x BIOS: American Megatrends Inc. Version 4201. Dated: 04/26/2022 CPU: AMD Ryzen 9 5900X 12-Core @ 3700 MHz HVM: Enabled IOMMU: Enabled Cache: 768 KiB, 6 MB, 64 MB Memory: 32 GiB DDR4 (max. installable capacity 128 GiB) Network: eth0: 1000 Mbps, full duplex, mtu 1500 Kernel: Linux 5.19.14-Unraid x86_64 vegchan-win11vm.xml
October 13, 20223 yr Same here. Reverted back to use Samba shares, reverted memorybacking setting in the xml and removed virtio-fs from VM. but I still get a frozen Windows 11 VM every 2 hours also. I should also note while virtio-fs io was setup, i do notice occational studder in terms of responsiveness. not sure if that is related. M/B: ASUSTeK COMPUTER INC. ROG STRIX X570-I GAMING Version Rev X.0x BIOS: American Megatrends Inc. Version 4403. Dated: 04/27/2022 CPU: AMD Ryzen 9 5900X 12-Core @ 3700 MHz HVM: Enabled IOMMU: Enabled Cache: 768 KiB, 6 MB, 64 MB Memory: 32 GiB DDR4 (max. installable capacity 128 GiB) Network: bond0: fault-tolerance (active-backup), mtu 1500 Kernel: Linux 5.19.14-Unraid x86_64 node-diagnostics-20221013-1203.zip Edited October 13, 20223 yr by LeoRX
October 13, 20223 yr Looks like I've been having this problem on my Windows 11 vm. Which sucks because I was looking forward to having access to my Unraid shares without needing to go through SMB. Has anyone yet filed an official bug report on this freezing issue?
October 13, 20223 yr Community Expert 3 hours ago, Taddeusz said: Looks like I've been having this problem on my Windows 11 vm. Which sucks because I was looking forward to having access to my Unraid shares without needing to go through SMB. Has anyone yet filed an official bug report on this freezing issue? What hardware do you have.
October 13, 20223 yr 1 minute ago, SimonF said: What hardware do you have. M/B: ASUSTeK COMPUTER INC. PRIME H370M-PLUS Version Rev 1.xx - s/n: 180323582100011 BIOS: American Megatrends Inc. Version 3201. Dated: 09/08/2021 CPU: Intel® Core™ i7-9700 CPU @ 3.00GHz HVM: Enabled IOMMU: Enabled Cache: 512 KiB, 2 MB, 12 MB Memory: 48 GiB DDR4 (max. installable capacity 64 GiB) Network: eth0: 1000 Mbps, full duplex, mtu 1500 Kernel: Linux 5.19.14-Unraid x86_64
October 13, 20223 yr I don't think its load related either. majority of the time when it freeze is either when I'm away from the desk and nothing is running or just playing a video on youtube. I've left task manager running, at the time of freeze, the task manager shows CPU at 2%, Memory 40%, 0% Disk, 0% Network, 0% GPU. I've also tried some troubleshooting of my own. Initially I was using the Virtio-FS service that was already there, but I've notice it does not include any dependencies so I created my own service with VirtioIO FS Driver and WinFsp.Launcher as dependencies. Does not make any difference appearntly. I've left task manager running, at the time of freeze, the task manager shows CPU at 2%, Memory 40%, 0% Disk, 0% Network, 0% GPU. Edited October 13, 20223 yr by LeoRX added taskmgr stat at the time of freeze.
October 14, 20223 yr Author I did the same, at the time of the freeze there is no indication anything is awry within Windows. Everything is simply completely frozen. No mouse or keyboard input, but the screen is still with the image of whatever was present at the time of the freeze. Just for the hell of it Im trying to leave memory backing config in place but without any virtio-fs references in the XML. Considering how many people have the exact same issue with this already there must be a pretty big issue somewhere. Edited October 14, 20223 yr by johnsanc
October 14, 20223 yr In the attempt to try and generate more log. I did the following step and the Windows guest have not freezed once in the past 6 hours. 1. Edited /etc/libvirt/libvirtd.conf and updated log_level = 3 (and remove the hash that was in front of it). 2. Set Enable VMs to "No", under VM Manager in the GUI and click "Apply". 3. Set Enable VMs back to "Yes" and click "Apply". At this stage, the VM Manager won't start, so I rebooted the server. 4. Reboot Unraid. That's it. I'm not seeing any log when click on "View libvirt log" button before or after change this setting. So I'm wondering if anyone that's having this problem have rebooted Unraid after setting up Virtio-FS. Perhaps a reboot is all that's needed? Can someone try and report back? I shall report back if the Windows guest freeze again. Edited October 14, 20223 yr by LeoRX
October 14, 20223 yr Spoke too soon. I did manage to get this error in libvirt log. 2022-10-14 05:55:47.004+0000: 23325: warning : qemuDomainObjTaintMsg:6464 : Domain id=1 name='Windows 11' uuid=3e508dc2-73ba-c317-b3cf-099c8f0be7a0 is tainted: custom-ga-command
October 14, 20223 yr Community Expert Do you have or are you able to create a Linux VM on your system to see if the issue is Linux or Windows guest specific?
October 14, 20223 yr Community Expert Maybe we should look a the options on the xml. <filesystem type='mount' accessmode='passthrough'> <driver type='virtiofs' queue='1024'/> <binary path='/usr/libexec/virtiofsd' xattr='on'> <cache mode='always'/> <sandbox mode='chroot'/> <lock posix='on' flock='on'/> </binary> <source dir='/mnt/cache/'/> <target dir='test'/> <alias name='fs0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> Are you able to remove cache and lock options to see if that makes a difference. Or you can change cache to =none or =auto
October 14, 20223 yr I haven't tested yet, seeing the issues, but during the weekend if I can help, tell me what to do. (I have a Win11 VM that is not critical to always run) Hardware is AMD based, X570 chipset, 32GB RAM (I give 12 to that VM), 10Gbit LAN. So if I can help to try scenarios and see what works or freezes, just guide me.
October 14, 20223 yr Author Well my test seems to narrow the issue down to memory backing. I left this in without any references to the virtiofs and it still froze the exact same way in a similar timeframe. Any ideas on why this would be the case? Or what to try next? <memoryBacking> <source type='memfd'/> <access mode='shared'/> </memoryBacking> Edited October 14, 20223 yr by johnsanc
October 14, 20223 yr Community Expert 3 hours ago, johnsanc said: Well my test seems to narrow the issue down to memory backing. I left this in without any references to the virtiofs and it still froze the exact same way in a similar timeframe. Any ideas on why this would be the case? Or what to try next? <memoryBacking> <source type='memfd'/> <access mode='shared'/> </memoryBacking> No but will see if there are any known issues.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.