johnsanc Posted October 9, 2022 Share Posted October 9, 2022 (edited) I upgraded to UnRAID 6.11.1 yesterday and everything went smoothly. However my Windows 10 VM appears to have stability issues and just freezes without any apparent errors as far as I can tell. There is nothing suspicious that I can see in the Windows Event Viewer logs and I don't see anything different in my VM log compared to when it was working fine. I used to have issues after I upgraded to 6.11.0 where the network connection to the VM would drop, but I resolved this by upgrading the virtio drivers to the ones from the latest stable ISO. This issue with 6.11.1 is different though, and I can tell the VM is completely frozen. I usually use RDP for accessing the VM, but if I plug in a monitor the graphics card I have passed through to the VM I can see the time on the windows lock screen is frozen. The only major change I did was enabled the virtiofs feature. I've attached my diagnostics in case anyone can take a peek to see what the issue might be. I am not really sure where to look. tower-diagnostics-20221009-1805.zip Edited October 9, 2022 by johnsanc 1 Quote Link to comment
VegChan Posted October 11, 2022 Share Posted October 11, 2022 (edited) I have the same issue. It seems to be an issue with Virtiofs. If i remove it from my VM and it no longer freezes. When the VM freezes, all the cores allocated to the VM are locked at 100%. If anyone has a fix for it, it would be great. Would love to keep using virtiofs. I'm running 6.11,1 with windows 11 vm. Edited October 11, 2022 by VegChan Added more information Quote Link to comment
johnsanc Posted October 11, 2022 Author Share Posted October 11, 2022 Thanks for confirming I’m not the only one. I just checked and all my CPUs assigned to that VM are pegged as well once the freeze occurs. i tried disabling the VirtIO-FS service in windows and it still freezes, so that sounds more like a host problem. Perhaps some configuration issue in the XML or bug? I’m open to suggestions since Virtiofs is sooooooo much faster than SMB shares for my use cases. I would love if we could get this VM freeze issue resolved. Quote Link to comment
VegChan Posted October 11, 2022 Share Posted October 11, 2022 The only thing that stopped the freezing for me was to remove the virtiofs mount from the xml as well as the memoryBacking. Quote Link to comment
johnsanc Posted October 11, 2022 Author Share Posted October 11, 2022 Yes, same here. Unfortunately I cannot find anything useful in any logs that even gives a clue. Perhaps @SimonF has a suggestion for where to look, or maybe he can reproduce the issue as well? 1 Quote Link to comment
SimonF Posted October 11, 2022 Share Posted October 11, 2022 (edited) 5 hours ago, johnsanc said: Yes, same here. Unfortunately I cannot find anything useful in any logs that even gives a clue. Perhaps @SimonF has a suggestion for where to look, or maybe he can reproduce the issue as well? My testing has been on a simple VM 8G mem, no external GPU just VNC and as yet no lockups, but no heavy utilisation. Also I have not setup a service I just run the exe in a cmd window for virtiofs.exe What is the setup of your machine, as I can add K4000 to VM. my CPU is 12600K. Edited October 11, 2022 by SimonF Quote Link to comment
johnsanc Posted October 11, 2022 Author Share Posted October 11, 2022 Here's the basic info of the host machine: M/B: ASRock X570 Creator Version BIOS: American Megatrends Inc. Version P2.40. Dated: 04/13/2020 CPU: AMD Ryzen 9 3900X 12-Core @ 3800 MHz HVM: Enabled IOMMU: Enabled Cache: 768 KiB, 6 MB, 64 MB Memory: 64 GiB DDR4 Multi-bit ECC (max. installable capacity 128 GiB) Network: eth0: 1000 Mbps, full duplex, mtu 1500 Kernel: Linux 5.19.14-Unraid x86_64 And I've attached my VM XML to this post. Note I changed my memory from 32gb down to 16gb because someone else I was chatting with recently mentioned they were having stability issues with anything 32gb and above. So far the VM has been running for about an hour and half with 16gb without any issues yet. I'll let you know how it goes. johnsanc-win10vm.xml 1 Quote Link to comment
johnsanc Posted October 11, 2022 Author Share Posted October 11, 2022 Just a quick update - So far so good with 16GB allocated to the VM (25% of my total RAM in the system). The VM has been up longer than it ever has been so far with using virtiofs. @VegChan - Can you share your setup and findings as well so far? I'm curious if you were also using either 32GB+ and/or 50%+ of your system RAM allocated to VM. 1 Quote Link to comment
johnsanc Posted October 11, 2022 Author Share Posted October 11, 2022 (edited) I spoke too soon. Froze again and CPUs pegged. Also the 0/12 CPU is not allocated to the VM... Edited October 11, 2022 by johnsanc Quote Link to comment
VegChan Posted October 11, 2022 Share Posted October 11, 2022 For me it freezes around the 2hr mark. Please find my details below, M/B: ASUSTeK COMPUTER INC. ROG CROSSHAIR VIII DARK HERO Version Rev X.0x BIOS: American Megatrends Inc. Version 4201. Dated: 04/26/2022 CPU: AMD Ryzen 9 5900X 12-Core @ 3700 MHz HVM: Enabled IOMMU: Enabled Cache: 768 KiB, 6 MB, 64 MB Memory: 32 GiB DDR4 (max. installable capacity 128 GiB) Network: eth0: 1000 Mbps, full duplex, mtu 1500 Kernel: Linux 5.19.14-Unraid x86_64 vegchan-win11vm.xml Quote Link to comment
LeoRX Posted October 13, 2022 Share Posted October 13, 2022 (edited) Same here. Reverted back to use Samba shares, reverted memorybacking setting in the xml and removed virtio-fs from VM. but I still get a frozen Windows 11 VM every 2 hours also. I should also note while virtio-fs io was setup, i do notice occational studder in terms of responsiveness. not sure if that is related. M/B: ASUSTeK COMPUTER INC. ROG STRIX X570-I GAMING Version Rev X.0x BIOS: American Megatrends Inc. Version 4403. Dated: 04/27/2022 CPU: AMD Ryzen 9 5900X 12-Core @ 3700 MHz HVM: Enabled IOMMU: Enabled Cache: 768 KiB, 6 MB, 64 MB Memory: 32 GiB DDR4 (max. installable capacity 128 GiB) Network: bond0: fault-tolerance (active-backup), mtu 1500 Kernel: Linux 5.19.14-Unraid x86_64 node-diagnostics-20221013-1203.zip Edited October 13, 2022 by LeoRX Quote Link to comment
Taddeusz Posted October 13, 2022 Share Posted October 13, 2022 Looks like I've been having this problem on my Windows 11 vm. Which sucks because I was looking forward to having access to my Unraid shares without needing to go through SMB. Has anyone yet filed an official bug report on this freezing issue? 1 Quote Link to comment
NLS Posted October 13, 2022 Share Posted October 13, 2022 I think they know and we are waiting feedback. 1 Quote Link to comment
SimonF Posted October 13, 2022 Share Posted October 13, 2022 3 hours ago, Taddeusz said: Looks like I've been having this problem on my Windows 11 vm. Which sucks because I was looking forward to having access to my Unraid shares without needing to go through SMB. Has anyone yet filed an official bug report on this freezing issue? What hardware do you have. Quote Link to comment
Taddeusz Posted October 13, 2022 Share Posted October 13, 2022 1 minute ago, SimonF said: What hardware do you have. M/B: ASUSTeK COMPUTER INC. PRIME H370M-PLUS Version Rev 1.xx - s/n: 180323582100011 BIOS: American Megatrends Inc. Version 3201. Dated: 09/08/2021 CPU: Intel® Core™ i7-9700 CPU @ 3.00GHz HVM: Enabled IOMMU: Enabled Cache: 512 KiB, 2 MB, 12 MB Memory: 48 GiB DDR4 (max. installable capacity 64 GiB) Network: eth0: 1000 Mbps, full duplex, mtu 1500 Kernel: Linux 5.19.14-Unraid x86_64 Quote Link to comment
LeoRX Posted October 13, 2022 Share Posted October 13, 2022 (edited) I don't think its load related either. majority of the time when it freeze is either when I'm away from the desk and nothing is running or just playing a video on youtube. I've left task manager running, at the time of freeze, the task manager shows CPU at 2%, Memory 40%, 0% Disk, 0% Network, 0% GPU. I've also tried some troubleshooting of my own. Initially I was using the Virtio-FS service that was already there, but I've notice it does not include any dependencies so I created my own service with VirtioIO FS Driver and WinFsp.Launcher as dependencies. Does not make any difference appearntly. I've left task manager running, at the time of freeze, the task manager shows CPU at 2%, Memory 40%, 0% Disk, 0% Network, 0% GPU. Edited October 13, 2022 by LeoRX added taskmgr stat at the time of freeze. Quote Link to comment
johnsanc Posted October 14, 2022 Author Share Posted October 14, 2022 (edited) I did the same, at the time of the freeze there is no indication anything is awry within Windows. Everything is simply completely frozen. No mouse or keyboard input, but the screen is still with the image of whatever was present at the time of the freeze. Just for the hell of it Im trying to leave memory backing config in place but without any virtio-fs references in the XML. Considering how many people have the exact same issue with this already there must be a pretty big issue somewhere. Edited October 14, 2022 by johnsanc Quote Link to comment
LeoRX Posted October 14, 2022 Share Posted October 14, 2022 (edited) In the attempt to try and generate more log. I did the following step and the Windows guest have not freezed once in the past 6 hours. 1. Edited /etc/libvirt/libvirtd.conf and updated log_level = 3 (and remove the hash that was in front of it). 2. Set Enable VMs to "No", under VM Manager in the GUI and click "Apply". 3. Set Enable VMs back to "Yes" and click "Apply". At this stage, the VM Manager won't start, so I rebooted the server. 4. Reboot Unraid. That's it. I'm not seeing any log when click on "View libvirt log" button before or after change this setting. So I'm wondering if anyone that's having this problem have rebooted Unraid after setting up Virtio-FS. Perhaps a reboot is all that's needed? Can someone try and report back? I shall report back if the Windows guest freeze again. Edited October 14, 2022 by LeoRX Quote Link to comment
LeoRX Posted October 14, 2022 Share Posted October 14, 2022 Spoke too soon. I did manage to get this error in libvirt log. 2022-10-14 05:55:47.004+0000: 23325: warning : qemuDomainObjTaintMsg:6464 : Domain id=1 name='Windows 11' uuid=3e508dc2-73ba-c317-b3cf-099c8f0be7a0 is tainted: custom-ga-command Quote Link to comment
SimonF Posted October 14, 2022 Share Posted October 14, 2022 Do you have or are you able to create a Linux VM on your system to see if the issue is Linux or Windows guest specific? Quote Link to comment
SimonF Posted October 14, 2022 Share Posted October 14, 2022 Maybe we should look a the options on the xml. <filesystem type='mount' accessmode='passthrough'> <driver type='virtiofs' queue='1024'/> <binary path='/usr/libexec/virtiofsd' xattr='on'> <cache mode='always'/> <sandbox mode='chroot'/> <lock posix='on' flock='on'/> </binary> <source dir='/mnt/cache/'/> <target dir='test'/> <alias name='fs0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> Are you able to remove cache and lock options to see if that makes a difference. Or you can change cache to =none or =auto Quote Link to comment
NLS Posted October 14, 2022 Share Posted October 14, 2022 I haven't tested yet, seeing the issues, but during the weekend if I can help, tell me what to do. (I have a Win11 VM that is not critical to always run) Hardware is AMD based, X570 chipset, 32GB RAM (I give 12 to that VM), 10Gbit LAN. So if I can help to try scenarios and see what works or freezes, just guide me. Quote Link to comment
johnsanc Posted October 14, 2022 Author Share Posted October 14, 2022 (edited) Well my test seems to narrow the issue down to memory backing. I left this in without any references to the virtiofs and it still froze the exact same way in a similar timeframe. Any ideas on why this would be the case? Or what to try next? <memoryBacking> <source type='memfd'/> <access mode='shared'/> </memoryBacking> Edited October 14, 2022 by johnsanc Quote Link to comment
SimonF Posted October 14, 2022 Share Posted October 14, 2022 3 hours ago, johnsanc said: Well my test seems to narrow the issue down to memory backing. I left this in without any references to the virtiofs and it still froze the exact same way in a similar timeframe. Any ideas on why this would be the case? Or what to try next? <memoryBacking> <source type='memfd'/> <access mode='shared'/> </memoryBacking> No but will see if there are any known issues. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.