mackid1993

If you remove the virtiofs shares also <source type='memfd'/> <access mode='shared'/> from <memorybacking> in your XML does it go away? If so you may want to see if hugepages help? I was having one core randomly spiking and my mainboard temp spike as well and since I've switched to hugepages it seems to have stopped. It's super weird, but from what I've seen I wouldn't use Virtiofs without hugepages. You also generally don't want your lowest CPU core pinned to the VM, Unraid uses that.

@jortan I took a look at virsh allocpages and I wouldn't suggest using it unless done right after a fresh boot. I have 128GB of RAM in my server and as a test tried to allocate another 4096 2MB pages and it failed, likely due to fragmentation. I could drop caches and compact, but I'm going to stand by setting this at boot and either allocating a little extra for dockers and then capping the memory of offending containers or finding a way in the config for a particular container to disable hugepages. It seems the majority of dockers that use hugepages by default are database engines like Postgres and Phpmyadmin, I read Apache was another culprit but I'm not sure if that was someone using Apache with mysql which would make more sense than Apache on it's own. I'd say if you aren't running a database, dockers stealing hugepages shouldn't be an issue for you, and if you are running a database it's probably smart to allocate hugepages for it so it performs better! It's a win win, hugepages rock!

Oh that's a good point. When I get up tomorrow I'll add a note about that in my guide and mention you. Thanks!

There is an issue with HTML code blocks in dark mode. For example. <div class="test"> <p style="text-align: center;" >Some Text</p> </div> Below is how this post renders in dark mode:

So you wouldn't want to do this in Unraid. There are many ways to set hugepages but setting the argument in the Syslinux config makes it persistent after every boot. You could add virsh allocpages to your go file and that should work as well but I think adding the arguments to the Syslinux is cleaner especially since nothing in Unraid is persistent unless it is stored on the flash. Moreover I wouldn't suggest using the default settings for Virtiofs because of the performance issues that build up as the memory usage settles in. Enabling hugepages makes a huge performance difference.

Maybe try to boot into a Windows installer and try uninstalling Winfsp from the command line to disable Virtiofs. You'll have to search around for the commands for that, but I believe dism can uninstall programs. A bluescreen is an issue with Windows and I personally have never seen Virtiofs cause this. I suggest keeping regular backups to make restoring from issues like this easier. I use Macrium Reflect Home Edition to take images of my VM and keep an iso of their rescue media with the Virtio drives baked in in /mnt/user/isos so I can easily revert my VM to my last backup if I break something.

Just a shot in the dark in your VM's XML do you have: <memoryBacking> <nosharepages/> <source type='memfd'/> <access mode='shared'/> </memoryBacking> If so, try taking out: <source type='memfd'/> <access mode='shared'/>

I don't think that's going to work, I tried to run some Steam games off a VirtioFS mount and they don't work. Which makes sense, they aren't NTFS drives it's FUSE and not really optimized to run programs from. It also may be a permissions issue. At least with my script Virtiofs.exe is running as SYSTEM.

Enabling Hugepages on Unraid for use with VirtioFS For anyone interested I thought this would be a good place to write a guide to enable hugepages in Unraid for use with VirtioFS. I've found that this helps decrease latency, especially when the Unraid server has been up for quite a while and also helps with memory fragmentation issues that can lead to other VMs that aren't using VirtoFS from being able to boot up reliably. When <nosharepages/> <source type='memfd'/> <access mode='shared'/> is enabled on a VM regardless of VirtioFS being in use or not it seems to slowly cause system memory to become more and more fragmented. If you aren't using VirtioFS, remove <source type='memfd'/> <access mode='shared'/> from your XML and be done with it, but if you are using VirtioFS read on-- this may improve performance for you. Hugepages should really help with these fragmentation issues that lead to sluggishness over time as it allocates a block of memory for the VM at boot that nothing else can touch (with the exception of some greedy Docker containers, more on that at the end). Enabling it is easy but requires a little basic math, as it reserves memory at boot make sure that you have enough RAM in your system to reserve for this. The Guide The first thing you'll want to do is navigate to the Main tab and click Flash: We need to modify our Syslinux with the highlighted configuration in this exact syntax, placed right after append (don't mess up the syntax or your server will fail to boot!!): We'll want to set: hugepagesz=2M That will set the size of each page to 2MB. Then we'll want to allocate our hugepages for the exact amount of RAM we'll be using for our VirtioFS enabled VM. This memory will be reserved whether the VM is booted or not, so if you have multiple VMs using VirtioFS you'll want to ensure you have enough memory to do this, especially if the VMs aren't spun up all of the time. This is best used for VMs that are always running as the memory will always be reserved and cannot be used by anything else unless this is removed from the Syslinux config. If the VM is always in use this doesn't matter because the memory will be used anyway, but for a VM that is used occasionally I wouldn't suggest this method. My main VM is always running and is the only VM I use VirtioFS on, it has 32GB of RAM assigned (32768 MB). So I'm going to set: hugepages=16384 16384*2=32768 As an example if your VM has 16 GB (16384 MB) assigned you'll want to set: hugepages=8192 Now reboot your server to allocate the memory. Next edit the VM's XML. Find the <memoryBacking> tag. It should look like this: <memoryBacking> <nosharepages/> <source type='memfd'/> <access mode='shared'/> </memoryBacking> Change <nosharepages/> to <hugepages/>: <memoryBacking> <hugepages/> <source type='memfd'/> <access mode='shared'/> </memoryBacking> Now as long as you allocated the right amount of memory assigned to the VM, it should spin right up! Greedy Docker Containers One thing to note, some docker containers may try to steal your hugepages! Apache and Postgres are two that will, if you have those installed it would be prudent to assign a little extra RAM to prevent a condition where the VM won't start. It would also be smart to limit the RAM of the containers that are stealing the hugepages so you don't run into a condition where the VM won't start. For example, if you know a container is doing this because it's keeping your VM from starting and you know it needs about 2 GB of RAM to run properly, assign an extra 1 GB in your Syslinux (because it's multiplied by 2) and add --memory=2G to the extra parameters of the offending container.

I let my system sit a bit longer and took: <source type='memfd'/> <access mode='shared'/> out of the XML. At this point the non Virtiofs VMs don't boot at all. As soon as I add the shared memory backing in they boot right way. It seems that when memory backing is in use on one VM it needs to be enabled on all VMs otherwise after the memory usage settles in other VMs will not start. Edit: Testing further I set vm.min_free_kbytes to 1% of my installed memory, I added: sysctl -w vm.min_free_kbytes=671089 to my go file and rebooted to clear everything out. It was very low before, I'm hoping that this addresses some of the fragmentation issues. Regardless enabling memfd and shared on the other VMs fixed everything, but I don't believe it got to the root of the problem.

Woah so I went and added: <source type='memfd'/> <access mode='shared'/> to the other two VMs and they sped right back up! So freakin strange, @johnsanc maybe try adding that to your memorybacking on the Win 10 VM.

They were 20 and 10 as well, but after letting it sit another 10 hours it got slow again. It almost seemed like that made it last longer. ☹️

Try rebooting your server to clear out the RAM or running echo 3 > /proc/sys/vm/drop_caches and echo 1 > /proc/sys/vm/compact_memory in a terminal. Then let it sit with the 96 GB Win 11 VM for a few hours and see if your Win 10 VM will start. I had this exact behavior, the memory fragmentation made it such that when I reduced the RAM on my Win 11 VM from 32 GB to 16 GB my other VMs would start. I think after you make that change in tips and tweaks you still need to clear everything out that built up and never got released either by running those commands or rebooting.

I can confirm after letting my system sit for quite a while setting vm.dirty_background_ratio to 2 % and setting vm.dirty_ratio to 3 % solves this memory related issue and improves VM performance when using memory backing. @SimonF this issue may affect more than just me as @johnsanc was seeing. After a memory backing enabled VM is up for a while other VMs won't start properly. I also personally noticed slowdowns on my main VM that seem to be related to this. Perhaps something needs to be set in Unraid automatically to modify these values when memory backing is in use. Is this something that can be done for 6.13 to improve VirtioFS support further?

Can you try installing the Tips and Tweaks plugin and setting these values lower, let me know if it helps you with memory backing on? The help text said to set them to 2 and 3 for gaming and streaming VMs and that lower values are better if you have a lot of RAM which in my case 64 GB probably qualifies. I won't know for another couple of hours if this has worked. The command cat /proc/buddyinfo is helpful for checking fragmentation. It shows free blocks of various sizes ranked from smallest to largest left to right. If you have a lot of small blocks and few large ones it indicated fragmentation based on the research I was doing. The Normal zone is most relevant to VMs.

I think this may have something to do with the Disk Cache settings in Tips and Tweaks. I lowered the vm.dirty_background_ratio and vm.dirty_ratio. I have a feeling Virtiofs is caching dirtypages to memory and they are building up and causing this issue. Does that make sense to anyone who knows more than me.

Well thanks so much for your help! When I get my RMA back from Corsair and have an extra 64 GB on hand I'll see if it solves the issue. If not throwing those commands on a crontab to run once a day at like 3 am probably isn't a bad idea.

You just saved my sanity, I thought something was up with my motherboard. So I guess I can't push my RAM to 84% usage without having performance issues? Since in figuring this out I found out my RAM was bad I had to run to Microcenter and buy a kit but I have a RMA in to Corsair right now for my original RAM. I may just put that in my server and call it a day now that I know what the problem is. Thanks so much for your help!!! 😊

Holy crap! That worked. I ran those commands and those VMs boot right away! Is this something can can be done inside Unraid to prevent this. Or should I just add these commands to a user script.

Oh! I never thought of that! Is there a way I can test for that? A command I can run? If so I'll increase the ram in my server to mitigate.

Hey @SimonF I found a pretty interesting bug. It's been driving me nuts for days and it only happens after 8-12 hours of server uptime, it it doesn't occur until the server has been running a while but it's related to memory backing. My configuration is my main Windows 11 VM with Virtiofs enabled (several mounts) with 32 GB of RAM assigned. I then have 2 test VMs without Virtiofs or memory backing, one is the Windows 11 Dev Channel and the other is the Canary channel each with 8GB of RAM. All VMs have access to all 20 threads of my 12700k. I recently set these two Dev and Canary VMs up which is when I noticed the problem. This is happening in 6.12.10 and 6.13 beta 1 so it has nothing to do with Unraid version or your php/bash script or even Virtiofsd. I even found that my memory was failing memtest and after replacing it the issue still occurs. I believe it has to do with memory backing being enabled on a VM. What will happen is after the server is up for a while 8-12 hours or so and I have my memory backing enabled VM booted I can boot a single VM with 8GB of RAM but when I go to boot the second VM with 8 GB of RAM it will hang on boot. At worst it's even crashed qemu after my server was up for a while. At one point I even had all cores on my CPU pegged to 100%. What I found was when I made an identical copy of my main Win 11 VM without any Virtiofs or memory backing the issue goes away. Moreover when it's happening and I type pkill virtiofsd it doesn't clear it up right away which tells me it's a qemu bug and not a virtiofsd bug. Interestingly if I drop the memory on my main Win 11 VM with memory backing/virtiofs to 16 GB the issue also clears up. This entire time my server will have at least 14 GB of RAM free so it's not like I'm out of memory, I have 64 GB. If I drop the two 8 GB VMs to use 4 GB each they boot immediately. Nothing interesting is in the logs from what I can tell. It's just a super weird edge case bug, not sure how to open a report or where to open it.

Memtest revealed I had bad RAM. Seems like that was the issue. Edit: It wasn't! It's a memory fragmentation issue somehow caused by memory backing being enabled on one VM for VirtioFS.

Could having XMP enabled be the issue here. I just disabled it, I won't know for 12 hours... but does this sound like XMP being the issue to anyone?

This ended up happening again. It seems to go away when I reboot my server and then after 12+ hours of uptime I can't start a second or 3rd VM. My primary VM has 32GB of RAM allocated, my server has 64 GB. The other two have 8GB. It's very strange everything is fine up until 12 hours and then the other two VMs won't start. Dialing the RAM on my primary VM back to 16GB resolves everything. Does anyone know why this is?

Hey @SimonF I decided to upgrade to 6.13 beta1 but noticed your php wrapper and bash script isn't there. Is that going to be in beta2? Just curious so I comment those lines out when I upgrade. I modified my go file to not copy rust virtiofsd but it looks like I still have to manually copy your bash script and php wrapper for now.