civic95man

That is very strange, but if it seems to be the root cause then its good to know. I guess if you stress test your system without it installed and don't see any call traces then you've found your solution!

I would be surprised if that was the cause; specifically, the VM is isolated from your unraid system. That corsair pump is just another USB peripheral as far as unraid, the VM, and Windows is concerned (much like a mouse or keyboard). If you want to pursue this route then it can't hurt anything.

Just do a flash backup from the web UI [main] -> [boot device -> flash] -> [flash backup] and keep it somewhere safe. If things go sideways, you just copy everything back to the flash drive. I would then update from within unraid

I was thinking of trying the 1G option since your processor supports it, but 2M still works. So you basically tell the kernel to set aside XXXX contiguous block of memory of 2M in size (instead of the default 4K). The kicker here is that some applications, besides the VM, can and will use the hugepages, so plan accordingly. If you want 16G set aside as hugepages then you would put "hugepages=8192", since 16GB / 2MB = 8192. I think the mis-configured hugepages was causing the oom killing spree in those diagnostics. This is in regard to the transition from prior versions of unraid to 6.9-rc; where there were excessive writes to the SSD and part of the solution was to align the partition to 1MB. This would require a repartitioning of the SSD - but you would have to manually invoke it. The issue is that supposedly, 6.8 and earlier doesn't recognize this layout (but I could be mistaken). In either case, just don't format the SSD in 6.9 and you'll be fine. Storage pools WAS introduced in 6.9 and there was a process to revert back to 6.8 and earlier, but I think those notes were lost in the beta release updates somewhere. As always, backup your flash drive before updating. At this point, I think the 6.9 route would be the best choice since it utilizes a newer kernel which should better support your MB and CPU, and possibly get rid of that BTS buffer allocation failure.

Looked through your diagnostics (both) and still see the OOM errors. The first diagnostics, which the system ran for about 2 days, was full of them as you stated. I find it very odd that your memory seems very fragmented, and thus why it can't allocate an order 4 block of contiguous memory - especially after a fresh reboot. Here is a suggestion: have you tried using Hugepages for your VM? It's typically only needed for very large capacities, or if you are suffering performance issues; however, in this case, it's worth a shot. Here is a post about how to utilize it: If that doesn't work then I would suggest either trying the 6.9-rc or adding more memory. The 6.9 series has added pools and changed the format option of the SSD cache, so while it's not a one-way trip, it's not as simple to revert to the prior 6.8 or 6.7 release. With that said, the 6.9-rc seems very stable and should work fine.

So it seems to be VM related. Might be good to post diagnostics AFTER this happens again. That snippet of the syslog left out a lot of details and I saw reference to another OOM error. Also, I looked into your previous OOM error from the first post one last time and I can *kinda* see how it gave you the error. If anyone is curious, technically, you ran out of memory on the Normal zone and couldn't assign a contiguous block (order of 4). I don't know why it didn't use DMA32 zone. Maybe someone else can answer that. seems to be related to intel integrated graphics??? for now, just assume it's nothing. I did find reference to your current issues in an older forum post: Their solution was to nuke the offending VM and start over. I guess you could try that. You could first try removing the xml but keeping the vdisk (assuming you're using vdisks for the VM). If that doesn't work then try creating a new vdisk, keeping the old one. If that doesn't work either, then you could always go with the latest 6.9.0 release candidate. The new kernel might help things out and I think there is a newer release of qemu rolled up in there as well.

I am by no means a network guru, but I *think* you have the subnet mask wrong. Anyway, I had a netgear router that i could never get it to route across two networks. I eventually gave up with that trash and switched to unifi. Best move I ever made!

I had to do this a few weeks back when my logs filled up and didn't want to reboot. If I remember right, I think I was able to delete the current syslog as well. Why? In my case, not entirely sure but it was related to the webgui server and some issues with it - probably because I left a browser window open on a computer for too long. I also had to nuke the web server and restart it. In hind sight, I probably should have just moved the logs off onto the array in case I needed to troubleshoot something. Anyway, I find this to be a good tip in case you don't want to or can't reboot. ymmv

@isrdude Maybe try updating the BIOS, you seem to be a few versions behind. Also try disabling "fast boot" or "quick boot" from the BIOS and see if that helps at all, if its enabled.

sudo echo 1 > /sys/bus/pci/rescan This should force a complete bus scan and re-add the previously removed (disconnected) device. Shouldn't cause any issues with already attached devices. I'm not around a *nix box at the moment otherwise I would try this, but I wonder if this would work: sudo echo 1 > /sys/bus/pci/devices/0000:XX:XX.X/rescan I'm really curious if this will actually reduce your power consumption

Do these error only come up when the VM is active? maybe a second cheap video card if you have room on the board have you tried the 6.9 beta yet? I would only proceed down that route with caution as it isn't declared stable yet. Maybe 6.9beta1 so you aren't messing with pools yet. Last resort you could use a phone or tablet if available to manage the VMs

I didn't see anything listed for an order so i can only assume it doesn't matter. Mine (supermicro mobo) was accessed via the BMC IP address. Within that, it gave me the option to both update the bios there and the firmware. It may also include it's own installer/updater (haven't checked) but it would most likely require a windows/dos environment.

That's from the nvidia plugin which calls the nvidia-smi to get a listing of the available cards. Since you're on stock, that program is nolonger there, hence the error. Its harmless. It will go away if you uninstall the plugin.

It might have been related to when you were using the nvidia build, or it could be from the plugin. if you don't plan on ever using that build again then you can delete the plugin. I see, yes I would have never noticed that! So I had another thought just now, you probably need to update both the BIOS **and** the BMC firmware at the same time for the onboard video to work again. And then hunt through the BIOS menus for that hidden option.

The stubbing is a kernel parameter which is passed at the time the kernel is loaded. So the kernel shouldn't try to touch that card, besides binding vfio to it so nothing else will use it, besides the VM. no stupid questions

This should prevent the card from loading any drivers and therefore the kernel will ignore this card as an option. It *should* grab the next available video adapter which would be the onboard. If like you say, the BIOS refuses to make the onboard video the primary adapter, then you might lose any POST messages and boot menu options. It could also be that the option of selecting the onboard video has moved to another menu in the BIOS. With that said, I looked up your mobo and didn't see that it had any onboard video?!?!?

I assume your are passing that card to your VM? Is it not stubbed?

Well, the next step in troubleshooting would be to boot the system in "safemode" which prevents any add-ons from loading. You could also disable VMs and docker. Then, after your system runs stable with no further page allocation failures, you slowly enable one thing at a time, run for a while to check stability, and repeat. Have you checked if you're using the latest BIOS for your board? It looks like there is a newer version available. This could very well be a BIOS issue in the way the memory is mapped.

Looking at your logs, you need to decode the reason why it was tainted. The letters after it says "Tainted" in the call trace indicate why. G indicates a proprietary module was loaded, W says that a kernel warning was issued at the time the module was loaded, and O means that it was an externally built module, or out-of-tree. Looking further back in your logs when your system comes up shows that the "igb" module is loaded and a warning is issued that it taints the kernel. That module is the driver for your network adapter and was added by limetech to the build. Seeing that such a large number of people use that same adapter, this can be safely ignored. It basically just lets the kernel developers know that if you submit a bug report about the kernel - not the unraid developers - that you have a unapproved configuration. In this case, it does not affect your system.

From what I understand, this isn't something to really "worry" about. Basically, it looks like some process tried to grab 2^4 pages of memory and failed, but was able to get it another way. This seems to result in the way the memory is mapped in your system and shouldn't be anything to indicate a failure or problem. The "order" means it tried to grab 2^4 pages but failed. Apparently if it fails when trying to grab 2^3 pages then the kernel initiates the OOM process. Now if the order is 0, then you have a problem and are truly out of memory. The call trace is there to "help" you figure out why the memory allocation failed. I found all of this information from this page https://utcc.utoronto.ca/~cks/space/blog/linux/DecodingPageAllocFailures There also seems to be ways to help mitigate this but it depends on how much you want to play with options. I guess you could also add more memory too?? oh, and the parts of the log that say the kernel is "tainted" has nothing to do with the memory allocation errors, they are due to the proprietary modules loaded for the nvidia build. Although, the memory issues don't cause the kernel to become tainted, the tainted kernel *could* cause the memory issues. It might be best to boot with stock unraid (not the nvidia build) and see if this still happens. If is still does then try safe mode and work backwards.

Thanks for the help @S80_UK that link was a life saver and I wouldn't have been able to update the firmware without it. Pro tip: don't use the UPS outlets while updating the firmware, the outlets shut off during the update. The update pushed me to the latest firmware (9.4) and I was able to enable MODBUS under the UPS configuration menu. From there it was just a matter of setting up unraid to use the UPS (cable->USB, type->MODBUS). Quick note: it will take a while for the UPS and unraid to talk and update the status. That was my problem as I was getting impatient, thinking that it wasn't working. Just leave it and it should work. Thanks again

good information! I'll check that out when I get home. Judging by that apc link, most models with the exception of a few are able to have their firmware upgraded - just there are some hoops you gotta jump through depending on how old your current firmware is.

I have the same UPS but I haven't yet incorporated it into my setup so I've been following this. Thats disappointing to find out. Would the network management card allow for the firmware to be upgraded? I thought I read somewhere that the card had this capability.

diagnostics? preferably while in the middle of a slow transfer

Is it slow during the entire transfer or starts fast and gradually/suddenly slows down? Its my understanding that SMR drives leverage a CMR cache area to speed up writes but once it fills then any other writes slows to a crawl.