meep

I recently spent some time improving the responsiveness of a Windows 11 VM on Unraid (Threadripper + RTX 3090 passthrough). The VM was usable but always felt slightly sluggish since a recent system transplant to a Threadripper 3960X based system., especially around UI responsiveness and window load times. After working through it properly, a few key changes made a noticeable difference: Main steps: Cleaned up CPU pinning (contiguous cores, proper SMT pairing, avoided CPU 0) Ensured CPU isolation (isolcpus) aligned with pinned cores Enabled hugepages and attached them in VM XML Disabled VBS fully (this was the biggest improvement) Tidied Hyper-V settings in XML (relaxed, vapic, spinlocks, vendor_id) Confirmed GPU was using MSI interrupt mode Verified CPU scaling was set to performance The biggest single gain came from disabling VBS. Everything else contributed incrementally. The VM is now noticeably more responsive, though still not quite bare metal. Next step for me is fixing NUMA exposure in BIOS, which should close that gap further. I’ve written up the full process with details here: One Box To Rule Them All: Improving Windows 11 VM Performance on Unraid (Threadripper) Hopefully useful to someone else running a similar setup.

72 hours with no sleeps. Considered fixed. Likely cause was a failing USB boot drive.

So I have an update; Came to my desk this morning and my server was still chugging away, not asleep. I checked the system log and it's clean as a whistle. While I don't have a definitive action I took to effect this change, I did encounter a significant problem since I last posted that's more than likely related if not the root cause: a failing boot thumb drive. Through the various restarts I conducted yesterday to check BIOS settings etc., I started experiencing GUI lockups of increasing rapidity a couple of minutes after restart. Checking the logs, I noticed reports of USB access issues leading to errors with device SDA, my boot drive. I tried swapping it around between ports, changing between efi and non efi boot etc., to no avail. I bit the bullet and transferred to a new boot drive (my 4th in 16 years, not bad going). Boot up was noticeably snappier and the USB errors stopped immediately. It looks like my USB boot drive had been on its way out. I was possibly missing the errors as they were getting lost in log spamming from another failing unassigned device in my older system, and I'd implemented a user script to clear syslogs overnight to prevent the system crashing from full logs. I'm not sure how the failing USB caused the system to go to sleep overnight, and maybe its unrelated. I'll keep everything as is for a few days for observation before I attempt an upgrade to 7.1.x. As this system has had a complete heart & lung transplant (MB, CPU, RAM, GPU, Case, Cooling, Some drive swaps) I would expect little niggles here and there so will remain on high alert. Fingers crossed. Thanks for all the input.

Yes indeed. It's conveniently placed but there's nothing brushing against it. I just checked and a short press of the power button will cause a full shutdown. (Spacebar will not reactivate it, so it's not asleep like the overnight issue I'm experiencing). I'll have another look through Bios now that I'm restarting to see if I've missed anything, otherwise will let it do its thing overnight tonight with full logs in place and see what's what tomorrow.

No pets, it's in my office. Though I will check what happens on a short power button press to see if that replicates the status I find the system in. Thanks

Will check power contacts for sure, though the system has logged the same entry at 4.40 for the past 2 nights, then nothing more I til I wake it up, so unlikely to be a freak contact. Won't rule it out though.

I'll see what happens tonight. I had a script that deleted system logs overnight as they were filling up on my old server and causing it to crash. I've disabled that script as the logging issue is no longer happening on this newer machine. So tomorrow I hope to at least have more logs of what's happening on the build up to the sleep event.

Yes, very sure. Fans switch off, server cannot be reached on network but a tap of the spacebar brings it back to life.

No, not installed and never previously installed. I was thinking of installing it and setting it to 'never sleep', if that's even a thing, but thought I'd ask first.

Hi folks Looking for pointers on this I 've just upgraded my unraid system from a threadeipper 2950 to a 3960X on a ROG Zenith II Extreme Alpha motherboard. No changes to drives or USB boot stick. Everything went very smoothly and all services and functionality is as before. However, I've noticed that my system is now going into sleep mode overnight. Whenever I get to my desk in the morning, it's sleeping. A tap on the keyboard will wake it up. I'd prefer to not have the system sleeping as it runs some home automation VMs and Dockers. Here's the log after I wake it up; Last action was at 4.40am, until I woke it around 8.43. I had a good look around the MB bios but don't see anything obvious in any power settings. I don't have anything sleep related in my unraid setup either. Any tips on where to look? I'm still on 6.12.13. I'd panned to update to current after this hardware change, but would like to get this issue solved before doing so. Thanks for looking.

@HHUBS I don't believe an isolated CPU can be utilised by the OS. The help on CPU Isolation settings says the following; CPU isolation allows the user to specify CPU cores that are to be explicitly reserved for assignment (to VMs or Docker containers). This is incredibly important for gaming VMs to run smoothly because even if you manually pin your Docker containers to not overlap with your gaming VM, the host OS can still utilize those same cores as the guest VM needs for things like returning responses for the webGUI, running a parity check, btrfs operations, etc. That would indicate to me that an isolated core is reserved exclusively, and not used by OS. It's typically recommenced that you never isolate, assign or pin CPU 0 as this is favoured by unRaid for OS specific tasks and processing. If you have CPU 0 pinned or assigned to Docker or VMs, that might contribute to your 100% usage issue? As always, I could be wrong, and other more knowledgeable may row in.

I use HDBaseT to run displays and peripherals from a VM on my unRaid server to a desk in a different building. I write a bit about it here; https://mediaserver8.blogspot.com/2019/07/routing-vms-anywhere.html I'm currently pushing a single display 1920x1080 @ 60Hz, and you'd need one transmitter / receiver per display. Not sure if they hit 144HZ, but you could check specs on AV Access devices, or similar technology. Mine runs across about 60M of Cat6A cable and is very stable.

Ah, something that must not have come through when I migrated, or perhaps something new. Is this what you refer to? So maybe my Switch Ultra could have worked after all?

The device shows as adopting; Then goes offline for a moment; Then repeats on a loop. On the first try, my logs show success; But the adopting / offline loop just keeps going. Docker Inspect shows the docker IP to be 172.17.0.10 When I had the Ultra, I couldn't SSH into it (not supported), but your post just reminded me this was a switch with SSH supported; I did a set inform to the IP of my unraid server, et voila, it worked! Though that brings up my next question....... Whenever I have occasion to stop and restart this docker, my USG-Pro-4 'forgets' its inform UL and I need to log in to the device UI and reset it. I suspect I'm going to have the same issue with this switch. Any thoughts on that?

I migrated from the legacy Unifi docker to this a few months ago and all seemed well, but now I'm having trouble adopting devices. (on repo /unifi:8.3.32-unraid) I had a Switch Ultra that started acting up and went into some kind of adoption / offline loop, and though it worked OK , and showed as adopted in logs, I could not access or configure it in the UI. I contacted Unifi support who ultimately requested that I DMA the switch. However, Now I've acquired a USW-Pro-24-PoE and it's doing the exact same thing!! I like running my controller in unRaid, but with the deprecation, migration hassle, and now this nonsense, I'm thinking of abandoning it and getting a Unifi key altogether. Any insights?

Oh sure. This was not a response to anything, just a general interjection.

A bit late to the party, but I found the process of migrating from old Unifi-Controller to this a little obtuse with a few speed bumps along the way. I documented my process for my future self, but it may help others. https://mediaserver8.blogspot.com/2024/06/migrating-deprecated-unifi-docker-on.html

Hi, sorry for the late reply. I’m not sure why I didn’t receive a thread update notification. Yes, I still have the card, and managed to get it working reliably in my system by shuffling around some cards / slots. However, it really is a nice to have as I’ve lots of other USB controllers I’m not using. Would consider sale.

So after a full weekend, and re-enabling all the various hardware and apps, the server has been stable. It's reasonable to say that disabling c-states was the fix for the issues I was encountering. These have been enabled since I built the server in August 2019, and the system has worked perfectly right through to 6.11.5. Only when I upgraded to 6.12 did regular crashes start, and these seemed to escalate in frequency with every point release I installed. At least now, thanks to the exhaustive testing, I've identified and removed a few CPU intensive plug ins I didn't really need, and identified a faulty SSD, so there's that at least.

I believe power supply idle control was already set to normal, but I'll double check when I next reboot . I want to have the server running for 24 hours without issue before restarting and adding back some if my expansion cards.

So we might have a winner here..... It's an oldie, but a goldie. Working through any and all possible troubleshooting, I disabled c-states in my bios and have had my first error free night in several weeks Not fully confirmed yet, and super confusing why this would impact my system now and not several years ago when this was actually a thing. Will report back and hopefully close out the thread if this turns out to be the actual solution.

That's the plan. Though I see quite a few similar reports here but also on Reddit. I think there's something fundamentally problematic in 6.12.x

I asked CHAT GPT for an analysis. Here's what I got back..... It seems like you're experiencing a recurring issue on your UNRAID server, and I'll walk you through the analysis of the provided logs. The logs indicate that there is a problem with your system's kernel, and it's encountering a "general protection fault." This is a type of error that occurs when the kernel detects that a process is attempting to access a part of memory it's not allowed to, often due to a software bug. Let's break down the information in the log: Timestamp and Process Information: The log entry starts with a timestamp indicating when the issue occurred. It mentions the process name (disk_load) and the PID (Process ID) of the process encountering the issue. Hardware Information: It provides information about the hardware, including the motherboard model and BIOS version. This is important because hardware-related issues can sometimes lead to kernel faults. Stack Trace: The RIP line indicates the Instruction Pointer at which the issue occurred. In this case, it points to the list_del function in the kernel code. The Code section shows the assembly code at the RIP location. The RSP register contains the Stack Pointer at the time of the fault. Registers like RAX, RBX, RCX, etc., show the values stored in those registers when the fault happened. Call Trace: This section provides a call stack trace, showing the sequence of function calls that led to the issue. It starts from the top-level function (list_del) and goes deeper into the kernel functions. Modules Linked In: This lists the kernel modules that are currently loaded. Kernel modules are pieces of code that can be loaded into the kernel to add functionality or drivers. Additional Information: The logs also indicate that this issue is affecting CPU 21. There are mentions of "rcu_preempt detected stalls." RCU (Read-Copy-Update) is a synchronization mechanism in the Linux kernel. These stalls might be related to the issue you're experiencing. Based on this information, it's clear that the issue is occurring in the kernel code, specifically in the list_del function, which is part of a linked list manipulation. The most likely cause of this issue is a software bug in the kernel or a kernel module. Kernel bugs can be very complex to diagnose and often require the expertise of kernel developers. Here are some steps you can take to further diagnose and potentially resolve the issue: Check for Updates: Ensure that your UNRAID server is running the latest kernel and kernel modules. Sometimes, updating the kernel can resolve known issues. Check for Specific Software: Determine if the issue started happening after installing or updating specific software or kernel modules. Try disabling or reverting those changes if possible. Kernel Debugging: Consider enabling kernel debugging and collecting more detailed logs. Kernel developers can use these logs to identify the exact cause of the issue. Community Forums: Check UNRAID or Linux kernel community forums for similar issues. Others may have encountered and solved similar problems. Contact Support: If the issue persists, consider contacting UNRAID support or the support channels for your specific hardware components. They may have specific insights or solutions. Hardware Testing: In some cases, hardware issues can manifest as kernel faults. Test your hardware components (e.g., RAM, CPU, motherboard) for any problems. Keep in mind that diagnosing and resolving kernel-level issues can be challenging, and it's essential to take precautions to avoid data loss during troubleshooting. If you're not comfortable with these steps, seeking assistance from experienced professionals or support channels is advisable.

@Dimtar I'm on IPVlan (not macvlan) @SirLupus I haven't tried that, but like you, I need SMB so not really an option I've now spent DAYS peeling back the onion, removing all cards etc. and adding them back in one at a time. I thought I'd solved it in that I identified a bad SSD (that I wasn't even using) and removed that and seemed to gain some stability. However, overnight last night, it all came tumbling down again (during a parity check). I'll paste a bit of the log captured below, but it looks like something is tripping up the kernel and then continues to generate a kernel exception every 3 mins exactly. In a previous version of this, I could see these were reporting issues in smartctl, which led me down the path to find the bad drive, but here I have my 3 minute exceptions back, and no smartctl references. Really stumped. Next thing is move back to 6.11.x, but I cant be staying on that forever (assuming it works). What's up here @unraid ??? Here's the start of the issue overnight. The 3rd one just keeps repeating every 3 mins exactly until the whole system locks up, or at the very least, the GUI freezes out and becomes unusable.

I would need to go all the way back to the last 6.11.x release, as thats the last time I had stability. I have the same inclination, which is why I'm currently focised on removing hardware, and will look into RAM, CPOU and Drive connections next.