Windows 10 VM poor performance, 100% CPU after 6.9 upgrade


Recommended Posts

Just upgraded to 6.9 yesterday and noticed that the CPUs pinned to my Windows 10 VMs are pegged at 100%.  Also, performance inside the Windows VM is horrible (slow spinning logon animation, 30 seconds to open Windows explorer, generally sluggish).  Performance on these VMs was excellent in version 6.8.3, it was difficult to tell I was even using a VM through RDP.

 

The two VMs in question are called "hcp" and "media".  I created "media2" after the 6.9 upgrade to see if I could find any differences in the xml.  Installation of Windows 10 was slow on "media2" and its performance is bad too.

 

I've been using this instance of unraid since 2015 and it's been rock solid.  I rely on these VMs for my home business and I can't figure out how to get their performance back to 6.8 .3 levels.

 

Other items of note:

  • The "nvr" VM is powered off
  • I do not pass though a GPU but I do pass through a USB card to "media"
    • I had to perform the steps outlined here after the upgrade to 6.9
  • I upgraded my cache drive a week before the 6.9 upgrade
    • I created a cache pool for the first time during that upgrade
  • htop shows /usr/bin/qemu-system-x86_64 using greater than 100% of CPU.  That's weird, right?
  • My CPU temps seem to stay stable.  Perhaps these are false CPU percentage readings?

 

Any ideas what could be going on?

dashboard.png

htop.png

cpu pinning.png

performance monitor.png

task manager.png

nas-diagnostics-20210304-1940.zip

media.xml hcp.xml

Edited by nukeman
added xml for VMs
Link to comment

I have noticed the same thing. My Win10 VM ran like a top on 6.8.3, and since 6.9.0 it's unbearably slow. I regularly watched streaming video in this VM, and now it struggles even to load the smallest applications. I haven't run numbers on the performance, and would have no baseline to compare them to if I did. Unless some solution turns up, I'll have to go back to 6.8.3.

Link to comment

I can confirm the same.

I've noticed it since I updated from 6.8.3 to 6.9.0 RC2 and it's the same in the release version. I'm going crazy about it. I've tried everything I've found on the internet about it.

I see System Interrupts in task manager a lot and I'm sure that's not good. I nuked my old W10 VM that I worked from home on the entire last year and it was just fine back then.

Installin a new one takes a much longer time than it should. 

I think also the linux VMs are much slower now.

I'll calm down that I'm not alone in this and I'm looking forward for a fix. I'll let you know if I find something for sure.

Link to comment

Hi all. I had this problem too after upgrading to 6.9 this week. My Win10 VM and even unRAID itself were running dog slow. System Interrupts was using 100% CPU in Windows.

For me what worked was changing a setting in the "Tips and Tweaks" plugin. I had "CPU Scaling Governor:" set to "Power Save", which I think is the default for Intel CPU. Changed it to "On Demand", and now my VM and unRAID are running much better.

Disclaimer: I don't understand much of that, but it seems to work. Maybe there's downsides. YMMV. Seeing a lot of posts with this issue, hopefully this helps you.

  • Like 1
  • Thanks 1
Link to comment

Um, so this is embarrassing.  I have "Tips and Tweaks" installed but I don't remember ever changing any settings listed there.  I changed "CPU Scaling Governor" to "On Demand" as you suggested.  I also changed  "Enable Intel Turbo/AMD Performance Boost" to "Yes".  I rebooted the server for good measure.  Performance inside the VMs seems much better and CPU usage on the dashboard has gone down as well.  I'll keep an eye on this for and see if it remains stable.  Thanks to @stor44 for the suggestion, hopefully it helps others too. 

  • Like 1
Link to comment
4 hours ago, stor44 said:

Hi all. I had this problem too after upgrading to 6.9 this week. My Win10 VM and even unRAID itself were running dog slow. System Interrupts was using 100% CPU in Windows.

For me what worked was changing a setting in the "Tips and Tweaks" plugin. I had "CPU Scaling Governor:" set to "Power Save", which I think is the default for Intel CPU. Changed it to "On Demand", and now my VM and unRAID are running much better.

Disclaimer: I don't understand much of that, but it seems to work. Maybe there's downsides. YMMV. Seeing a lot of posts with this issue, hopefully this helps you.

 

Wow you're a life saver. Thank you so much for the solution. It works as expected now.

 

BEFORE

 

CrystalDiskMark_20210304222500.png.95508aaae9244ce4d34518fc21bc8b45.pngbefore.JPG.13a82f4cf7f660a29b5d19fdf7da81b8.JPG

 

AFTER

 

CrystalDiskMark_20210306193727.png.c5d4c0f1e95dca0acd5f05b17aa834f4.pngafter.JPG.f07d8dd2d84b3ac0ac4ac63c3ed5ebef.JPG

  • Like 1
Link to comment
1 hour ago, benfishbus said:

Agreed, it’s night and day. I had already enabled turbo, so the improvement for me was all down to CPU scaling. Would like to understand why this needed to be changed and what other consequences will be. 

On Demand is the Default option. I remember changing it to Power Save a few months ago without noticing a performance impact but hoping for a better power consumption.

So changing it back I expect a more aggressive frequency stepping with a small increase in power consumption. I hope I'm not wrong.

Link to comment
  • 1 month later...

I checked my system and both of them were selected, I've never touched those settings and it feels kind of sluggish though. I added 10 Gigs more of Disk space for a total of 50 Gigs on the SSD cache drive and went a little bit better, but not to well. 

Do you know something else that may help?

Link to comment

I have the same problem, but my CPU Scaling is set to Performance, so changing that didn't solve the problem. The CPU usage and RAM usage in task manager inside the Windows VM show low to medium usage, and the usage shown in Unraid is also equally low, and yet the Windows VM is super choppy and laggy even in things as simple of opening file explorer or simple web browsing. It was running perfectly fine before the upgrade to Unraid 6.9.1, so that's the only thing I can think of that could have caused this. It seems so weird that it would lag like that when the CPU usage, RAM usage, Disk usage, etc. are all fairly low according that task manager and Process Lasso in Windows, and the CPU clock speed is at 4.8 GHz (checked using `grep MHz /proc/cpuinfo` in the unraid terminal).

 

Edit: After updating from 6.9.1 to 6.9.2, the problem seems to be fixed. I will come back and add more if the problem reappears

 

Edit 2: The problem came back about a day later, it is intermittently lagging with occassional moments of perfect smooth performance. I have tested while viewing the resource usage in Unraid and there is no correlation at all between changes in Unraid CPU usage or memory usage vs. the weird lag inside the VM, and same with the resource usage reported by task manager. It is like there is a bottleneck somewhere, but there is not sign of additional load anywhere and it was running flawlessly on Unraid 6.7.X and 6.8.X, 

I also tried update the VFIO guest drivers, and updating graphics driver in Windows, neither of those had any impact on the problem

 

I may make a separate thread for my scenario, since my computer's symptoms are now starting to deviate from the OP of this thread

Edited by NerdyGriffin
New information to add
  • Like 1
Link to comment

Likewise I had Windows 10 installed and now it rarely starts. I have just tried to create a new Win10 VM in case my previous one corrupted but no luck, it keeps dying on me.

 

I'm on 6.9.2 and have made the Tips and Tweaks change to Performance and Turbo enabled.

 

All my cores end up reaching 100% even if I only allocated one or two, then the VM dies.

 

EDIT: Ubuntu 20.04 VM is fine, just the Win10 one that won't work

 

Edited by phoebus
Link to comment

Hey Folks,

 

For anyone running out of ideas, many of these issues sound exactly like what I was experiencing. CPU usage normal, plenty of ram and disk space. Set to Performance and Turbo enabled. My VM was so slow and experienced so much lag that it was pretty much unusable. It came down to Interrupts in my Win10 VM getting messed up in 6.9+. This is what I did to fix it.

 

You can probably skip right to step 5, as I'm pretty sure that's what fixed it, but if you want to be extra careful, it doesn't hurt to clean up all the video drivers as well (I had a bunch of different video cards/drivers in the system, which is why I thought steps 1 to 4 might help as well).

Link to comment
On 4/8/2021 at 4:46 PM, NerdyGriffin said:

I have the same problem, but my CPU Scaling is set to Performance, so changing that didn't solve the problem. The CPU usage and RAM usage in task manager inside the Windows VM show low to medium usage, and the usage shown in Unraid is also equally low, and yet the Windows VM is super choppy and laggy even in things as simple of opening file explorer or simple web browsing. It was running perfectly fine before the upgrade to Unraid 6.9.1, so that's the only thing I can think of that could have caused this. It seems so weird that it would lag like that when the CPU usage, RAM usage, Disk usage, etc. are all fairly low according that task manager and Process Lasso in Windows, and the CPU clock speed is at 4.8 GHz (checked using `grep MHz /proc/cpuinfo` in the unraid terminal).

 

Edit: After updating from 6.9.1 to 6.9.2, the problem seems to be fixed. I will come back and add more if the problem reappears

 

Edit 2: The problem came back about a day later, it is intermittently lagging with occassional moments of perfect smooth performance. I have tested while viewing the resource usage in Unraid and there is no correlation at all between changes in Unraid CPU usage or memory usage vs. the weird lag inside the VM, and same with the resource usage reported by task manager. It is like there is a bottleneck somewhere, but there is not sign of additional load anywhere and it was running flawlessly on Unraid 6.7.X and 6.8.X, 

I also tried update the VFIO guest drivers, and updating graphics driver in Windows, neither of those had any impact on the problem

 

I may make a separate thread for my scenario, since my computer's symptoms are now starting to deviate from the OP of this thread

I have the same issues. VMs been rock solid on 6.8.3 and upgraded to 6.9.1 and then 6.9.2 and the load is generally low on the VM but super laggy after about 24hrs of uptime on the VM and/or the unRAID server. I have no GPU passtrough though.

 

Edit: At glance, it seems like changing the scaling governor actually helps. Strange.

Never been an issue for me earlier. Had it schedueled for "power save" from 0100-1600hrs and "performance" 1600-0100 hrs. Changing it to "on demand" reduced VM lag as it seems. 

Edited by iamnypz
Link to comment
  • 3 weeks later...

I have tried literally all of the suggestions in this thread, and non of them have fixed the problem for me. It used to be perfectly find in Unraid 6.8, and now I cannot play games anymore because the whole computer will just freeze for about a second and then jump up to normal FPS, and then freeze again a second or two later, and then repeat until I `Alt+f4` out of the game. Even just web browsing it laggy, every time I try to open the file explorer or a web browsers it freezes up for a few seconds before going back to seemingly normal

 

The task bar usage doesn't show anything unusual, and these lag spike are unpredictable and do not seem to correlate with anything that is going on inside the VM or in the server (docker, other VM's, etc.)

 

I just wish I knew what the cause was, even if I cannot fix it, 

it is super annoying having it behave like this and there is seemingly no cause other than the Unraid update nearly a month ago.

Link to comment

I have a test machine with a Nivida K4000. Created a new test VM Win10 and it seems to freeze. If i pause and resume it seems to continue then freezes again. VM Manager says it running so not pausing.

 

CPU usage becomes near zero when this happens. after a few pause/resume the VM may restart.

 

I have noticed this is only an issue if I have a sound card setup. VM was originally built on 6.9.2 so downgraded to 6.8.3 and it works fine with Sound Card.

 

Upgraded to 6.9.0 issue appears. Remove sound card VM is fine.

 

Not sure if the issue is specific to Nvidia or other sound cards/devices.

 

Will start to investigate if know issues with QEMU 5.1 and sound cards in Win10, Will build a Ubuntu VM to see if issue is only Windows Specific.

 

Any one else able to try testing without a sound card passed thru?

 

 

Link to comment
On 5/3/2021 at 12:21 PM, SimonF said:

...

Any one else able to try testing without a sound card passed thru?

 

 

Out of curiosity, are you referring to setting the "Sound Card" dropdown menu to the device corresponding to the Nvidia GPU, or do you mean passing through a PCIe soundcard?

 

If you are referring to the "GPU sound card", then I will try out changing that setting on my own computer and let you know what happens. If not, then I guess there isn't much I can do to help

 

I used to pass through a PCIe soundcard to my Windows VM, but I have since bought an analog mixer board that has a USB DAC built in, so now am using that mixer as my audio for the Windows VM and I gave the PCIe soundcard to a Volumio VM instead. I made that switch a month or two before the update to Unraid 6.9, so it probably isn't relevant here, but I figured I'd mention it just in case

Link to comment
3 hours ago, SimonF said:

Yes the GPU sound

I made the change (set "Sound Card" to none in VM template) and unfortunately it made no difference for me.

 

Though I would say that based on your description of your problem and mine, we are dealing with a completely different issue

You seem to have full freezing/crashing/restarting, whereas mine is technically "stable" but it just freezes or stutters momentarily, but I haven't had an crashes or restarts.

 

Mine is technically functional, but the performance is seriously handicapped or bottlenecked compared to what it did before the 6.9 update.

 

 

Link to comment

I am actually running a game right now as a test, and if I don't touch anything at all, it sits smoothly at 60 FPS, but if I move the mouse around it freezes/stutters...

 

Update: Oh wow, I tried moving the character in game with the keyboard while intentionally keep the mousing still, and there is absolutely no lag at all with keyboard input, so somehow it is slightly better than before, in the sense that only the mouse triggers game lag? This still does not explain the lag problems outside of games though... It still has lag just typing in the powershell terminal, web browsers, etc....

 

Edit: This is of course just one game and not a very rigorous test, so not enough to make assumptions about the cause of the problem

Edited by NerdyGriffin
Added a clarification or disclaimer
Link to comment
10 hours ago, NerdyGriffin said:

...

Edit: This is of course just one game and not a very rigorous test, so not enough to make assumptions about the cause of the problem

 

To add some new information: I have since done some more serious testing using benchmarks such as 3DMark, and the results were possibly informative. 

Qualitatively looking at the graphs, the GPU benchmarks showed completely normal performance, but the CPU benchmarks showed a massive and obvious drop in performance compared to what I have seen on the same computer in the past. By massive, I mean the scores or FPS or whatever measure were about 20-40% of how this system performed last time I tested it (which was probably a year or more ago).

 

My interpretation of this: Benchmarks showed that it is most likely a CPU bottleneck or emulation overhead of some kind, because the performance only drops/stutters during CPU-focused workloads (load inside the VM only).

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.