Unraid lockup on vm shutdown


bigjme

Recommended Posts

Hi Everyone,

 

So i have been having this issue for a while and have yet to find the cause. Every so often, if i shut down a vm the entire unraid machine will lock up and only a power reset will fix the issue.

This issue is becoming more and more apparent especially when windows reboots itself for updates and locks up the entire server.

 

Again this is sporadic and may happen once every 10 shutdown/reboots but it is still an issue

Has anyone else faced these sorts of issues? Below is some information about my VM's

 

VM1:

 

Windows 10

100GB OS drive

100GB Storage drive

1 USB 3.0 Controller passed through

Running 7 of 16 cores

12GB of 32GB Ram

GTX 750Ti

USB sound card accessed via the usb hub

 

VM2:

 

Windows 10

100GB OS drive

100GB Storage drive

1 USB 3.0 Controller passed through

Running 7 of 16 cores

16GB of 32GB Ram

GTX 780

Onboard audio

 

 

Unraid has 2 cores and 4GB ram allocated to it and it only runs network shares (no plugins or dockers)

The lockup issue is caused by both vm's and i have had instances where shutting down a vm crashes the unraid gui but the server is still live.

 

Its a strange issue.

 

Regards,

Jamie

Link to comment
  • 5 months later...

Almost certainly related to GPU passthrough (it could be your USB controller also, but not as likely).

Can almost guarantee if you remove the GPU from the VM and test headless, you'll be able to reboot/shutdown a gazillion times without issue.

However this doesn't exactly fix the issue, but it does isolate it.

Unfortunately I don't have your specific GPU's, so I cannot provide much in the way of solutions for your specific setup.

You can attempt to pass the rom for the card, instructions here http://lime-technology.com/wiki/index.php/UnRAID_6/VM_Management#Edit_XML_for_VM_to_supply_GPU_ROM_manually however I'm uncertain how much that is common practice anymore for "stubborn" cards.

Either way attempt that, and report back.

 

Link to comment

  • After installing the beta, but before rebooting, 2 Windows VMs with GPUs were both able to shutdown without locking the whole system
  • I did make some CPU assignment changes recently; I isolated most of the CPU cores used by VMs so that unRAID would not use them
  • After reboot into beta 21 I'm totally broken because I didn't disable autostart for my VMs

Specifically, I can't startup the VM options in unRAID, I get no error messages or other helpful information, I can't find the configuration files for my VMs on the box, and so I can't fix it.

 

I also don't see my NVMe SSD, as I did when running the Beta previously, but it doesn't show up in lspci anymore either.

INLINE EDIT: This issue was due to a bad motherboard; the PCIe slot the NVMe SSD was in was failing intermittently.

 

Some background info:

  • My GPUs are a Nvidia GT 710 and Nvidia GTX 950, with a Nvidia GT 9500 in slot one for the host. I seem to have the issue that I must leave a GPU for the host to be bound to
  • I started out attempting to use the beta, but was not able to install Windows VMs successfully, and reverted to 6.1.9

My thread for this beta issue is at: http://lime-technology.com/forum/index.php?topic=49042.msg470409#msg470409

Link to comment

I had a similar problem where my unraid server would lockup when i shutdown a vm with usb passthrogh.

 

With only gpu passthrogh it locked up for ~2 seconds (which is completely normal) but with usb and gpu passthrogh it locked up for ~20 seconds and broke four usb ports until i shut down, turned off the psu and started the host.

 

If the lockups still happen you can try this fix: https://lime-technology.com/forum/index.php?topic=36768.msg475897#msg475897

 

If this still doesent fix the lockups you can try to stub the GPU (and its soundcard) as well.

 

EDIT: On beta 22 you have to create a share called "system" for the VMs to work

Link to comment
  • 2 weeks later...

I'm having the same problem. 1 of 5 times I boot my win10 VM (passthrough USB and GPU) the server locks up, and my only option is the power button.

 

Booting during automatic updates MAY be the problem I have, when I find my server in a locked state every now and then. I will try to disable automatic updates (if that's possible now?).

Link to comment

Wouldn't the better solution be to fix/replace the hardware causing the issue?

There are certainly many combinations that work properly to alleviate this issue.

Something about proactively disabling updates to fix an issue that is not actually the root cause, leaving you exposed to possible vulnerabilties seems a bit backwards and ill advised.

I suppose for a short term fix it is reasonable, but certainly not long term.

Link to comment

Ok so let me repeat the fix i did for all of these issues

 

1. I switched to the beta and changed my 1 vm to ovmf - this resolved my shutdown lockup issues

2. I was having other issues where my system would lock up sometimes when playing videos on amazon prime - this was caused by an nvidia driver and downgrading to a slightly older version fixed the issue

 

This has resolved all my issues on the server entirely, i have run the server for well over a month with no issues, no lockups, and no problems at all - i am now running more VMs than i was before with no issues

 

In my instance there was no hardware to fix/replace and it was an issue with drivers and unraid itself

 

I stubbed my usb controllers which allows you to select them easily in the new unraid but have since taken the usb controllers off my vms and am using the libvirt hotswap plugin to add/remove usbs from my vms

 

I don't condone removing windows updated entirely but i will say how to if asked, and if you really need to, there is an option to defer updates which will allow your security updates to come through

 

I hope this helps

 

Jamie

Link to comment

I don't condone removing windows updated entirely but i will say how to if asked, and if you really need to, there is an option to defer updates which will allow your security updates to come through

 

Wasn't specifically at you, sorry if you took it that way!

Thanks for providing the path to your solution, his/her situation may differ but hopefully he/she can get it resolved and not resort to stopping updates for a long period.

Cheers.  8)

Link to comment

I used this tip to turn off updates:

http://www.thewindowsclub.com/turn-off-windows-update-in-windows-10

 

It gives me a notify instead of booting at free will.

 

This is only a quick and easy fix for now, to see if that is actually the problem, or if the system locks up for some other reason. I guess I will know in about a week. I think i have maximum uptime of 15 days on my server before something crashes it completely.

 

I also stopped plex media server and sabnzbd. (I know. stupid to do three things at the same time...)

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.