Hi Everyone,

 

So i've had this issue happen a few times now and i can not resolve the issue myself. Every few days my entire unraid server will lockup.

 

By lockup i mean the following:

• All VM's on the system stop responding (video stays on the screen and just does nothing)
  
• All Dockers (nginx etc.) no longer respond to requests
  
• The unraid GUI is not available
  
• No network shares are accessible
  
• I am unable to connect via SSH or telnet (so the safe shutdown plugin can not be run)
  

 

The only fix to this is to reboot the entire unraid server via a power reset which is really not ideal. Checking my windows VM log, it performed a reboot for updates at 03:15:03 and no further logs were recorded till i rebooted at 11am.

 

I enabled gui notifications the other day and when booting unraid backup this morning i got the following message

unRAID Status: 19-12-2015 00:20
Notice [sERVER] - array health report [FAIL]
Array has 3 disks (including parity)

unRAID Parity disk message: 19-12-2015 00:37
Notice [sERVER] - Parity disk returned to normal temperature
WDC_WD40EFRX-68WT0N0_WD-WCC4E76XEAUN (sdd)

 

I checked all the drives by smart status and they all report that they have never failed any checks. Unraid must have locked up after the disk returns to "normal temperature" however. The drives sit around 36 - 38'C when doing a parity check which seems perfectly fine to me?

 

The only thing i have noticed is that this seems to happen more regularly when shutting down a vm or doing a vm reboot. In this instance i found that the vm mentioned above had rebooted for windows updates and unraid had locked up the second it shut down (before auto restarting). This VM has a PCI-E usb controller and a GPU passed through to it.

 

It is always the same behavior as described above so i never have any access to any logs. Can anyone help with this?

 

Regards,

Jamie

 

Edit:

I have since seen my HDDs getting up to 45'C, i have 8 new fans on order to resolve this issue. This temperature is only reached after hours of gaming but the system does not crash with the higher temps.

No ideas from anyone?

 

I haven't had the issue in a few days but i'm right now too worried to reboot my vms for updates in fear that it may lockup unraid and cause harm to my drives

 

Regards,

Jamie

Tools - Diagnostics.

 

And give us some details about your hardware.

Tools - Diagnostics.

 

And give us some details about your hardware.

Does the Diagnostics store data after a reboot? I wasn't sure if it did or not.

 

Attached is my diagnostics files.

 

Unraid Version: 6.1.6

 

System specs

CPU: Intel 5960X @ 4.4GHz

Memory: 32GB DDR4

Motherboard: asrock x99 ws-e

HDD Array: 3 x 4TB WD Red (1 Parity, 2 Storage)

Other Storage: Samsung SM951 NVMe

GPU's: Nvidia 780, Nvidia 750ti, Nvidia 210

Other Extras: 2 USB 3.0 Controllers

 

Storage Allocation

HDD Array: 5 Shares, stores 2 vdisks for VM's

NVMe Drive: Stores 2 vdisks for VM's

 

VM's

 

VM 1

7 CPU Cores

12GB Ram

100GB Image (On NVME Drive)

USB 3 Controller (passed directly via PCI)

GTX 750ti

 

VM 2

7 CPU Cores

14GB Ram

100GB Image (On NVME Drive)

USB 3 Controller (passed directly via PCI)

GTX 780

Onboard Audio

 

Dockers

Nginx - linuxserver.io

MySQL - linuxserver.io

EMBY Server

 

Plugins

Powerdown Package

Unassigned Devices

 

The issue occured before i started using my NVMe drive so i know its not an issue caused by using that. Before hand the 2 images on that drive were on the HDD Array

 

Do you need anymore system information?

 

Regards,

Jamie

server-diagnostics-20151222-1412.zip

On a side note which may help this issue. I removed my storage vdisk from both my VM's as it was refusing to let the HDD's spin down whilst mounted to the guest so it was basically running my drives 24/7 and boiling them even with a fan blowing on them.

 

I now have my drives much cooler but one weird thing i am noticing is if i manually spin down the disks, then access a network share to boot them up again, the unraid GUI reports no temperatures on the overview but does in the smart status values. It is reporting the drive as active and updating the writes as well.

 

I had this issue again just when shutting down one of my VM's to edit the config. Same as before, i am able to do nothing with the server so was not able to get the log.

 

The VM's do for what ever reason seem much faster now i have removed the storage vdisk even though they are simply using a mapped network drive now instead. Not sure why that is though

 

Regards,

Jamie

I have had same  issues but usually can still telnet it. Still not resolved. Running Dockers, 1 vm.

I had this same thing happen a few days ago on my fairly new unRAID box.  WebUI unresponsive, no telnet, no access to VMs or Docker apps.  However, I was able to attach a monitor and keyboard to it and the command-line worked.

So i had this issue happen again just. I rebooted vm1 for an update the all of unraid locked up.

 

I already have unraid console connected to a screen so i connected a keyboard back to it. The keyboard is not detected by unraid therefore i am unable to run any command lines.

I don't know if this means that unraid itself is not responding at all or if it just wont load the usb after the crash (this keyboard works before the crash if i plug it in)

 

My final attempt now is to pass through all audio devices. As it stands, vm1 had no audio and vm2 had onboard.

I have attached the audio from the 750ti to vm1, and the 780 to vm2 so the audio and video are passed together along side passing oboard to vm2 to see if that makes a difference. If not i am entirely lost as i can not get any access to run the diagnostics.

 

Is there no way to get unraid to store the syslog after reboots? This would be very useful for issues like this and won't take up that much room.

 

Regards,

Jamie

As another updated, i had this issue happen again last night. VM2 rebooted for windows updates and unraid locked up.

Again i tried connecting another keyboard to unraid to dump a diagnostics file and didn't get anywhere. The console has no response at all.

 

I've emailed unraid directly for this but have yet to have a reply. This time i managed to get 10 days before lock-up so it is 100% to do with vm's shutting down.

 

Again, if anyone knows how to get the syslogs to perhaps write to the usb so i can see them after a system reboot that would be great. I'm hoping this issue is resolved in 6.2 but as we don't have a time frame for it, its best i try to get a response.

 

I have even tried unplugging my second lan cable in case it was something stupid to do with bridged networks but that hasn't worked. I'm at the end of my list of things to try. The only thing i have not done is add my pci-e devices to the linux boot so they don't get seen by unraid at all.

 

I don't think this is needed for passing through GPU's and usb controllers but someone may know better and say i need to do this?

 

Regards,

Jamie

 

 

---

 

Edit: So its been less then 24 hours and another lockup. Again VM1 rebooted for windows updates and locked up unraid. Since i can not prevent updates in windows 10, this is getting concerning as i now have parity errors creeping in when i reboot and i'm sure its doing my drives a world of good.

 

I would honestly love any kind of help here as my system feels like it could fall over at any point which isn't ideal

Jamie,

 

Sorry to hear about the problems you are having.  System lockups are difficult to diagnose because we literally have no idea what might be causing them without a log.  What you could try is booting up unRAID and having a monitor and keyboard attached to the system for the unRAID console, then login to the command line and type the following:

 

tail /var/log/syslog -f

 

This will keep the system log output going to your monitor.  If the system locks up again, take a picture of that monitor with the log on the screen so we can see the last messages.  Another thing I would try is removing your NVMe device(s) and using standard AHCI/SATA SSDs or HDDs for your VM vdisk storage, so we can at least determine if that has something to do with it (which I suspect it does).

Hi Jonp,

 

Can i run this command via ssh on my booted session? Unraid has a screen connected at all times just not a keyboard and mouse? - stupid question, answered it myself

I had this issue happening before the NVMe device was being used (running the 2 vm's from the normal storage array)

 

I have attached another keyboard to the system for direct console. As this happens when i reboot / shutdown my vm's, i will try to for it to happen

 

Regards,

Jamie

 

Edit: Typical, 20 reboots and windows updates being installed and no crash yet.

I am getting linux syslogs being dumped on my screens when the vm's boot. Is that normal/won't effect what you need to see?

 

 

Edit 2: On a side note, i am wondering if this is an issue from seabios which will be fixed when 6.2 comes out as i will switch to OVMF. I can reboot like normal and perform shutdowns like normal for now. It seems to take atleast a few hours before this issue starts to manifest

Hi Jonp,

 

Can i run this command via ssh on my booted session? Unraid has a screen connected at all times just not a keyboard and mouse? - stupid question, answered it myself

I had this issue happening before the NVMe device was being used (running the 2 vm's from the normal storage array)

 

I have attached another keyboard to the system for direct console. As this happens when i reboot / shutdown my vm's, i will try to for it to happen

 

Regards,

Jamie

 

Edit: Typical, 20 reboots and windows updates being installed and no crash yet.

I am getting linux syslogs being dumped on my screens when the vm's boot. Is that normal/won't effect what you need to see?

 

 

Edit 2: On a side note, i am wondering if this is an issue from seabios which will be fixed when 6.2 comes out as i will switch to OVMF. I can reboot like normal and perform shutdowns like normal for now. It seems to take atleast a few hours before this issue starts to manifest

The reason I want you to tail the syslog from the local console is that the system crash may happen too quickly for the messages to reach your SSH session.  It may even be too quick for them to reach the local console (in which case we would need a serial cable connection to get at them).

Hi Jon.

 

That is understandable. I've got this running on another screen so I can see it (directly on the unraid console). It may take a few days to happen again but it likewise may not.

 

I will post back the second I find anything. I know this one seems to be a very rare occupancy so isn't a large issue for yourself

 

Regards,

Jamie

I have this same issue where it completely locks up and I even lose video. It appears to be caused by Sonarr and not sure why. It appears to happen after the moving of files from sonarr to the drives. I have tried with a docker and from another pc running sonarr and get the same results.

Need.....dem.....logs.....

 

Gotta get a smoking gun here or we can't help.

How can I get a log if it locks up completely?

Connect the unraid gpu to another screen and run the response from Jonp posted earlier.

 

I have yet to have my crash (it's weird hoping for one )

I have the same symptoms on my new build server.

 

http://lime-technology.com/forum/index.php?topic=45196.0

 

Needs a few hours (overnight) usually before it locks up. Seems to be triggered by restarting a particular VM which has a couple of USB devices attached.

 

I'm currently running the VM without the USB devices passed through to see if it still locks up.

 

tail now running on the console.

Hi Jonp, I just managed to screenshot the console from IPMI as my system crashed with very similar symptoms to the OP. Could you take a look please?

 

Screenshot here

 

http://lime-technology.com/forum/index.php?topic=45196.msg435499#msg435499

 

Ok, so i managed to leave my system overnight and get the crash. I have attached my console.

Weirdly as you can tell, there is nothing after it starts the device 0b.

 

Now for this, i started both of my usbs. They have have a single usb device passed via the unraid gui, devices that aren't ever used.

I noticed that once i started up my vms, the 3rd keyboard i attached for unraid was unusable. It is not being passed to anything i just can not type with it.

 

And as i said, when booting windows, the logo moves across my screen as if there is text being dumped to the other screens (gpu's passed directly to the vm) this only happend once i enabled the tail

 

I know my tail gives no information but it may give some information on another issue that is clearly lurking around?

It seems as if unraid doesn't want to give up control of the other gpus and monitors. Honestly this is what i think is the issue, and will most likely be fixed when OVMF is sorted in 6.2 due to EFI support and easier passthrough but i have no idea

 

Regards,

Jamie

Jamie, do you have xhci (usb3) enabled or disabled in the BIOS? I am just trying with usb3 disabled to see if that makes a difference. Will have to wait until tomorrow morning to see if it crashes.

Usb3 is enabled as my motherboard doesn't pick up my usb 3 boot for unraid without it.

 

 

Edit:

So an update to things. I shutdown my vms to perform the 6.1.7 update and got another system lockup. The server had been running 3 hours

Booted up and updated unraid. Reboot and started my vms. As before, i got the same screenshot i attached before except for a difference.

 

Vm2 on a dedicated screen booted with the logs being printed to the screen.Once windows finished loading windows took full control of the gpu again.

VM1 which uses the same screen as the unraid console (on a different display port) started to dump logs to the screen and did nothing with the vm. The gui showed it as loaded but unraid console kept control of the screen (see attachment for logs)

 

I forced vm1 to shutdown and then started it again, this time it did the same as VM2 and took over control when windows finished loading.

 

Nothing it output to these screens before i boot a vm and the gpus become in use. Clearly my unraid is grabbing hold of the GPUs whilst the vm's are trying to as well. Which would explain an issue if the vm shut down, disconnected the card and tried to grab it again whilst unraid was trying to take over it at the same time.

 

Perhaps jonp can read the above and see if what i have said is possible?

 

 

Edit:

So i just had another crash and again, no updates to the logs on the main screen when i know there should be.

Not sure how to get round this one now?

Its been a while so i wanted to post an update.

 

I am still having the same issue with the log going onto the other screens so i'm still unable to get any logging to post here.

The issue is still occurring but i have noticed a few things since my last post.

 

[*]If i shut down vm2, it doesn't always crash immediately. Once shut down, i can still move the mouse on vm1. If i try to access anything on vm1, even a folder, unraid crashes. (i'm not sure if loading the unraid gui causes the same, this behavior is the same if i shutdown vm1 or vm2)

[*]I have a dedicated keyboard attached to unraid and a dedicated screen. If i boot up a vm at all, the keyboard attached to unraid will no longer type. Before i boot the vm it works fine however. This keyboard is plugged directly into the motherboard and not passed to any vm

 

I'm still expecting this to be resolved with 6.2 and will be re-installing all my vm's to the UEFI enabled ones (can't remember the name)

 

The only thing i still have not tried is editing the boot command to not connect the pci devices going to the vms.

Is this something i should try? Jonp it would be great if you can drop a comment on this one please

 

Regards,

Jamie

from the v6 manual

 

If you rely upon a locally-attached monitor and keyboard to interact with the unRAID terminal directly, you will lose this ability once you create a SeaBIOS VM with a GPU assigned. This is due to a bug with VGA arbitration and cannot be solved. This does NOT affect your ability to access the console using a telnet or SSH session, but local console access directly will appear to be frozen (blinking cursor, but no visible response to keyboard input). It does not matter if you are using on-board graphics for the console compared to a discrete GPU for the pass through to a VM or not. With OVMF, however, VGA isn't utilized, therefore arbitration isn't needed and therefore your console graphics will remain intact. Note that not all GPUs support OVMF as OVMF requires UEFI support on your GPU.

Hi Squid,

 

That rules out my keyboard issue then. I will have to switch to OVMF once 6.2 comes out, i believe it has better UEFI support? i'm not sure

 

Regards,

Jamie