Unraid server crashes overnight, boots up unjoined from domain


ThomasE

Recommended Posts

*edited, link removed and syslog directly attached to post*

 

Hello all, a couple weeks ago our unraid server went down in the middle of the night, and booted back up unjoined from the domain it was previously joined to. I thought it may have been the UPS, as it's a pretty old unit so I replaced it. Fast forward to last night, the same thing happened again. It went down around 3 or 4 in the morning and rebooted unjoined from our domain. I was wondering if someone more knowledgable with unraid could take a look at my syslog and see if there's something in the log to indicate what's going on? Many thanks!

 

Dell T110 II

Xeon E3 1200 V2

16GB Ram

4 - 4TB spinners (2 parity)

2 - 512GB SSD (cache pool)

Unraid 6.7.2

syslog.txt

Edited by ThomasE
syslog attachment
Link to comment
9 minutes ago, trurl said:

I'm not going to that external site and give them my email and agree to their terms.

 

Now that you have been approved you can attach the complete Diagnostics zip file (Tools - Diagnostics) directly to your next post. Have you enabled the Syslog Server feature?

 

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=781601

(

I have not, I will get that setup today. *edit, it is now enabled*

Edited by ThomasE
Link to comment

I don't see anything out of the ordinary in your syslog (though it obviously was grabbed after the reboot so it won't show what led to the crash) but you probably need to check the virtualisation settings in your BIOS if you're planning to run VMs:

Oct 17 09:28:13 BasinNAS root: Starting virtlockd...
Oct 17 09:28:13 BasinNAS root: Starting virtlogd...
Oct 17 09:28:13 BasinNAS root: Starting libvirtd...
Oct 17 09:28:13 BasinNAS kernel: kvm: disabled by bios
Oct 17 09:28:13 BasinNAS root: modprobe: ERROR: could not insert 'kvm_intel': Operation not supported

 

I don't use a domain myself and have little experience of them but I suppose this is the line in the syslog that shows the attempt to join:

Oct 17 09:26:47 BasinNAS emhttpd: req (1): DOMAIN=BASINDESIGN.local&DOMAIN_SHORT=BASINDESIGN&DOMAIN_LOGIN=UNRAID&DOMAIN_PASSWD=&cmdJoinDomain=Join&csrf_token=****************

So it seems to be trying to join but is it being rejected by the domain controller? You maybe need to check the domain controller's logs.

 

Link to comment
5 minutes ago, John_M said:

I don't see anything out of the ordinary in your syslog (though it obviously was grabbed after the reboot so it won't show what led to the crash) but you probably need to check the virtualisation settings in your BIOS if you're planning to run VMs:


Oct 17 09:28:13 BasinNAS root: Starting virtlockd...
Oct 17 09:28:13 BasinNAS root: Starting virtlogd...
Oct 17 09:28:13 BasinNAS root: Starting libvirtd...
Oct 17 09:28:13 BasinNAS kernel: kvm: disabled by bios
Oct 17 09:28:13 BasinNAS root: modprobe: ERROR: could not insert 'kvm_intel': Operation not supported

 

I don't use a domain myself and have little experience of them but I suppose this is the line in the syslog that shows the attempt to join:


Oct 17 09:26:47 BasinNAS emhttpd: req (1): DOMAIN=BASINDESIGN.local&DOMAIN_SHORT=BASINDESIGN&DOMAIN_LOGIN=UNRAID&DOMAIN_PASSWD=&cmdJoinDomain=Join&csrf_token=****************

So it seems to be trying to join but is it being rejected by the domain controller? You maybe need to check the domain controller's logs.

 

John, Thank you for taking time to look into this and reply. 

 

Sadly this motherboard does not support virtualization, Our VM's are all hosted by a more modern server running dual 16 core xeon's and much more ram than this old dell machine that's running Unraid. 

 

As far as the domain join failure, that particular line is when I realized the drives were down and manually re-joined it to the domain in the GUI. The server rebooted a little before 4 this morning, the failed domain join would have been around that time as well (at least I'm assuming..)

Link to comment
7 minutes ago, ThomasE said:

Sadly this motherboard does not support virtualization, Our VM's are all hosted by a more modern server running dual 16 core xeon's and much more ram than this old dell machine that's running Unraid.

In that case you'll want to turn off the VM service in Settings -> VMs to stop it trying to start up.

Link to comment
4 minutes ago, John_M said:

In that case you'll want to turn off the VM service in Settings -> VMs to stop it trying to start up.

I've tried in the past, when I go to settings I get "VM Manager" and within VM Manager I get a notice saying "Your hardware does not have Intel VT-x or AMD-V capability. This is required to create VMs in KVM. Click here to see the Unraid Wiki for more information" and all that's available is a done button. I do not see any way to disable this.

Link to comment

 

4 hours ago, johnnie.black said:

It's an old issue, you can disable it by editing /config/domain.cfg on the flash drive and changing service="enable" to "disable", though it won't cause any issues leaving it enable.

Thanks for this info, i'll definitely dive into that this weekend when I did the memtest.

3 hours ago, johnnie.black said:

It defaults to on on a new install and servers without iommu can't disabled, except by editing the cfg like mentioned.

 

3 hours ago, John_M said:

Here's what I see. My hardware does have AMD-V capability and I can enable or disable VMs at will. Did you perhaps migrate your server from different hardware that supported virtualisation, allowing you to turn it on, but now not turn it off?

 

298797273_ScreenShot2019-10-17at17_50_20.thumb.png.8f9da342e2896b61109907885ff054e2.png

Negative, this is my first Unraid build. This is all I get.

image.thumb.png.2a7c8ad37fa1115324d2ff7b2ec681f7.png

Link to comment
  • 3 weeks later...

A little update.. We are still facing the issue of reboots overnight, they are becoming almost nightly now. Our efforts to counter this have been: We have powered the unraid server with the same UPS that our other servers are being powered from and are not experiencing power outages. Still getting reboots, so we ran a memtest all day, zero errors.

 

Right now I have it booted up into safe mode to see if that may prevent the reboots. 

One very odd thing that I'm noticing, it always happens around 1AM.. Which is the exact same time that my mover is scheduled.

 

**edit** I changed the mover schedule to 4am to see if the crashes will follow the mover. It survived two nights in a row so far without a reboot while in safemode.. I'm starting to wonder if it may be one of the plugins I have installed.

Edited by ThomasE
Link to comment
  • 4 weeks later...

So myself and @arcane had the same issues. We have been conversing back and forth for a couple weeks now and I am confident that the problem (for me) was the FixCommonProblems plugin. I rebooted the unraid into normal mode and wiped FCP off of it immediately. I am currently sitting at almost 14 days of uninterrupted use of my unraid server. 

 

@arcane eliminated FCP and delugevpn, and he has had the same results. At this point, I am going to consider the removal of FCP as a successful fix.

 

If anyone else has issues, please look at your plugins. My issues started very very soon after installing FCP, so think back to any plugins you may have installed relatively close to when you started getting problems.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.