ThomasE Posted October 17, 2019 Share Posted October 17, 2019 (edited) *edited, link removed and syslog directly attached to post* Hello all, a couple weeks ago our unraid server went down in the middle of the night, and booted back up unjoined from the domain it was previously joined to. I thought it may have been the UPS, as it's a pretty old unit so I replaced it. Fast forward to last night, the same thing happened again. It went down around 3 or 4 in the morning and rebooted unjoined from our domain. I was wondering if someone more knowledgable with unraid could take a look at my syslog and see if there's something in the log to indicate what's going on? Many thanks! Dell T110 II Xeon E3 1200 V2 16GB Ram 4 - 4TB spinners (2 parity) 2 - 512GB SSD (cache pool) Unraid 6.7.2 syslog.txt Edited October 17, 2019 by ThomasE syslog attachment Quote Link to comment
trurl Posted October 17, 2019 Share Posted October 17, 2019 I'm not going to that external site and give them my email and agree to their terms. Now that you have been approved you can attach the complete Diagnostics zip file (Tools - Diagnostics) directly to your next post. Have you enabled the Syslog Server feature? https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=781601 ( Quote Link to comment
ThomasE Posted October 17, 2019 Author Share Posted October 17, 2019 (edited) 9 minutes ago, trurl said: I'm not going to that external site and give them my email and agree to their terms. Now that you have been approved you can attach the complete Diagnostics zip file (Tools - Diagnostics) directly to your next post. Have you enabled the Syslog Server feature? https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=781601 ( I have not, I will get that setup today. *edit, it is now enabled* Edited October 17, 2019 by ThomasE Quote Link to comment
trurl Posted October 17, 2019 Share Posted October 17, 2019 Have you done a memtest? Quote Link to comment
ThomasE Posted October 17, 2019 Author Share Posted October 17, 2019 I have not, I'll have to come in and run it over night or over the weekend when employee's are not accessing the drives. Quote Link to comment
John_M Posted October 17, 2019 Share Posted October 17, 2019 I don't see anything out of the ordinary in your syslog (though it obviously was grabbed after the reboot so it won't show what led to the crash) but you probably need to check the virtualisation settings in your BIOS if you're planning to run VMs: Oct 17 09:28:13 BasinNAS root: Starting virtlockd... Oct 17 09:28:13 BasinNAS root: Starting virtlogd... Oct 17 09:28:13 BasinNAS root: Starting libvirtd... Oct 17 09:28:13 BasinNAS kernel: kvm: disabled by bios Oct 17 09:28:13 BasinNAS root: modprobe: ERROR: could not insert 'kvm_intel': Operation not supported I don't use a domain myself and have little experience of them but I suppose this is the line in the syslog that shows the attempt to join: Oct 17 09:26:47 BasinNAS emhttpd: req (1): DOMAIN=BASINDESIGN.local&DOMAIN_SHORT=BASINDESIGN&DOMAIN_LOGIN=UNRAID&DOMAIN_PASSWD=&cmdJoinDomain=Join&csrf_token=**************** So it seems to be trying to join but is it being rejected by the domain controller? You maybe need to check the domain controller's logs. Quote Link to comment
ThomasE Posted October 17, 2019 Author Share Posted October 17, 2019 5 minutes ago, John_M said: I don't see anything out of the ordinary in your syslog (though it obviously was grabbed after the reboot so it won't show what led to the crash) but you probably need to check the virtualisation settings in your BIOS if you're planning to run VMs: Oct 17 09:28:13 BasinNAS root: Starting virtlockd... Oct 17 09:28:13 BasinNAS root: Starting virtlogd... Oct 17 09:28:13 BasinNAS root: Starting libvirtd... Oct 17 09:28:13 BasinNAS kernel: kvm: disabled by bios Oct 17 09:28:13 BasinNAS root: modprobe: ERROR: could not insert 'kvm_intel': Operation not supported I don't use a domain myself and have little experience of them but I suppose this is the line in the syslog that shows the attempt to join: Oct 17 09:26:47 BasinNAS emhttpd: req (1): DOMAIN=BASINDESIGN.local&DOMAIN_SHORT=BASINDESIGN&DOMAIN_LOGIN=UNRAID&DOMAIN_PASSWD=&cmdJoinDomain=Join&csrf_token=**************** So it seems to be trying to join but is it being rejected by the domain controller? You maybe need to check the domain controller's logs. John, Thank you for taking time to look into this and reply. Sadly this motherboard does not support virtualization, Our VM's are all hosted by a more modern server running dual 16 core xeon's and much more ram than this old dell machine that's running Unraid. As far as the domain join failure, that particular line is when I realized the drives were down and manually re-joined it to the domain in the GUI. The server rebooted a little before 4 this morning, the failed domain join would have been around that time as well (at least I'm assuming..) Quote Link to comment
John_M Posted October 17, 2019 Share Posted October 17, 2019 7 minutes ago, ThomasE said: Sadly this motherboard does not support virtualization, Our VM's are all hosted by a more modern server running dual 16 core xeon's and much more ram than this old dell machine that's running Unraid. In that case you'll want to turn off the VM service in Settings -> VMs to stop it trying to start up. Quote Link to comment
ThomasE Posted October 17, 2019 Author Share Posted October 17, 2019 4 minutes ago, John_M said: In that case you'll want to turn off the VM service in Settings -> VMs to stop it trying to start up. I've tried in the past, when I go to settings I get "VM Manager" and within VM Manager I get a notice saying "Your hardware does not have Intel VT-x or AMD-V capability. This is required to create VMs in KVM. Click here to see the Unraid Wiki for more information" and all that's available is a done button. I do not see any way to disable this. Quote Link to comment
JorgeB Posted October 17, 2019 Share Posted October 17, 2019 8 minutes ago, ThomasE said: I do not see any way to disable this. It's an old issue, you can disable it by editing /config/domain.cfg on the flash drive and changing service="enable" to "disable", though it won't cause any issues leaving it enable. Quote Link to comment
John_M Posted October 17, 2019 Share Posted October 17, 2019 Here's what I see. My hardware does have AMD-V capability and I can enable or disable VMs at will. Did you perhaps migrate your server from different hardware that supported virtualisation, allowing you to turn it on, but now not turn it off? Quote Link to comment
JorgeB Posted October 17, 2019 Share Posted October 17, 2019 4 minutes ago, John_M said: Did you perhaps migrate your server from different hardware that supported virtualisation, allowing you to turn it on, but now not turn it off? It defaults to on on a new install and servers without iommu can't disabled, except by editing the cfg like mentioned. 1 Quote Link to comment
ThomasE Posted October 17, 2019 Author Share Posted October 17, 2019 4 hours ago, johnnie.black said: It's an old issue, you can disable it by editing /config/domain.cfg on the flash drive and changing service="enable" to "disable", though it won't cause any issues leaving it enable. Thanks for this info, i'll definitely dive into that this weekend when I did the memtest. 3 hours ago, johnnie.black said: It defaults to on on a new install and servers without iommu can't disabled, except by editing the cfg like mentioned. 3 hours ago, John_M said: Here's what I see. My hardware does have AMD-V capability and I can enable or disable VMs at will. Did you perhaps migrate your server from different hardware that supported virtualisation, allowing you to turn it on, but now not turn it off? Negative, this is my first Unraid build. This is all I get. Quote Link to comment
ThomasE Posted October 22, 2019 Author Share Posted October 22, 2019 Good morning gents The unraid server went down again around 1:30 this morning, I've attached the syslog.. however it doesn't show the failure, it's got some trimming log info from the morning before, and then it just skips to the reboot. syslog Quote Link to comment
JorgeB Posted October 22, 2019 Share Posted October 22, 2019 9 minutes ago, ThomasE said: and then it just skips to the reboot. Yep, nothing on the log, which makes it impossible to guess the problem, you could try running the server in safe mode without any docker or VMs for a few days, if it still crashes like that it's likely a hardware problem. Quote Link to comment
ThomasE Posted November 6, 2019 Author Share Posted November 6, 2019 (edited) A little update.. We are still facing the issue of reboots overnight, they are becoming almost nightly now. Our efforts to counter this have been: We have powered the unraid server with the same UPS that our other servers are being powered from and are not experiencing power outages. Still getting reboots, so we ran a memtest all day, zero errors. Right now I have it booted up into safe mode to see if that may prevent the reboots. One very odd thing that I'm noticing, it always happens around 1AM.. Which is the exact same time that my mover is scheduled. **edit** I changed the mover schedule to 4am to see if the crashes will follow the mover. It survived two nights in a row so far without a reboot while in safemode.. I'm starting to wonder if it may be one of the plugins I have installed. Edited November 8, 2019 by ThomasE Quote Link to comment
ThomasE Posted November 11, 2019 Author Share Posted November 11, 2019 Well, its been up for almost 5 days with zero issues.. I'm starting to think that safe mode is the reason for that. My questions now is this: What exactly does safe mode do? I know it disables add on's, but does it do anything else? Quote Link to comment
JorgeB Posted November 11, 2019 Share Posted November 11, 2019 Mostly disables the plugins, you can uninstall them all and install one by one. Quote Link to comment
ThomasE Posted December 6, 2019 Author Share Posted December 6, 2019 So myself and @arcane had the same issues. We have been conversing back and forth for a couple weeks now and I am confident that the problem (for me) was the FixCommonProblems plugin. I rebooted the unraid into normal mode and wiped FCP off of it immediately. I am currently sitting at almost 14 days of uninterrupted use of my unraid server. @arcane eliminated FCP and delugevpn, and he has had the same results. At this point, I am going to consider the removal of FCP as a successful fix. If anyone else has issues, please look at your plugins. My issues started very very soon after installing FCP, so think back to any plugins you may have installed relatively close to when you started getting problems. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.