October 30, 201411 yr Author is there any problem with a folder called "scripts" on the root of my flash drive? I've put the preclear, diskspeed and other user created scripts in that folder to keep them all together. I did this about the same time these troubles started, and renamed it OLDscripts in case it was a problem (overly cautious I know). I just want to be sure it's okay to change it back to scripts.
November 1, 201411 yr There was another thread somewhere in the forum (That I can't find right now to save my life) where someone theorized that their crashes were related to a driver problem with the supermicro AOC-SASLP-MV8 card. He had found a well documented issue with another linux distro that were somewhat similar to what we were seeing in unraid. I have one of these cards, and I was crashing on a semi-daily basis. Other symptoms included the shutdown button not working in the gui, but the reboot button does, and the console locks up as well. On a hunch I changed the spindown setting to never, and I went one day shy of 4 weeks without a crash. This is in comparison to it happening at minimal every 2 days, and once twice in the same day with the spindown set to 2 hours. Of course the shutdown button still didn't shutdown the server properly, but at least I wasn't crashing anymore. Yesterday it locked up the same way it had been, but after a hard boot so far so good. I wonder if others having issues with unraid locking up also have the supermicro card. I know at one point it was a pretty popular option. And if you dig into the forum, there are a lot of posts with people describing similar things in different ways, post upgrading to an unraid 6 beta.
November 1, 201411 yr I have 2 of those cards but my lockups were X-Boost O/C feature from ASRock motherboard. My cards have the .21 fw both.
November 1, 201411 yr I have the same cards in 2 different unRaid servers, and one use to crash/other issues, and the other never had a single issue. The one machine that was having an issue seemed to be solved by booting under Xen for some reason. It was strange, prior to 10a I was getting machine hangs and crashes, after 10a I don't get crashes with non-Xen, but I would get parity errors every parity check, which were usually the same secotors repaired every time. Under Xen, parity checks are clean and no other issues. I don't use any Xen VM's, just booted up under the Xen kernel. Also, the server that was crashing had a different controller card and had the same issues. i replaced it with the supermicro to try to eliminate the issue, and didn't make any difference.
November 1, 201411 yr Author I don't have one of those cards, and I couldn't find any xboost in my BIOS. I removed the section from the syslinux.cfg file, and my server ran fine for about 2 days. I added back a modified version of that section (now I have pcie_acs_override=multifunction) and it's been running for about 8 hours now without freezing/dying. Too early to say it's 'fixed', and I haven't re-added my plugins, so maybe it's a combination of things. Hopefully beta11 will be released soon and will 'fix' whatever my issue is/was.
November 3, 201411 yr Well, the card may very well be unrelated to the issue. I was just putting it out there. But the fact remains that my unraid server was locking up once every day or so, sometimes twice a day sometimes every 2 days. And I changed the spindown setting to never and it went 27 days without locking up. That has to mean something.
November 3, 201411 yr Well, according to what you are saying I see 2 things that can be responsible, 1 - card (which somewhat has been ruled out) and 2 - drives (maybe you need to upgrade fw on your drives?) I have 2 of these super micro cards and all my drives spin down (and spin back up) perfect without any issues. I use 3 similar Western Digital drives on my array: WD20EARS, WD20EARX, WD20EZRX.
November 3, 201411 yr Well, according to what you are saying I see 2 things that can be responsible, 1 - card (which somewhat has been ruled out) and 2 - drives (maybe you need to upgrade fw on your drives?) I have 2 of these super micro cards and all my drives spin down (and spin back up) perfect without any issues. I use 3 similar Western Digital drives on my array: WD20EARS, WD20EARX, WD20EZRX. The only drives that have changed since upgrading to a 6.0 beta are the SSD's I added as a raid1 cache. All other drives are the same as what I had with Unraid 5, and it was very stable. The raid 1 cache capability was the main reason I jumped on the beta bandwagon. I usually wait until a stable release is out before upgrading. I already tried disabling them in the gui with no change. I may try disconnecting them completely, see if that makes a difference. To those who may be curious, I will report back my results.
November 3, 201411 yr I was getting some kernel crashes with v5b10a. I had recently added some WD 6TB Red drives. Some were attached to motherboard ports, and others to an AOC-SASLP-MV8 card I have set these 6TB drives to never spin down and the kernel errors seemed to have stopped. I wonder if the fact these WD 6TB Red drives take a relatively long time to spin up (one of the reasons I guess they are so low on power consumption) is an issue?
November 3, 201411 yr Author Interestingly enough, I was about to post that I've not had any issues for the last week or so, but then the system just crashed. I had just upped the memory and CPU's I sent to the windows 7 VM, and it started, and ran for a bit, but when I started playing a video on my laptop, it stuttered, then stopped, and at the same time the VM crashed, and I checked to find out the entire server crashed. So, I suspect it might be a lack of memory or CPU availability to the host that caused this last crash. I have 8GB and 4CPU's in the server (no hyper-threading) I had been, and am currently passing 4GB and 2 CPU's to the VM. I had bumped that up to 6GB and 3 CPU's which is what I had when it died. I've dialed it back, and have been watching a video on the VM for the last 15 minutes, so i suspect I'm okay again, but will report back if I have any more issues.
November 4, 201411 yr Interestingly enough, I was about to post that I've not had any issues for the last week or so, but then the system just crashed. I had just upped the memory and CPU's I sent to the windows 7 VM, and it started, and ran for a bit, but when I started playing a video on my laptop, it stuttered, then stopped, and at the same time the VM crashed, and I checked to find out the entire server crashed. So, I suspect it might be a lack of memory or CPU availability to the host that caused this last crash. I have 8GB and 4CPU's in the server (no hyper-threading) I had been, and am currently passing 4GB and 2 CPU's to the VM. I had bumped that up to 6GB and 3 CPU's which is what I had when it died. I've dialed it back, and have been watching a video on the VM for the last 15 minutes, so i suspect I'm okay again, but will report back if I have any more issues. Do you run cache_dirs? If so how many disks/files on the server. Under high pressure of memory with cache_dirs and many drives you can have OOM issues. Although the whole server should not lockup/crash.
November 4, 201411 yr Author Do you run cache_dirs? If so how many disks/files on the server. Under high pressure of memory with cache_dirs and many drives you can have OOM issues. Although the whole server should not lockup/crash. nope. I haven't run cache_dirs since going to v6
November 4, 201411 yr Author and, it just crashed again. How can I help diagnose and fix this issue? I really want to have my server work without issue for more than 2 days. I'm so exhausted with fighting with it
November 4, 201411 yr and, it just crashed again. How can I help diagnose and fix this issue? I really want to have my server work without issue for more than 2 days. I'm so exhausted with fighting with it can you post the /etc/rsyslog.conf or /etc/syslog.conf, maybe we can add a line in it to write to the flash or cache drive.
November 4, 201411 yr Author can you post the /etc/rsyslog.conf or /etc/syslog.conf, maybe we can add a line in it to write to the flash or cache drive. I just ran from putty... cp /etc/rsyslog.conf /boot/rsyslog.conf and my server rebooted, and now is stuck in a boot loop. the thumb had to be 'repaired' to boot again. I then booted and ran nano /etc/rsyslog.conf, but it's too long to copy the whole thing. Here are the parts that are not commented out... ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) # Set the default permissions for all log files. # $FileOwner root $FileGroup root $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 # tmm - everything goes to syslog. *.debug -/var/log/syslog # Emergency level messages go to all users: *.emerg :omusrmsg:* /etc/syslog.conf is blank thanks for your help
November 4, 201411 yr mkdir -p /boot/logs Add this line and put it on your flash (temporarily) or cache drive. *.debug -/boot/logs/syslog This could add some wear to your flash. if you have a sacrificial flash or want to use your cache, change the destination output. Point is, you want to duplicate the debug line to some static storage. In days gone by I would also duplicate this line to /dev/tty12. Then switch to virtual console 12 so all syslog messages would be on screen as well. *.debug /dev/tty12 Then switch to virtual console 12 so all syslog messages would be on screen as well. After this you need to hup the syslog daemon. do pgrep rsyslog A pid will be displayed do ps -fp (pid from before). should show the syslog daemon. kill -1 (pid from before). Or you can do something like kill -1 `cat /var/run/syslog.pid` To test do logger -ttest test message and look at the syslogs, you should see the message getting duplicated.
November 4, 201411 yr Author mkdir -p /boot/logs Add this line and put it on your flash (temporarily) or cache drive. *.debug -/boot/logs/syslog This could add some wear to your flash. if you have a sacrificial flash or want to use your cache, change the destination output. Point is, you want to duplicate the debug line to some static storage. In days gone by I would also duplicate this line to /dev/tty12. Then switch to virtual console 12 so all syslog messages would be on screen as well. *.debug /dev/tty12 Then switch to virtual console 12 so all syslog messages would be on screen as well. After this you need to hup the syslog daemon. do pgrep rsyslog A pid will be displayed do ps -fp (pid from before). should show the syslog daemon. kill -1 (pid from before). Or you can do something like kill -1 `cat /var/run/syslog.pid` To test do logger -ttest test message and look at the syslogs, you should see the message getting duplicated. It didn't seem to work. I added /mnt/cache/logs to put it on my cache drive I added these lines to /etc/rsyslog.conf # tmm - everything goes to syslog. *.debug -/var/log/syslog *.debug -/mnt/cache/logs/syslog *.debug /dev/tty12 Then I saved, then ran these commands... root@media:~# nano /etc/rsyslog.conf root@media:~# pgrep rsyslog 1272 root@media:~# ps -fp 1272 UID PID PPID C STIME TTY TIME CMD root 1272 1 0 10:29 ? 00:00:00 /usr/sbin/rsyslogd -i /var/run/rsyslogd.pid root@media:~# kill -1 1272 root@media:~# logger -ttest test message I didn't see any logs on screen, nor in the logs folder. Where did I go wrong? Should i just reboot the server?
November 4, 201411 yr Author can you post the /etc/rsyslog.conf or /etc/syslog.conf, maybe we can add a line in it to write to the flash or cache drive. I just ran from putty... cp /etc/rsyslog.conf /boot/rsyslog.conf and my server rebooted, and now is stuck in a boot loop. the thumb had to be 'repaired' to boot again. I then booted and ran nano /etc/rsyslog.conf, but it's too long to copy the whole thing. Here are the parts that are not commented out... ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) # Set the default permissions for all log files. # $FileOwner root $FileGroup root $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 # tmm - everything goes to syslog. *.debug -/var/log/syslog # Emergency level messages go to all users: *.emerg :omusrmsg:* /etc/syslog.conf is blank thanks for your help
November 4, 201411 yr Author apparently not a real "crash", as I still have the console showing stuff. But, I cannot type anything into the console at this point, so pretty much crashed. Hopefully when it reboots, the logging changes I made will work.
November 4, 201411 yr The logging changes are not permanant. also from *.debug to the -/mnt/cache/logs/syslog use tabs to separate, not spaces. I forgot to tell you to touch the file first. If the output file does not exist, syslog will not create it. so do mkdir -p /mnt/cache/logs touch /mnt/cache/logs/syslog Add the two entries suggested. then do the pgrep and kill -1 the pid. or do kill -1 `/cat /var/run/syslog.pid` Keep in mind you will not be able to stop the server normally with syslog being written to /mnt/cache/logs. i.e. /mnt/cache cannot be unmounted. You'll need to move the file elsewhere or rename it. Then do kill -1 `/cat /var/run/syslog.pid`
November 4, 201411 yr Being frank, I'm not running unRAID 6 yet. Otherwise i would make a quick script to do this. Sorry. I'm still in the process of setting one up.
November 4, 201411 yr Author Being frank, I'm not running unRAID 6 yet. Otherwise i would make a quick script to do this. Sorry. I'm still in the process of setting one up. No problem, i appreciate the help. I'm doing an online webinar right now, so I started a memtest on the server, which I'll leave for a few hours, until this class is done, then i'll reboot and make the changes you suggested above. thanks again.
November 4, 201411 yr Author The logging changes are not permanant. also from *.debug to the -/mnt/cache/logs/syslog use tabs to separate, not spaces. I forgot to tell you to touch the file first. If the output file does not exist, syslog will not create it. so do mkdir -p /mnt/cache/logs touch /mnt/cache/logs/syslog Add the two entries suggested. then do the pgrep and kill -1 the pid. or do kill -1 `/cat /var/run/syslog.pid` Keep in mind you will not be able to stop the server normally with syslog being written to /mnt/cache/logs. i.e. /mnt/cache cannot be unmounted. You'll need to move the file elsewhere or rename it. Then do kill -1 `/cat /var/run/syslog.pid` is it 'better' to do it one way vs. the other? I tried the second, one-step option, but failed... root@media:~# kill -1 `/cat /var/run/syslog.pid` -bash: /cat: No such file or directory kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] I assume I need to run it from a different directory. i tried changing to /boot, but that didn't change anything. I then tried running the other commands, but the syslog file in the /mnt/cache/logs directory is still empty... root@media:/boot# pgrep rsyslog 1252 root@media:/boot# ps -fp 1252 UID PID PPID C STIME TTY TIME CMD root 1252 1 0 15:57 ? 00:00:00 /usr/sbin/rsyslogd -i /var/run/rsyslogd.pid root@media:/boot# kill -1 1252 root@media:/boot# logger -ttest test message root@media:/boot# logger -test test message I'm sure I'm doing something stupid, but I'm not sure what
November 4, 201411 yr I think it was a type-o the "/" before the cat and it is rsyslogd.pid. Try: kill -1 `cat /var/run/rsyslogd.pid`
Archived
This topic is now archived and is closed to further replies.