November 3, 201312 yr Hi All - The tower just becomes unresponsive (crashes) even with a monitor connected it does not respond to any commands, cannot type anything, samba and nfs is gone, emhttp is gone as well. Basically the fans are spinning on the system and that is it, I know the hardware is working but the OS is dead. Similar to a BSOD on windows I would say. Here is what I have tried so far: 1) Reverted to stock - so basically reformatted the USB, copied the 5.0 stable files over, copied the config directory from my back up without the plugins, packages or the modified go file. Eveything is stock - no plugins at all. Not even the SF based new enhanced GUI. Not even uu. 2) Ran a memtest on the system for over 7-8 hours, 6 tests - all passed no errors found. 3) Tried both with the updated GUI and the stock GUI - but unfortunately I have the freezing up occuring even with the stock GUI. I am totally lost at this point and have no idea what is causing this, I used to run rc13 without any issues but even that seems to be not working fine, it will run longer than the 5.0 stable but eventually it will freeze up as well ( so totally at a loss). I have moved all my plugins from unraid to my windows based system and that would be where they are staying, but all my media is housed in this server and I have no way of accessing it. Sometimes, not always, even when it is running the transfer speeds become excruciatingly slow and sometimes XBMC/Plex would buffer every 2 seconds (literally). Any one know what is going on here?
November 3, 201312 yr See SYSTEM LOG in my sig for advice on collecting the syslog in real time. Version 5 has a Log button in the upper right of the management page. Click it and leave the window open until the system crashes. Once the Log window is open make a copy of the complete syslog (Utils->System Log). Post both the complete log and the contents of the log window after a crash.
November 4, 201312 yr Author Here are the logs - thanks in advance for taking a look! Unraid_Log_-_real_time.txt Unraid_Sys_log.txt
November 4, 201312 yr Author The system has crashed as before - fans are running, shares are not accessible!
November 4, 201312 yr I don't see any "smoking gun" in your syslogs (that's not to say there's not one there). But it looks like your mover schedule has been modified, probably by the enhanced webGUI. Have you had this problem (unresponsive) when running in "safe-mode?" Is it the kind of problem that's very predictable or just randomly occurs? I don't run NFS, but you might try just using SMB for a while to try and isolate the issue. You said you reverted to a stock unRAID build, is the hardware about as stock as you can get it? I can't help but think maybe there's a issue with the wireless keyboard driver. Hopefully you have disabled all unneeded on-board devices (sound) on the mobo. I wish I could help you more.
November 5, 201312 yr Author After a crash, can you access the server via telnet or an attached console? No for both. Telnet never seems to work even when the server is running. Should I install the ssh package for it to work? Sent from my SCH-I535 using Tapatalk
November 6, 201312 yr Author Telnet should work out of the box. I suspect that your not doing it right. Well putty configuration has not changed before and after the reinstall, and it is after the reinstall that putty was not working. My saved unRaid profile is what I am cliking on to try and get connected, but I either get timed out or the host cannot be found error. Host name: tower; Port: 22; Type: ssh. (screenshot attached). So not sure what I am doign wrong!!! The other saved putty profile was for crashplan, but since the new install is stock I have not installed crashplan yet. Anyways does the telenet issue help explaining the crashes or were you just curious? Because even with a monitor attached I can see that the ystem becomes unresponsive. Since the last update I even ran a flash drive check and found no errors at all. unRaid still crashes, any indication of what is causing this from the log files.
November 6, 201312 yr Author I don't see any "smoking gun" in your syslogs (that's not to say there's not one there). But it looks like your mover schedule has been modified, probably by the enhanced webGUI. Have you had this problem (unresponsive) when running in "safe-mode?" Is it the kind of problem that's very predictable or just randomly occurs? I don't run NFS, but you might try just using SMB for a while to try and isolate the issue. You said you reverted to a stock unRAID build, is the hardware about as stock as you can get it? I can't help but think maybe there's a issue with the wireless keyboard driver. Hopefully you have disabled all unneeded on-board devices (sound) on the mobo. I wish I could help you more. Thank you for taking a look. I have not tried running in a safe mode, since it was completely stock, but will try and report once I get back home. I am travellling for work this week. I do use nfs a lot, all my xbmc paths are using the nfs share and so does a few other programs. But if that is worth a try then I can stop nfs and see if it works. (At this point I am up for trying anything to get the tower running without crashin after a few hours ). It just randomly occurs, but sometimes emhttp crashes and then recovers on its own, and atother times everything stops. The hardware is as stock as it can get the only addition was an Intel NIC and a SATA port expansion card. Keyboard, right now since I plugged in a monitor I am using one, otherwise no, and the crashes are happening with and without....!!!Checked the flash with a flash drive checking utiltiy and found no errors, memtest ran okay with 6 passes, so no problem with the hardware that I could identifiy.
November 6, 201312 yr My Putty profile for unRAID is for Telnet on port 23. I don't think SSH is stock out-of-the-box.
November 6, 201312 yr Author Okay maybe that is what is not woking. So if I change it to port 23 and telnet it should work right? But does that give us any clue as to why the system is crashing. Or are these 2 seperate issues?
November 6, 201312 yr But does that give us any clue as to why the system is crashing. Or are these 2 seperate issues? That's a good question! I think when your system goes totally unresponsive AND you can't telnet to the box that says more than emhttp is just dying. The fact that the locally connected keyboard stops working would, to me, indicate a big lockup. I don't recall seeing the amount of memory you are using, but I think the syslog indicates 4GB. Is that right?
November 6, 201312 yr in looking over your attached logs there seems to be some issue with the mover..... Nov 4 03:40:01 Tower logger: mover started and things start moving....... Nov 4 04:34:41 Tower logger: Cannot stat file /proc/2114/fd/676: No such file or directory Nov 4 04:34:41 Tower logger: Cannot stat file /proc/2114/fd/683: No such file or directory Nov 4 04:34:41 Tower logger: Cannot stat file /proc/2114/fd/684: No such file or directory there is a group of these and then back to normal moving, but then almost 3 HOURS later Nov 4 05:28:44 Tower logger: >f+++++++++ TV Shows/Shark......... that's the end of the log and maybe the system dies soon after? From googling STAT it seems to be a part of the copy process. If I understand the mover correctly the mover copies or rsyncs files on the cache drive to the array. The STAT error comes when the copy process fails to check the status of the file it just copied (I think). Granted we're copying .MKV's, but three hours?? Something doesn't seem right. Maybe it's time to check the disks or controller?
November 7, 201312 yr Author But does that give us any clue as to why the system is crashing. Or are these 2 seperate issues? That's a good question! I think when your system goes totally unresponsive AND you can't telnet to the box that says more than emhttp is just dying. The fact that the locally connected keyboard stops working would, to me, indicate a big lockup. I don't recall seeing the amount of memory you are using, but I think the syslog indicates 4GB. Is that right? Yes it is 4GB and the memtest passed with 6 tests - all pases.
November 18, 201312 yr Author Run reiserfsck check. See Check Disk Filesystems in my sig. Okay just got a chance to finish running it on all the disks (md1 - md5 & sdg1(cache)) - all came back with no corruptions found. Using Telnet instead of SSH has got putty working so I can login remotely to the system. Also I disabled the mover and looks like the crashes have also stopped for the time being. I would like to use a cache disk and enable mover at some point, hopefully we can figure out what is casuing this and fix the issue soon. Thank you for everyone's help so far.
November 23, 201312 yr Author See here: http://lime-technology.com/forum/index.php?topic=28484.0 Thank You! Sorry for the late reply, but just got back home yesterday. Read and tried the steps outlined by Tom but getfattr –d /mnt/disk1/Movies did not not give any messages or errors for any top level directories on disks 1 -5 and cache. So did not have to run the script. The crashes are still happening especially when I try performing any kind of reading or writing activity - the whole tower just freezes up.
November 23, 201312 yr Does it matter which disk is accessed? Do all disk accesses cause a crash? What model PSU? How many HDD?
November 25, 201312 yr OK, I believe that I'm running into the exact same problem. I'm running 5.0 (stable release) on a new build (about 1 month old) and have never had it running stable. I have another server that has been running for years without any issues (5.0 as well). I have to turn off the tower to reboot and get everything started again. I'm running 5 drives (1 parity, 1 cache and 3 data). I thought originally I was having a network problem as I cold see the tower for a period of time (usually about 30-60 minutes) I have the IP fixed in both the tower and my router. I now know that the entire system crashes (I've plugged a monitor and keyboard into the tower and I cannot type anything). I've recently disabled the cache disc in the Settings/Share Settings - I still crashed. I recently noticed the last time the system crashed, the system was in the midst of a parity check and it was stuck at 16.5% and the estimated speed was 1MB/s with the estimated time being 5 days. Also, I'm pretty sure the parity check isn't the cause as many times I've stopped the check as I've rebooted and I've still crashed. Here is my syslog from the log link on the top right of the main page\ /usr/bin/tail -f /var/log/syslog Nov 24 20:53:36 Tower emhttp: get_config_idx: fopen /boot/config/shares/Movies.cfg: No such file or directory - assigning defaults Nov 24 20:53:36 Tower emhttp: get_config_idx: fopen /boot/config/shares/Music.cfg: No such file or directory - assigning defaults Nov 24 20:53:36 Tower emhttp: get_config_idx: fopen /boot/config/shares/Photos.cfg: No such file or directory - assigning defaults Nov 24 20:53:36 Tower emhttp: get_config_idx: fopen /boot/config/shares/TV.cfg: No such file or directory - assigning defaults Nov 24 20:53:36 Tower emhttp: Restart SMB... Nov 24 20:53:36 Tower emhttp: shcmd (39): killall -HUP smbd Nov 24 20:53:37 Tower emhttp: shcmd (40): ps axc | grep -q rpc.mountd Nov 24 20:53:37 Tower emhttp: _shcmd: shcmd (40): exit status: 1 Nov 24 20:53:37 Tower emhttp: shcmd (41): /usr/local/sbin/emhttp_event svcs_restarted Nov 24 20:53:37 Tower emhttp_event: svcs_restarted It just crashed again and I have this syslog to report Nov 24 21:57:39 Tower kernel: [] native_flush_tlb_others+0x26/0x28 Nov 24 21:57:39 Tower kernel: [] flush_tlb_mm_range+0x14d/0x154 Nov 24 21:57:39 Tower kernel: [] tlb_flush_mmu+0x40/0x6e Nov 24 21:57:39 Tower kernel: [] tlb_finish_mmu+0x12/0x31 Nov 24 21:57:39 Tower kernel: [] unmap_region+0x90/0x98 Nov 24 21:57:39 Tower kernel: [] ? vma_rb_erase+0x1ab/0x1b3 Nov 24 21:57:39 Tower kernel: [] do_munmap+0x1d8/0x246 Nov 24 21:57:39 Tower kernel: [] vm_munmap+0x32/0x43 Nov 24 21:57:39 Tower kernel: [] sys_munmap+0xe/0x10 Nov 24 21:57:39 Tower kernel: [] syscall_call+0x7/0xb
November 25, 201312 yr OK, I believe that I'm running into the exact same problem. I'm running 5.0 (stable release) on a new build (about 1 month old) and have never had it running stable. I have another server that has been running for years without any issues (5.0 as well). I have to turn off the tower to reboot and get everything started again. I'm running 5 drives (1 parity, 1 cache and 3 data). I thought originally I was having a network problem as I cold see the tower for a period of time (usually about 30-60 minutes) I have the IP fixed in both the tower and my router. I now know that the entire system crashes (I've plugged a monitor and keyboard into the tower and I cannot type anything). I've recently disabled the cache disc in the Settings/Share Settings - I still crashed. I recently noticed the last time the system crashed, the system was in the midst of a parity check and it was stuck at 16.5% and the estimated speed was 1MB/s with the estimated time being 5 days. Also, I'm pretty sure the parity check isn't the cause as many times I've stopped the check as I've rebooted and I've still crashed. Here is my syslog from the log link on the top right of the main page\ /usr/bin/tail -f /var/log/syslog Nov 24 20:53:36 Tower emhttp: get_config_idx: fopen /boot/config/shares/Movies.cfg: No such file or directory - assigning defaults Nov 24 20:53:36 Tower emhttp: get_config_idx: fopen /boot/config/shares/Music.cfg: No such file or directory - assigning defaults Nov 24 20:53:36 Tower emhttp: get_config_idx: fopen /boot/config/shares/Photos.cfg: No such file or directory - assigning defaults Nov 24 20:53:36 Tower emhttp: get_config_idx: fopen /boot/config/shares/TV.cfg: No such file or directory - assigning defaults Nov 24 20:53:36 Tower emhttp: Restart SMB... Nov 24 20:53:36 Tower emhttp: shcmd (39): killall -HUP smbd Nov 24 20:53:37 Tower emhttp: shcmd (40): ps axc | grep -q rpc.mountd Nov 24 20:53:37 Tower emhttp: _shcmd: shcmd (40): exit status: 1 Nov 24 20:53:37 Tower emhttp: shcmd (41): /usr/local/sbin/emhttp_event svcs_restarted Nov 24 20:53:37 Tower emhttp_event: svcs_restarted It just crashed again and I have this syslog to report Nov 24 21:57:39 Tower kernel: [] native_flush_tlb_others+0x26/0x28 Nov 24 21:57:39 Tower kernel: [] flush_tlb_mm_range+0x14d/0x154 Nov 24 21:57:39 Tower kernel: [] tlb_flush_mmu+0x40/0x6e Nov 24 21:57:39 Tower kernel: [] tlb_finish_mmu+0x12/0x31 Nov 24 21:57:39 Tower kernel: [] unmap_region+0x90/0x98 Nov 24 21:57:39 Tower kernel: [] ? vma_rb_erase+0x1ab/0x1b3 Nov 24 21:57:39 Tower kernel: [] do_munmap+0x1d8/0x246 Nov 24 21:57:39 Tower kernel: [] vm_munmap+0x32/0x43 Nov 24 21:57:39 Tower kernel: [] sys_munmap+0xe/0x10 Nov 24 21:57:39 Tower kernel: [] syscall_call+0x7/0xb Start a new thread. Please do not hijack this one. You can post a link to this thread.
November 25, 201312 yr Author Does it matter which disk is accessed? Do all disk accesses cause a crash? No it does not matter which disk is accessed. Sometimes it just crashes like right now for no apparent reason. I have not accessed tower in over 3-4 hours and came back now to see that it is unresponsive. Not necessarily. Not all activity causes a crash, I was watching videos and movies non stop for at least 3 hours yesterday and then sometime today morning through XBMC. All files are stored on tower - no problems occured through out this time. emhttp crashed a couple of time but it came right back. Just restarted the tower right now and it took a really long time to start the array and saw this in the log Nov 25 16:08:28 Tower logger: exportfs: Warning: /mnt/user/TV Shows does not support NFS export. Nov 25 16:08:28 Tower logger: exportfs: Warning: /mnt/user/Other Media does not support NFS export. What model PSU? How many HDD? PSU: Corsair AX 430W ATX - http://www.newegg.com/Product/Product.aspx?Item=N82E16817139026&Tpk=cx%20430 HDD: 1 parity (3TB) & 5 Data (3TB) - 1 Cache (2TB). Parity and 4 Data disks are these - Seagate Barracuda 7200 RPM 3TB http://www.newegg.com/Product/Product.aspx?Item=N82E16822148844&Tpk=ST3000DM001 1 Data is Hitachi Deskstar 3TB http://www.amazon.com/Hitachi-Deskstar-7K3000-HDS723030ALA640-SATA-600/dp/B004D8KO7Y and cache is the 2TB version of the Seagate Barracuda - http://www.newegg.com/Product/Product.aspx?Item=N82E16822148834
Archived
This topic is now archived and is closed to further replies.