January 4, 200917 yr OK. The Hero members here have taught me to NOT panic but something is wrong and I'm starting to worry. I'm running v4.5. I was copying a directory of files from one drive to another via MC when unraid disappeared from my network. Couldn't even ping. So, had to restart via the power button. Now, it starts to boot. Gets as far as installing custom packages. On the console screen I can see that it gets as far as running the *.auto_goscript then it freezes. I tried again and still this is as far as it gets. All 5 drives & parity are going nuts like a parity sync is in progress. Don't know what to do. Don't see any options other than letting it continue until it (hopefully) finishes whatever it's doing. Could use some advice right about now.
January 4, 200917 yr Can you telnet to the server? If yes, that is a good first sign. Since you forced the power down, it is doing a full parity check. I'd let it finish. Then, not knowing what you put in your "go" script, it is best to put it back the way it was originally. I'd remove any "auto_goscript" files for now. One of those might be causing your server to not finish booting. Then, capture and post a syslog... You are running a beta version of unRAID. Are you using NFS? I have no experience with its quirks... I'm on 4.4. Joe L.
January 4, 200917 yr Author Can't telnet. Can't ping. The server is invisible on my network. Have a console, but it is in the middle of finishing go script stuff and don't have a prompt. Comforted with the hope that all the drive activity is likely a harmless parity sync. Kind of freaking out and worried. Not using NFS. The most exotic thing in my go script is loading unMenu and installing update version of rsync.
January 4, 200917 yr Can't telnet. Can't ping. The server is invisible on my network. Have a console, but it is in the middle of finishing go script stuff and don't have a prompt. Comforted with the hope that all the drive activity is likely a harmless parity sync. Kind of freaking out and worried. Not using NFS. The most exotic thing in my go script is loading unMenu and installing update version of rsync. Alt-F2 through Alt-F6 will give you alternate "consoles" to log in via. (Alt-F1 will switch back to the first) Parity sync is normal. Don't fret it. If you can log in on one of the consoles, type ps -ef | grep emhttp See if the emhttp process is running. If not, you can try to start it yourself by typing: /usr/local/sbin/emhttp & Joe L.
January 4, 200917 yr Author I can CRLF (Enter) and scroll the lines up, but the Alt-Fx trick is not giving me an alternate console (doing nothing as if I didn't press any keys at all). Drives are still going like crazy. Still assuming a parity sync is in progress because that give me comfort. Thinking, maybe, CTRL-C or something like that to stop the go script. But not sure. Scared to do much of anything without guideance.
January 4, 200917 yr Rename your go script temporarily and reboot with no go script. Then run the commands in the go script one at a time, and give them time to complete. It could be a command farther up in the go script that is only causing the problem several seconds or even minutes after it was run.
January 4, 200917 yr Author OK. Woke up this morning. looks like the parity sync is done. At least all the drive activity has stopped. Unable to get into the console now. I'm not sure if it's in software or hardware, but my unraid server shuts off the monitory after a period of time and I hit the space key to wake it up. That isn't working at the moment. I'm thinking I'm going to have to go the route that bubbaQ suggests. I have another key that I can play with, but I'm not 100% sure what from the first key I'd need to transfer over so I don't screw things up.
January 4, 200917 yr OK. Woke up this morning. looks like the parity sync is done. At least all the drive activity has stopped. Unable to get into the console now. I'm not sure if it's in software or hardware, but my unraid server shuts off the monitory after a period of time and I hit the space key to wake it up. That isn't working at the moment. I'm thinking I'm going to have to go the route that bubbaQ suggests. I have another key that I can play with, but I'm not 100% sure what from the first key I'd need to transfer over so I don't screw things up. Move the flash drive to your PC, copy the existing config/go file to config/go_copy Then, edit the "go" script to leave ONLY the original top 3 lines #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & And actually, only the third line is really needed. /usr/local/sbin/emhttp & Then, eject the flash drive from the PC, put it back in the unRAID server, and boot it up. It will start a parity check once more, but with any luck you should be able to get to the management interface... You might want to do a memory check at some point if you suspect hardware. Joe L.
January 4, 200917 yr Author - renamed the go script and rebooted. I got a prompt on the console and was able to telnet. Strange, the telnet session required only a user name and no password. - executed /usr/local/sbin/emhttp & and it appears that parity sync started immediatly. - ran console version of unmenu and the status of all my drives appears normal. Interesting - noticed at the top of the screen while running the console version of unmenu there is a message that the 'local time zone must be set - blah blah'. I don't think that I've ever noticed that before. Confirmed that parity sync is in fact in progress. - Able to browse all disks and shares. At first I was not able to browse disk1 at the console, but after browsing the disk remotely (via 'My Network Places') I was then able to browse disk1 from the console. Not sure if I should let parity sync finish before trying my go script again. This is strange since I haven't made any changes in weeks. I was just copying files from /mnt/disk1/ to /mnt/disk4/ when it went unresponsive. Then the reboot wouldn't complete. The server is not letting me attach a file right now so I'm not able to share my syslog.
January 4, 200917 yr - renamed the go script and rebooted. I got a prompt on the console and was able to telnet. Strange, the telnet session required only a user name and no password. - executed /usr/local/sbin/emhttp & and it appears that parity sync started immediatly. - ran console version of unmenu and the status of all my drives appears normal. Interesting - noticed at the top of the screen while running the console version of unmenu there is a message that the 'local time zone must be set - blah blah'. I don't think that I've ever noticed that before. Confirmed that parity sync is in fact in progress. - Able to browse all disks and shares. At first I was not able to browse disk1 at the console, but after browsing the disk remotely (via 'My Network Places') I was then able to browse disk1 from the console. Not sure if I should let parity sync finish before trying my go script again. This is strange since I haven't made any changes in weeks. I was just copying files from /mnt/disk1/ to /mnt/disk4/ when it went unresponsive. Then the reboot wouldn't complete. The server is not letting me attach a file right now so I'm not able to share my syslog. It sounds like you are back in control of your server. It could be you ran out of RAM during some operation in MC when you were moving files and unRAID killed idle processes to free up some more RAM. You might never know for sure... but now that you can connect via telnet, open a window and type: tail -f /var/log/syslog Then, just let it run... If something weird occurs, you might have some clue what happened. Joe L.
January 4, 200917 yr Author I can access and download the syslog from unmenu, but I just can't post it to this forum. It tells me the file is too big but it's only 56K. This is the message the forum server returns: Your attachment couldn't be saved. This might happen because it took too long to upload or the file is bigger than the server will allow. Please consult your server administrator for more information. Looking at the syslog myself, I couldn't say if anything is out of the ordinary. Running the tail command right now to monitor the log during the parity sync. At the moment, this is what I see: Jan 4 17:43:27 UNRAID kernel: ReiserFS: md1: replayed 219 transactions in 160 seconds Jan 4 17:43:27 UNRAID kernel: ReiserFS: md1: Using r5 hash to sort names Jan 4 17:43:27 UNRAID kernel: can't shrink filesystem on-line Jan 4 17:43:28 UNRAID emhttp: shcmd (14): cp /etc/exports- /etc/exports Jan 4 17:43:28 UNRAID emhttp: shcmd (15): mkdir -m 700 /mnt/user Jan 4 17:43:28 UNRAID emhttp: shcmd (16): /usr/local/sbin/shfs /mnt/user Jan 4 17:43:29 UNRAID emhttp: shcmd (17): killall -HUP smbd Jan 4 17:43:29 UNRAID emhttp: shcmd (18): /etc/rc.d/rc.nfsd restart Jan 4 17:52:22 UNRAID ntpd[1177]: time reset -0.184985 s Jan 4 17:53:20 UNRAID ntpd[1177]: synchronized to 149.20.54.159, stratum 2
January 4, 200917 yr I can access and download the syslog from unmenu, but I just can't post it to this forum. It tells me the file is too big but it's only 56K. This is the message the forum server returns: Your attachment couldn't be saved. This might happen because it took too long to upload or the file is bigger than the server will allow. Please consult your server administrator for more information. Looking at the syslog myself, I couldn't say if anything is out of the ordinary. Running the tail command right now to monitor the log during the parity sync. At the moment, this is what I see: Jan 4 17:43:27 UNRAID kernel: ReiserFS: md1: replayed 219 transactions in 160 seconds Jan 4 17:43:27 UNRAID kernel: ReiserFS: md1: Using r5 hash to sort names Jan 4 17:43:27 UNRAID kernel: can't shrink filesystem on-line Jan 4 17:43:28 UNRAID emhttp: shcmd (14): cp /etc/exports- /etc/exports Jan 4 17:43:28 UNRAID emhttp: shcmd (15): mkdir -m 700 /mnt/user Jan 4 17:43:28 UNRAID emhttp: shcmd (16): /usr/local/sbin/shfs /mnt/user Jan 4 17:43:29 UNRAID emhttp: shcmd (17): killall -HUP smbd Jan 4 17:43:29 UNRAID emhttp: shcmd (18): /etc/rc.d/rc.nfsd restart Jan 4 17:52:22 UNRAID ntpd[1177]: time reset -0.184985 s Jan 4 17:53:20 UNRAID ntpd[1177]: synchronized to 149.20.54.159, stratum 2 Everything there looks very normal. It appears as if Tom needs to administer something on his web-site to add more space for "uploads"
January 4, 200917 yr Author I disabled about everything in my go script and in the unmenu package manager set all packages to not load at startup. Upon reboot it got stuck again. On the console screen I can see that it gets as far as running the *.auto_goscript then it freezes. I'm running a memory check right now. Currently at 66% and all is good. I noticed that on the web version of unmenu and the standard unraid config page that there is a message that the 'local time zone must be set '. I've never seen this before. Could this be a clue as to what is going on?
January 4, 200917 yr I disabled about everything in my go script and in the unmenu package manager set all packages to not load at startup. Upon reboot it got stuck again. On the console screen I can see that it gets as far as running the *.auto_goscript then it freezes. I'm running a memory check right now. Currently at 66% and all is good. I noticed that on the web version of unmenu and the standard unraid config page that there is a message that the 'local time zone must be set '. I've never seen this before. Could this be a clue as to what is going on? I'm sorry, but there is no way you disabled everything if it is running "*.auto_goscript" That is something you added. The 4.4 release of unRAID broke the "custom" time-zone handling. It is not likely to be the cause of your error. It is just an annoyance. I've sent Tom a PM about the time-zone issue when he first made the 4.4 release available. He has not responded. The "goscript" package manager is *very* beta. I did not write it, and as initially deployed has an error in its script. (it has "sleep(30)" where is should have "sleep 30") I think you should remove all *.auto_goscript" code from your server. Odds are it is what is causing the issue. Joe L.
January 4, 200917 yr Author Hmm. Ok, Joe. That's the direction I will go. I'm puzzled that it ever booted before, but not now. The memory check just finished with no errors. I'll keep you posted. Thanks for listening. It's comforting to know that I've got this community
January 4, 200917 yr Author Suspecting my issue has something to do with the Go Script Manager, I disabled the 'Enable Script on Re-Boot' option via unMenu (in the Go Script Manager plugin page). Now, I can boot the server and everything works as advertised. I don't fully understand because the only thing I'm using the Go Script Manager for is to run 'uu' upon boot. The contents of my unMENUstart-unmenu-goscript.conf are as follows: PACKAGE_NAME Start unMENU PACKAGE_DESCR Script to start unmenu at boot up. PACKAGE_INSTALLATION sleep 10 PACKAGE_INSTALLATION /boot/unmenu/uu Pretty basic stuff. I don't see why it worked before and now it causes an issue. I added this command to my go script manually and rebooted and everything appears fine (for now).
January 5, 200917 yr I just checked the .conf files I posted in this thread: http://lime-technology.com/forum/index.php?topic=2876.msg24281#msg24281 It seems that the content of the files didn't exactly corrolate to the actual contents of the files... Please check that you have the right .conf and that the command is pointing to the correct directory. I have updated the files/definitions in the conf message linked above. The actual code that runs the .auto_goscript files is the same as that of the package manager .auto_install files. The only change that was made was to replace every instance of .auto_install with .auto_goscript. Both types of files are just simple script files that are compiled based off of the information available in the package .conf files (built in the same method). I (unknowingly) have been using a version of the conf without the sleep command for the last week or so and it has worked every time (i've been testing alot lately for a new version I'm working on). If the .auto_goscript file is definetly causing the issue, please double/triple check that the .conf contains the correct information by selecting the "view executed code" button. Cheers, Matt
January 5, 200917 yr I just checked the .conf files I posted in this thread: http://lime-technology.com/forum/index.php?topic=2876.msg24281#msg24281 It seems that the content of the files didn't exactly corrolate to the actual contents of the files... Please check that you have the right .conf and that the command is pointing to the correct directory. I have updated the files/definitions in the conf message linked above. The actual code that runs the .auto_goscript files is the same as that of the package manager .auto_install files. The only change that was made was to replace every instance of .auto_install with .auto_goscript. Both types of files are just simple script files that are compiled based off of the information available in the package .conf files (built in the same method). I (unknowingly) have been using a version of the conf without the sleep command for the last week or so and it has worked every time (i've been testing alot lately for a new version I'm working on). If the .auto_goscript file is definetly causing the issue, please double/triple check that the .conf contains the correct information by selecting the "view executed code" button. Cheers, Matt One possible issue might be if a script executed before the emhttp process has had a chance to set up the "md" devices expected. Or if Any script was waiting for a response to continue, or ended up in a loop and never exited? Lots of potential for errors that the package manager never had to contend with. Have you tried your scripts when rebooting? What happens if somebody accidentally puts a CR/LF instead of LF in a goscript .conf file? It is very easy for that type of error to occur. What happens if a process emits a "Press any key to continue" and waits for input? I'm sure you'll figure it out though... just beware, any script run when you reboot has the potential to cause the process to abort in a way Tom never anticipated. All of the scripts I run when I reboot are in the folder /boot/custom/etc/rc.d Perhaps that is where you might put your excuitable files. I personally do not use the "Go Script" plug-in, as I have everything already in place to run all my start-up tasks in the rc.d folder. The very first script run there is a delay of 30 seconds to allow the emhttp process to come on-line. Joe L.
January 5, 200917 yr Author During the last parity sync, the drive activity stopped way before I'd expect parity sync to be complete and the server became invisible on the network. Rebooting now and letting parity sync start again.
January 5, 200917 yr One possible issue might be if a script executed before the emhttp process has had a chance to set up the "md" devices expected. Or if Any script was waiting for a response to continue, or ended up in a loop and never exited? Lots of potential for errors that the package manager never had to contend with. As I said before, this is not something I have ever run into in my testing of the uu go script package, I am going to include a feature to automatically delay 30 seconds before proceeding with any .auto_goscript files in the plugin just in case. Thanks for pointing out this potential problem. Have you tried your scripts when rebooting? What happens if somebody accidentally puts a CR/LF instead of LF in a goscript .conf file? It is very easy for that type of error to occur. What happens if a process emits a "Press any key to continue" and waits for input? I'm sure you'll figure it out though... just beware, any script run when you reboot has the potential to cause the process to abort in a way Tom never anticipated. the plugin replaces the cr/lf to just lf with the gsub command before building the .auto_goscript file. the only way for cr/lf returns to be present are to manually edit the auto_goscript file manually in a text editor. My plugin was designed with the thought of being the only thing that would edit those files. The only way I could "correct" a file edited that way was to ensure that the go script manager (containing the correction code) was run AFTER the modifications but BEFORE or during the next boot the next boot. I guess I could include code to go through all the .auto_goscript files and run them through a fromdos command with my delay command on boot-up... In what ways could I deal with a "press any key" or infinite loop situation? All of the scripts I run when I reboot are in the folder /boot/custom/etc/rc.d Perhaps that is where you might put your excuitable files. I personally do not use the "Go Script" plug-in, as I have everything already in place to run all my start-up tasks in the rc.d folder. The very first script run there is a delay of 30 seconds to allow the emhttp process to come on-line. Joe L. Thanks for the pointers, I'm going to do some more testing on the boot up to see if I can get the same kind of error as jarodtufts described. So far no luck. I found that every once in a while after a new boot, that the web interface(stock) does not SEEM to come online when you refresh an older browser window to view the web interface(stock). I needed to open a NEW browser window and attempt to connect to the interface again (which worked). Telnet will work while the web interface will APPEAR to be down. I don't know if anyone else has foun this problem? Cheers, Matt
January 5, 200917 yr Author Matt, I'm very puzzled about this. Even though disabling the Go Script Manager seemed to allow things to boot again, I'm not yet a believer in the cause & effect relationship. I say this first because I didn't have the Go Script Manager doing anything exotic beyond running uu upon startup. And, second, since disabling I had one more lockup during parity sync. However, now I'm safely through a parity sync and things are running again. I do need to test re-enabling the Go Script Manager. But for now I just want to enjoy a functioning unRaid server for a while. Actually before I test re-enabling the Go Script Manager I'm going to try to copy that group of files I was trying to copy when the server first went down. This time, though, I'm going to have a tail running on the syslog in a telnet session so maybe I'll have a better clue as to what is happening if it does go down again.
January 6, 200917 yr Author Well, I tried to do the file copy again this evening. This time I used cp at the command line of a telnet session instead of using mc. Looks like it got about a 3rd of the way through the copy (there are about 5000 or so MP3 files being copied). Then it locked up again. Lost drive & share access on the network. My telnet windows froze too. I had one telnet running a tail on the syslog so I could catch a clue if it locked up again. But, no info came across in the syslog at the time of lockup. I have to hit the power button now. No choice. Man I'm getting tired of having to re-run the parity sync. Next (after parity sync finishes) I'm going to try to copy the files using windows explorer. Yea, it'll take a long time, but if it doesn't lock my server up I'll be happy.
January 6, 200917 yr Author So, as it stands now my issue is repeatable. I'm trying to copy about 5000 MP3 files (sorted in subdirectories by artist and album) from /mnt/disk1 to mnt/disk4. I've tried using mc and I've tried in a telnet window with the cp command. Every time the process will get about a 3rd of the way through (I'm guessing) and then the system becomes unresponsive. It just completely stops and disappears from the network (no telnet). Next I want to test copying the files with windows explorer, but even if that works if mc or cp can't do the job than I still have a problem. I've run the memory test a few times with no issues. Every time I have to powerdown to bring the server back up I have to do a parity sync. In the last 5 days there have been 5 completed parity syncs with no errors. At this point I see I have two options. 1) drop back to v4.4 final from v4.5 beta that I'm running now. Or 2) I've got another motherboard that I'll be able to try in a few days. Which option sounds like the better one to try first?
January 7, 200917 yr Turn off user-shares and try the copy. It is as likely as anything you are running out of ram for it to create the two copies of user-shares for that many files. No free ram = server crash. Then, once all files are moved, re-enable them.
Archived
This topic is now archived and is closed to further replies.