June 24, 200818 yr I recently installed a DS-3151SSBK (5 drives in 3 bays). Everything is working fine after the install except that every morning when attempting to access a drive share, I can't access any nor disk shares. I also cannot access the unraid user interface via the ip address. I have to go over the server shut it down and start it back up, then I can access the unraid interface and the disk and user shares. I don't think I change any settings in the mother board bios, and I did not do any up grades at the time that I was installing the 5 in 3. I am not sure how to correct this. Thanks for any help. I am not sure if any specs are needed but my unraid is 4.3-beta6 and I am using xp-pro.
June 25, 200818 yr Please see my sig for the Troubleshooting page. We need a syslog. Are there errors on the server monitor when you have to shut it down? If possible, capture the syslog *after* it has failed.
June 25, 200818 yr I don't know whether it means anything or not, but what is your disk spin down delay set to?
June 25, 200818 yr Author I have the syslog, I used "cp /var/log/syslog /boot" at a telnet prompt this morning after the problem but before a reboot. I then powered down took the thumb drive (boot drive) over to my xp machine to open the syslog, and noticed the file syslog is 438mb. I am not sure if that is or is not normal. I need to know how many pages or to what point in the file do you need info from? My spin down delay is one hour
June 25, 200818 yr I have the syslog, I used "cp /var/log/syslog /boot" at a telnet prompt this morning after the problem but before a reboot. I then powered down took the thumb drive (boot drive) over to my xp machine to open the syslog, and noticed the file syslog is 438mb. I am not sure if that is or is not normal. I need to know how many pages or to what point in the file do you need info from? My spin down delay is one hour rename it to syslog.txt and then zip it up before attaching it to a post to this thread, odds are it will compress a lot as most lines in it are going to be duplicates if previous lines. If still too big to attach, the last few pages in it will probably give us the clues we need. From your logs' file size, I can tell exactly why your server crashed. You ran out of RAM. Basically, the log file used up all that was available. Once that happens, the Linux OS starts making more space available by killing off processes. Eventually, it kills the web-server and telnet server... once it runs out of processes to terminate, it crashes as it does not have the ability to allocate more memory. The underlying cause is what needs to be fixed. Something is writing a lot of lines of status/error messages to the syslog. Once you figure out that, it needs to be corrected. Joe L.
June 25, 200818 yr Author Attached is my syslog. Just at first glance, it looks like the mover is struggling to move lots of files or the mover has to create a file for every file moved and ends up eating up space which is causing the problem.
June 26, 200818 yr The syslog confirms what Joe said, you are out of memory, and it's shutting down processes. To save space, you cut out all of the mover lines, but could you give an example of what was there, doctored if necessary? You say 'thousands', but if I estimate 2000 files moved with one line logged per move, I still only get a syslog of about 200K, leaving over 437MB of something else. Even 10,000 files is less than one megabyte. There had to have been a lot of error messages in there. The syslog looks fine up to the cut at mover start.
June 26, 200818 yr If you were moving "thousands" of files, or if you were caught in a file-system loop, where it was moving the same file thousands of times, then the syslog could get filled. It might have taken weeks to fill, but once it did, you crashed. (looking closer, your first "log" entry and last are only an hour apart... it did not take weeks. ) According to this paragraph in the release notes that introduced the cache drive, the logging of entries to the syslog could be disabled by simply making a tiny edit to the mover script: When the mover moves files from the cache disk to the array, a pair of syslog entries are made for each file. There is currently no control for turning this off, though an advanced user can do so by editing the script "/usr/local/sbin/mover", removing the "-v" option on the mv command. This: (cd /mnt/cache ; \ find . \( -type d -regex '[.]/[^.].*' -exec mkdir -p /mnt/user0/{} \; \) , \ \( -type f -regex '[.]/.*/.*' -exec mv -v {} /mnt/user0/{} \; \) \ ) would then be changed to look like this (notice the "-v" has been removed): (cd /mnt/cache ; \ find . \( -type d -regex '[.]/[^.].*' -exec mkdir -p /mnt/user0/{} \; \) , \ \( -type f -regex '[.]/.*/.*' -exec mv {} /mnt/user0/{} \; \) \ ) To fix a tiny bug where a file could be moved before it was completely written, an even better fix would be this (notice an additional call to fuser has been added): (cd /mnt/cache ; \ find . \( -type d -regex '[.]/[^.].*' -exec mkdir -p /mnt/user0/{} \; \) , \ \( -type f -regex '[.]/.*/.*' ! -exec fuser -s {} \; -exec mv {} /mnt/user0/{} \; \) \ ) As long as you are in there editing the file, might as well fix the bug too. (This is supposed to be fixed in release 4.3.2, not yet released) It is described in this thread: http://lime-technology.com/forum/index.php?topic=2152.msg15825#msg15825 Joe L.
June 26, 200818 yr Author Are there any disadvantages to disabling the logging entries (are they used by unraid to figure out what files need to be deleted from the cache drive at a latter time or doe unraid delete the file as it is copied/moved to the share drive?)? At the moment it seems that I have to disable it so there is an immediate advantage. I discovered that in between log entries there was some info I missed and should of put in the file I uploaded, so I have uploaded a new file with this missing info, which is labeled as ********* "Section I missed when uploading the file the first time" I am in the process of recovering files from a 500gig drive. I am using virtuallab or by binarybiz to recover the data. There were a lot of files on the drives. Mostly files I probably will never use but I keep archived just incase there is a need for this old data. The files my programs produce for one project could be 200 or so. I do multiple projects a day and some times rerun the same project 3 or 4 times to try and get better results, and if you figure 13 years of doing that plus some redundancy of old back ups and the files start adding up. I am always short on house cleaning time and I still have not found a "round-to-it". I have tried to delete all the starting and closing info in the syslog file and only have the logging entries that shows line by line what file is being handled, then copy and paste these lines to excel so I could see how many lines there were, but excel says that it could not handle all the lines (excel only allows for something like 66,000 lines) I can't figure out ow to get a count on the qty of entries so we can see how many files were handled. If the logging entries don't really help much and I can go without, then I will just make the change as suggested by Joe. Thanks for the suggestions.
June 26, 200818 yr That additional section was just another instance of the kernel killing off one more process in an attempt to free ram. In this specific case, it killed the unRAID web based management interface (emhttp) Jun 25 04:34:11 Tower kernel: Out of memory: kill process 1999 (emhttp) score 348 or a child Jun 25 04:34:11 Tower kernel: Killed process 1999 (emhttp) There is no added value to logging the cache file moves. They are not errors, just a log that the file was moved. Based on what you have shown can happen, Tom should probably remove the "-v" option permanently. I would add the " ! -exec fuser -s {} \; " piece, especially if you anticipate the "mover" script kicking in while you are still writing files to the cache drive. Joe L.
July 1, 200818 yr Author I have not done any editing to unRAID, prior to this. I telneted in and typed mc for midnight commander and then used the up and down arrows along with the enter key to navigate around to the "/usr/local/sbin/mover" location, then I used used the esc key along with the "4" key to edit the file. I arrowed down to the command line and removed the "-v" as recommended by Joe L. Thanks to Joe, I figured out how to do the editing from a different topic http://lime-technology.com/forum/index.php?topic=1922.0 I will try out "mcedit file_name" on my next go with editing, but I will have to do more homework to try "vi" (any good links for using "vi" with unRAID ??) The "! -exec fuser -s {} \;" command as Joe mentioned would be released in 4.3.2, and by the time I made the edit above I had just upgraded to 4.3.2 so I did not have to add "! -exec fuser -s {} \;" because it was already there. I would of fixed this sooner but had to much stuff going on. I should know by tomorrow am if this fixes my issue. Thanks for everyones help.
July 2, 200818 yr Author I was able to get to my share drives this morning without any issues. I also was able to get to the unRAID user interface via a browser. Thanks for the fix Joe.
July 2, 200818 yr I was able to get to my share drives this morning without any issues. I also was able to get to the unRAID user interface via a browser. Thanks for the fix Joe. OK, just be aware, the file will be replaced with an un-edited version every time you reboot the server. You might want to copy your newly edited version of "mover" to your flash drive. Log in as root, then, at the command line: cp /usr/local/sbin/mover /boot/mover To copy it back: cp /boot/mover /usr/local/sbin/mover In fact, once you have a copy on your flash drive, you might want to add that exact line to the end of the "go" script in the /boot/config folder. That way, every time you reboot, it will use your edited version. Joe L.
July 2, 200818 yr Author I was able to get to my share drives this morning without any issues. I also was able to get to the unRAID user interface via a browser. Thanks for the fix Joe. OK, just be aware, the file will be replaced with an un-edited version every time you reboot the server. You might want to copy your newly edited version of "mover" to your flash drive. Log in as root, then, at the command line: cp /usr/local/sbin/mover /boot/mover To copy it back: cp /boot/mover /usr/local/sbin/mover In fact, once you have a copy on your flash drive, you might want to add that exact line to the end of the "go" script in the /boot/config folder. That way, every time you reboot, it will use your edited version. Joe L. Thanks for the heads up on the edits going away after a reboot. I copied the mover file to the flash via "cp /usr/local/sbin/mover /boot/mover" and I did go in to edit the "go" script, but I was not sure if "cp /boot/mover /usr/local/sbin/mover" needs to be the first line, last line, or does it really matter? Also, is there a thread that covers a majority of commands that would be good adds to the "go" script or other files?
July 2, 200818 yr l Also, is there a thread that covers a majority of commands that would be good adds to the "go" script or other files? There are but we are waiting for Tom to comment on this thread http://lime-technology.com/forum/index.php?topic=1953.0 Other items which may be of assistance http://lime-technology.com/forum/index.php?topic=2167.0 And my first post with script examples on the subject. NOTE: The directory structure has changed, so the script path locations are different now. Pending is a comment from Tom, before I publish anything further. http://lime-technology.com/forum/index.php?topic=1511.0
July 2, 200818 yr Author Does the "cp /boot/mover /usr/local/sbin/mover" need to be the first line, last line, or does it really matter where it goes in the "go" script? (not sure if you saw this question buried in my last post so I am posting again. Thnaks for the links for the other commands.
July 2, 200818 yr I did go in to edit the "go" script, but I was not sure if "cp /boot/mover /usr/local/sbin/mover" needs to be the first line, last line, or does it really matter? I'd add it at the end, after the last existing line. but it probably does not matter. As described, once Tom has an idle moment or two and can find time to comment on the thread WeeboTech described, there might be a more standardized way of doing these things. Joe L.
Archived
This topic is now archived and is closed to further replies.