February 21, 201511 yr Hey All, Out of the blue yesterday my unRaid 5.0.5 box started to hard crash on me. I can't seem to get much information on it aside from the screenshot attached. I really don't know my way around Linux so not sure where to look for dump files or error messages ect. When the crash occurs the box is completely non-responsive both from a remote terminal, unraid web interface, shares or the command line itself. Upon reboot the box seems to stay up for about 5 minutes then crashes again. Any help would be GREATLY appreciated. syslog.txt
February 21, 201511 yr Author I have turned on the logging in this thread just now, perhaps that will give me some insight. http://lime-technology.com/forum/index.php?topic=9880.0 Attached is the output that came on the screen whit the command running new__1.txt
February 21, 201511 yr Author After googling around and seeing some other posts I ram a memtest and immediately saw errors. Pulled out one of the DIMMS and reran memtest and there were no errors. Booting up unRAID now to see if this solved the problem. Will report back. EDIT Hmm...now a new problem. I have brought the unRAID server back up but I am not getting a report of an unformatted drive. It has a green ball but it is showing unformatted.
February 21, 201511 yr Author So I have rebooted the array and unRAID is still showing one unformatted drive. This is terrible news because I have already found missing data. Anybody out there know how to recover from this and get my data back?
February 21, 201511 yr So I have rebooted the array and unRAID is still showing one unformatted drive. This is terrible news because I have already found missing data. Anybody out there know how to recover from this and get my data back? The first thing to remember is to Never click Format. Your best bet is to actually wait for at least 30 minutes and examine the log file. Most likely the system is replaying transactions that were not flushed out when your system crashed. Thats likely why the drive was unmountable. Tail -f /var/log/syslog If that is the case, then once the transactions are pkayed back and finish, then a reboot of the system should fix the issue as the drive will then be mountable.
February 21, 201511 yr Since there was a hard crash it is also possible that some sort of file system corruption has caused to the disk to be unmountable. This can nearly always be fixed using the reiserfsck utility.
February 21, 201511 yr Author Thank you both for the reply, sincerly appreciate it. I will wait a couple hours and reboot and see what happens. If that doesn't work I will then run reiserfsck utility. I will report back in a couple of hours. I did not click the format button so the drive is still sitting there untouched.
February 21, 201511 yr Author OK so bad news. I waited a couple hours and rebooted. No luck, the drive still shows up as unfofrmatted. I then ran reiserfsck as per the directions here http://lime-technology.com/wiki/index.php/Check_Disk_Filesystems root@NAS:~# reiserfsck /dev/md9 reiserfsck 3.6.24 Will read-only check consistency of the filesystem on /dev/md9 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes reiserfs_open: the reiserfs superblock cannot be found on /dev/md9. Failed to open the filesystem. If the partition table has not been changed, and the partition is valid and it really contains a reiserfs partition, then the superblock is corrupted and you need to run this utility with --rebuild-sb That is the output that came up and per the instructions I have not run it yet with the argument --rebuild-sb so I am posting here for more help please.
February 21, 201511 yr Author So I removed the drive from the array and stated up the array to see if I could get to the data. No dice, didn't show any of the missing data. I then added the drive back to the array and I am now getting a BLUE BALL suggesting a new drive is added. What will happen if I start the array? Will unRAID rebuild my missing data on to the drive?
February 21, 201511 yr So I removed the drive from the array and stated up the array to see if I could get to the data. No dice, didn't show any of the missing data. I then added the drive back to the array and I am now getting a BLUE BALL suggesting a new drive is added. What will happen if I start the array? Will unRAID rebuild my missing data on to the drive? unRAID can only rebuild what was there before. If you couldn't see any data with the drive removed then you won't see any after a rebuild. It will just leave you where you were, with an unformatted drive.
February 21, 201511 yr Author So I removed the drive from the array and stated up the array to see if I could get to the data. No dice, didn't show any of the missing data. I then added the drive back to the array and I am now getting a BLUE BALL suggesting a new drive is added. What will happen if I start the array? Will unRAID rebuild my missing data on to the drive? unRAID can only rebuild what was there before. If you couldn't see any data with the drive removed then you won't see any after a rebuild. It will just leave you where you were, with an unformatted drive. Anyone have any advice then, all of this data was there before the hardware failure and my parity was checked not long ago with 0 errors? Where would all of my data gone?
February 21, 201511 yr Author Anyone?? Please I don't want to lose 3TB of data. This would be the end of unRAId for me.
February 21, 201511 yr Anyone?? Please I don't want to lose 3TB of data. This would be the end of unRAId for me. That's a bit extreme, considering that the root cause of this was bad memory. You probably do need to run reiserfsck with the rebuild-sb option. However, DO NOT DO IT until an expert on running this pipes in (and I am not one). If you fail to answer the questions correctly you WILL lose data.
February 21, 201511 yr Author Anyone?? Please I don't want to lose 3TB of data. This would be the end of unRAId for me. That's a bit extreme, considering that the root cause of this was bad memory. You probably do need to run reiserfsck with the rebuild-sb option. However, DO NOT DO IT until an expert on running this pipes in (and I am not one). If you fail to answer the questions correctly you WILL lose data. Thanks for the response but, like you say, this was bad memory and if bad memory can bring down a 3TB drive then unRAID probably isn't the best solution for me. Hoping an expert can chime in on rebuild-sb
February 21, 201511 yr Anyone?? Please I don't want to lose 3TB of data. This would be the end of unRAId for me. That's a bit extreme, considering that the root cause of this was bad memory. You probably do need to run reiserfsck with the rebuild-sb option. However, DO NOT DO IT until an expert on running this pipes in (and I am not one). If you fail to answer the questions correctly you WILL lose data. Bad memory can bring down a drive on ANY system. On most NAS systems it would probably bring down ALL drives. Having said that the ResierFS system that is used in unRAID v5 is very resilient and there is an excellent chance that as long as one goes about it carefully the majority of the data on the disk can be recovered. If it is only the start of the disk that has been corrupted then maybe all the data can be recovered. Thanks for the response but, like you say, this was bad memory and if bad memory can bring down a 3TB drive then unRAID probably isn't the best solution for me. Hoping an expert can chime in on rebuild-sb
February 21, 201511 yr Bad memory can bring down a drive on ANY system. On most NAS systems it would probably bring down ALL drives. Just thought that this really needed to be emphasized. It's why the IT department in most enterprises will replace all the memory if a single ECC error ever happens.
February 21, 201511 yr In terms of recovering from the problem is a spare 3TB drive available? If so then the safest thing to do is to first make a sector-for-sector copy of the current drive before trying recovery. Since the superblock seems to have been corrupted the recovery process is very sensitive to the right answers being given when trying to recover the disk using the --rebuild-sb option to reiserfsck. In my experience the defaults are normally the correct answer, but if you have a copy of the disk you are in a position to have another go if that turns out to not be the case. If there is not a spare disk available, then the best thing to do will be to remove the physical drive and try and recover on the emulated drive (/dev/md9 I believe). If that succeeds then afterwards you can 'rebuild' the corrected drive onto the physical drive to get the array back to an operational protected state.. If it fails you still have the physical drive to fall back on for further recovery attempts.
February 21, 201511 yr Anyone?? Please I don't want to lose 3TB of data. This would be the end of unRAId for me. Any data loss is almost certainly due to issues that were caused by your failed memory, which can cause very unpredictable results on ANY system. That's not a problem unique to UnRAID. Neither UnRAID nor any other storage system -- whether fault tolerant or not -- is a substitute for backing up your data. If you do indeed lose data, the lesson isn't that you need to abandon UnRAID ... it's that you DO need to backup your data. But before assuming your data is lost, I'd do the following: (1) Connect the failed drive to a PC; install the free Linux Reader [ http://www.diskinternals.com/linux-reader/ ]; and see if you can "see" any of the data on the disk. DO NOT write to that disk in any way ... just see if you can see the data; and, if so, copy it from the disk to another location. (2) If #1 doesn't work; then put the disk back in your UnRAID box and run Reiserfsck again to rebuild the superblock. Be CERTAIN that you are running it against the correct disk -- AND that your answers are all precise, as any error will result in an incorrect Superblock. I would print out and thoroughly read both Brian's post [ http://lime-technology.com/forum/index.php?topic=1483 ] and this post: http://lime-technology.com/forum/index.php?topic=4021.msg35473#msg35473 Although they are nearly 5 years old, I believe they're still accurate for the --rebuild-sb option. After you rebuild the superblock, I'd run a --check again before doing anything else (i.e. don't to a rebuild-tree unless the check advises you to) If you're going to use this option, wait a bit and see if Brian (bjp999) comments -- I've sent him a note asking that he confirm that his outline for using --rebuild-sb is still correct. He's one of the better "Linux gurus" on the forum ... so wait and see if he has additional guidance before you do this.
February 21, 201511 yr Author I will head out to the store and get a spare 3TB drive to create a backup of this and then try the suggestions offered by Garycase and itimpi
February 21, 201511 yr I am out and about with no access to my array. I believe that the reiserfsck instructions are unchanged. I helped a user several months ago (maybe a year) and he verified that the rebuild superblock parameters were unchanged. Good luck!
February 21, 201511 yr Author OK so I have run reiserfsck --rebuild-sb /dev/md9 and it finished. I have now run reiserfsck --check /dev/md9 and this is the response root@NAS:~# reiserfsck --check /dev/md9 reiserfsck 3.6.24 Will read-only check consistency of the filesystem on /dev/md9 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Sat Feb 21 16:19:56 2015 ########### Replaying journal: Done. Reiserfs journal '/dev/md9' in blocks [18..8211]: 0 transactions replayed Killed I am assuming that is bad yes?
February 21, 201511 yr I suspect that's the case ... but wait until bjp999 chimes in to confirm. If you still can't "see" any data on the disk, I'd be inclined to do a --rebuild-tree, but again, wait for a Linux guy (bjp999) confirms that.
February 21, 201511 yr Author I suspect that's the case ... but wait until bjp999 chimes in to confirm. If you still can't "see" any data on the disk, I'd be inclined to do a --rebuild-tree, but again, wait for a Linux guy (bjp999) confirms that. Found a great thread (I am a relentless searcher)http://lime-technology.com/forum/index.php?topic=6815.10 telling me that Reiserfsck is running out of memory. I only have 1GB now since I removed the other 1GB stick so I have to figure out how to do this with just 1GB. This is an old box and I don't have any other DDR2 RAM kicking around. Going to follow the instructions by Joe L in the thread and see what happens.
February 21, 201511 yr Good plan -- Joe L is probably the best guy on this forum for Linux issues (He's been a bit scarce lately ... but his advice is always excellent)
February 21, 201511 yr Author So I followed the instructions by Joe L but unfortunately no dice. Reiserfsck still killed early. Can you check you syslog to see if there are out-of-memory errors at the time the reiserfsck aborts? If there are, and the kernel out-of-memory process is killing reiserfsck, then we can try to minimize the use of the disk cache hogging the RAM. Did you try setting the cache_pressure to 200? sysctl vm.vfs_cache_pressure=200 Did you clear the existing buffer cache to free up as much memory as possible? sync echo 3 > /proc/sys/vm/drop_caches If so, I suggest also trying: 1. Disable user shares temporally. (They use up memory which is in short supply.) You can re-enable them once the disk is repaired. 2. Creating a swapfile in memory, bypassing the disk cache entirely. I know this sounds strange, but if we can keep the out-of-memory kernel process from killing the reiserfsck, it might just finish. This will reserve some amount of memory just for "swapping" This will reserve 100Meg of your ram as a file-system to hold the "swapfile" echo 10 >/proc/sys/vm/swappiness sysctl vm.vfs_cache_pressure=200 sync echo 3 > /proc/sys/vm/drop_caches mkdir /mnt/swap_fs mount -t ramfs -o size=100m ramfs /mnt/swap_fs dd if=/dev/zero of=/mnt/swap_fs/unraid_swapfile bs=1M count=100 The above commands will create a 100 Meg file on the RAM file system in memory. This space is "reserved" and will not be used for disk cache, and will not be "swapped" out. Next, Initialize the swapfile. mkswap -L SWAP /mnt/swap_fs/unraid_swapfile chmod 600 /mnt/swap_fs/unraid_swapfile Then, Enable the use of the swapfile /sbin/swapon -v /mnt/swap_fs/unraid_swapfile To free up some more RAM, stop SAMBA and even the unRAID web-interface. killall emhttp /root/samba stop Then, try the reiserfsck once more... To re-start the web-interface so we can stop the array and reboot, type: cd / nohup /usr/local/sbin/emhttp & Good luck. Joe L.
Archived
This topic is now archived and is closed to further replies.