Long Story short.. power failure during a time when i was off my UPS.. (i know i know.. its back now...)

 

13 Drive array (1 parity + 12 data disks, no cache)..

running 4.7 Pro for YEARS without any issue.. love this system..

screen shot of what my current situation is...  i have 2 brand new 2tb drives ready to go in.. but i want to try to preserve at least whats online and good now..(10 disks of data)

 

https://www.dropbox.com/s/mqfhsz8h38iazkj/unraiddrivestatus4.7.jpg?dl=0

 

im "ok" with the loss of these 2 data drives if its too late.. but i thought id check.. maybe 1 of these 2 can still be rebuilt..  long term goal is the array re-protected and 12 disk capacity again (even if new drives both need to be used)

 

what processes/commands would you guys like run/posted? please be gentle/explain steps a bit.. i can run commands OK but i'm a newb as for so long this setup has been rock solid i never had to tshoot any of this before.. only ever had 1 drive die out and replaced no issues a couple times.. this seems more complex...

 

 

 

There is an excellent chance the data on those disks is basically all there, but some file system level has occurred that stops them being mounted. This can easily happen during a power cut.  UnRAID displays the unformatted message any time the mount fails - it does not necessarily mean the disk is really unformatted.

 

This can be checked by stopping the array, and restarting it in Maintenance mode.  Then from a console/telnet session run a command of the form

reiserfsck --check /dev/md10

The command can take a while to run so if using telnet ensure you keep the telnet window open for the duration.  The output of that will indicate whether corruption has been found and if so suggest the recovery action.  Repeat for the other disk disk.

 

If a recovery action is suggested then it is highly likely that doing it will get the data back intact.  However you may want to check back here before carrying out the suggested action.

awesome.. i will run these commands and report back with the results.. i have opened the case and confirmed/recheck the seating of all cables and i "think" thats not any of the issue.. but im not sure..

 

i will attempt your steps and post back here with the results.. THANK YOU!!!

 

-Matt

 

 

please forgive me and my newb-osity

 

i have rebooted the server and i dont have any option for "maintenance mode" is is a command line thing as well?  here is what i'm seeing if i stop the array after it boots up.

 

 

https://www.dropbox.com/s/k2s0fb11f2h5k55/arraystopped.jpg?dl=0

Most of us don't remember how this works on 4.7

 

aye.. gotcha.. i know its ancient at this point.. i wondered if this was a 5+ feature

 

 

my assumption is that i need to settle/solve these drive issues.. get the array up fully and protected again.. AND THEN.. look into upgrading the version of unraid.. seems anything else would be

foolish

 

 

aye.. gotcha.. i know its ancient at this point.. i wondered if this was a 5+ feature

 

 

my assumption is that i need to settle/solve these drive issues.. get the array up fully and protected again.. AND THEN.. look into upgrading the version of unraid.. seems anything else would be

foolish

I would tend to agree.

 

Maybe search the forum. See search tips in my sig.

https://www.dropbox.com/s/jpmw3fayy5hld67/towerputty.log?dl=0

 

 

ok.. 2 msgs.. one thats a "not found" for "md10"... so i guessingly.. looked at my devices tab (array is STOPPED).. and that drive is /dev/sdi... so i tried that and got a different msg..

 

 

not sure about that rebuild option it suggests... gurus?

update to above..

 

disk 12 is "sdg" so i retried with that drive instead.. same msgs in putty for that command as well (the rebuild block one)

update to above..

 

disk 12 is "sdg" so i retried with that drive instead.. same msgs in putty for that command as well (the rebuild block one)

Ok, with 2 drives down parity is useless, so working on the drives themselves is probably your best bet. HOWEVER... when working directly with the drives (/dev/sd?) you have to specify that you want to operate on the first partition (/dev/sd?1). So, try the check again with /dev/sdg1 and see what message you get. If you do a rebuild on /dev/sdg without the 1 at the end, you will corrupt whatever data is there.

ok.. its running with sdg1.. thanks for that

 

updated log as it goes:

https://www.dropbox.com/s/jpmw3fayy5hld67/towerputty.log?dl=0

sdg1 check command complete

 

log: https://www.dropbox.com/s/jpmw3fayy5hld67/towerputty.log?dl=0

 

pic: https://www.dropbox.com/s/t3uukhrf1u14i1y/sdg1checkdone.jpg?dl=0

 

ill execute with sdi1 for the other drive and post the set for that next

 

 

sdi1 check completed (aborted)

 

so this drive seems much worse.. likely a loss im guessing..  thats "OK".. even a loss of both id deal with.. not losing the entire array other drives is my hope/focus

 

 

sdi1 pic: https://www.dropbox.com/s/9rvyp4f7z939kbp/sdi1checkdone.badnews.jpg?dl=0

 

sdi1 log: https://www.dropbox.com/s/eet0b9wr67k53k2/towerputty-sdi1checkdone.log?dl=0

After looking at the first pic you posted, I must ammend what I said. I am not at all sure parity is useless at this point, and I don't think it's a good idea to write directly to the devices until we can determine what is going on. It appears in the first pic that you have one drive that is red balled (a write failed so unraid is emulating it using parity) and an orange ball, which I don't remember for sure, but maybe unraid just can't get it to mount cleanly. If the check came back clean on the drive that appears as a red ball in the first pic, I think there is a good chance that you could use the command line option to redo the drive configuration and trust parity. Thing is, I don't remember the sequence off hand, and I'm uncomfortable telling you to do things blindly at this point. For the best chance at getting things back, I suggest emailing Tomm at limetech to get his help. He's really good at keeping up with email, so you have a much better chance that way than here in the forum.

well thanks.. yea im happy to email him

 

the orange ball drive is actually the one where it ran cleanly.. sdg..

 

the red ball drive is the..sdi.. where its talking about a hardware failure of the disk and aborted the process

 

 

 

 

 

sdi1 check completed (aborted)

 

so this drive seems much worse.. likely a loss im guessing..  thats "OK".. even a loss of both id deal with.. not losing the entire array other drives is my hope/focus

 

 

sdi1 pic: https://www.dropbox.com/s/9rvyp4f7z939kbp/sdi1checkdone.badnews.jpg?dl=0

 

sdi1 log: https://www.dropbox.com/s/eet0b9wr67k53k2/towerputty-sdi1checkdone.log?dl=0

If sdi is the drive with the red ball in the first pic, that's actually to be expected. Unraid probably failed the drive because of a hardware fault. Perhaps you should pull the drive and replace it with one of your spares so unraid can rebuild it. Rebuilding won't allow the drive to be mounted, it will probably look exactly like it does right now, but at least you will be able to do file system checks and hopefully get your data back.

I think in 4.7 you have to manually umount the drives so you can run the checks against /dev/md? devices, and the diagnostic mode in later versions did that automatically.

 

PLEASE email Tom, and point him to this thread. If this were my data I wouldn't do anything else until he comments.

yea i have done as you suggested.. emailed him with a pointer to this thread..

 

ill wait for now and see what he comes up with to attempt next

 

 

thank you jonathanm for all your help/thoughts

See check disk file systems in my sig. Replace the red drive if you have a spare. Post a smart report for the current red drive.

I can't think of any way you can salvage parity at this point, as there almost certainly have been lost writes, plus transactions and journals replayed, plus one drive with unreadable sectors.  I don't think the problem is necessarily serious, although it could be, but it only takes one write failure to fail a drive.  And if the 2 drives were on the same IDE channel, real or emulated, the failure of one can cause the pseudo failure of the other.  It really really would have been better to have upgraded by now.  Just sayin' ...

 

I'm not Tom, but what I would advise for now is collect SMART reports for all your drives, and run the reiserfsck --check on all of your data drives (not the parity drive).  You can get the drive symbol from the Devices page, and as stated above, make sure you add the 1 to it (e.g. sda1, sdb1, etc).  Hopefully you have previous SMART reports to compare these new ones with?  What we want to do is determine the exact condition of the drives (both the physical sectors and the file system), and apply the appropriate fix.  You can also run SMART short tests on all the drives (run it, wait 3 minutes, then obtain another SMART report and make sure the test result was fine).  Then try a SMART long test on the 2 bad drives, to see if it can complete.  It probably can finish on sdg Disk 12, probably cannot on sdi Disk 10.

 

If there are bad sectors, then the best option I think is to run badblocks with the non-destructive option, which *should* force a rewrite in place of the bad sectors, hopefully forcing either a sector recovery or sector replacement.  You may want to wait for more advice, and decide what to do.  Personally, I think your parity is already a writeoff.  If you read up on badblocks and decide to try it, the form of the command I recommend is (do not add the 1 this time, it operates on the drive not the partition, and replace X appropriately):

 

  badblocks -b 4096 -c 256 -nsv /dev/sdX

 

It takes a VERY LONG time!  It will do its best to preserve all data.  It saves a megabyte of data, writes a megabyte of test patterns, tests it, then writes the data back and moves on.

 

As you probably know, any time you reboot, recheck the drive device symbols for changes (e.g. sdi changed to sdf, etc).