Running 5 beta 12.  Came home to find I could not write to network shares (received a windows IO error).  Went to shut down sickbeard and SABNZBD to do a restart (drives won't unmount with them running) and got error screens on both (python related, I think).  Tried a restart anyway, and drives would not unmount.  Did a hard restart.  Went to start up the array and two drives were shown as "unformatted".  Unraid was had automatically started a parity check (as it always does when it has a bad shutdown) and had found a few errors.  Stopped the check and attempted another reboot, but drives would not unmount (sickbeard and SAB were not running as they are not set to auto-install on reboot).

 

When the system came back up and the two drives were still showing unformatted, I started to get nervous.  Searched the forums and based on various similar situations I decided I shouldn't attempt anything else without asking for advice first.

 

My last full parity check was less than a week ago and found no errors at that time.  I've attached the current syslog, and here is a screencap of the UI showing which drives are reporting "unformatted": http://i.imgur.com/YG6FYai.png

syslog.txt

I haven't looked at the syslog, but I suspect if you had waited, the drives would have eventually come back online. I have seen the replay transactions part of an unclean restart take a very long time to complete, especially since it is competing with the necessary parity sync process. I would not have canceled the parity check, it's needed after a bad shutdown.

 

At this point you probably need to do a file systems check on those two drives, look for a post by dgasck and follow the directions in his sig line.

I've seen the array take a while to come back up after a bad restart, but the drives would say "resizing" while the system was doing its thing.  I've never seen "unformatted" before.  This is definitely something new.

 

Ran reiserfsck as per instructions in the wiki, but it found no errors.  Stopped and re-started the array again (to get out of maintenance mode), and the two drives are still reporting unformatted.  What next?

See here: http://lime-technology.com/forum/index.php?topic=5072.msg47037#msg47037

Are you sure this is the way to go?  My problem seems very different from the one in that thread, and my reiserfsck results are also different (screenshot attached).  I'm not using a gigabyte board, so that's definitely not the culprit.

THere are diagnostics described to determine if the partition actually starts at 63 or 64 vs. where unRaid expects it to start. Either 63 or 64 is correct.

my reiserfsck results are also different (screenshot attached).

Why did you check your first disk when your previous screenshot showed problems with disk 6 and 7?

Sorry, I didn't realize that's what "md1" was.  None of the drives are labeled md1 in the unraid menu, so I figured that was the designation for the whole array or something.

 

I tried the check again using hda and hdb, as well as md6 and md7 (figuring one of those had to be right) and this is what I got for all attempts:

 

 

 

Tried again for md8, just to verify, and that seems to be working.  The test is running now because I don't know how to interrupt it.

 

Grasping at straws:  The two drives are plugged into two on-board SATA ports that are physically next to each other.  They're the last two in a line of six ports.  These are the only two drives whose unraid designations start with "hd" - the others all begin with "sd".  Is it possible that they are on a separate SATA controller on the MB (a biostar A760G M2+) and that this controller has taken a dive?  I do have two spare ports on the supermicro RAID card that I could connect these drives to for testing.  Would this be worthwhile?

The BIOS needs to be set for AHCI.

It is.

Drives beginning with hd are configured as IDE. Are all drives configured a AHCI? It should be AHCI only. Combined mode is no good.

There was only a single point to enable AHCI, and it was on.  If I remember correctly, there was another option for "combined mode", which I think was enabled.  I'll double check tonight.

 

Last night, figuring "it can't get any worse, can it?", I tried moving the two drives off the MB ond onto the supermicro card.  They showed up as "hd" again, even on the other card.  Weird that it has worked this way for two years and now all of a sudden it has problems.

 

I re-ordered all the drives on the web interface so that everything was where it used to be and attempted to start the array.  This time there were 4 or 5 drives showing up "unformatted".  Crap.  Stopped the auto-parity, stopped the array, and shut the machine down (properly - it worked this time).

 

I'm pretty sure at this point that I have a hardware issue.  During my fiddling, I had to reboot the machine multiple times, and more than half the time, it failed to start back up completely.  It would either hang at a motherboard diagnostic page or it would lock up during the unraid booting process.  The MB diag screen shows voltages, and some of them looked a little weak.  One of the 12V lines was only showing 11.4V.  I am powering 12 drives off of a crappy came-with-the-case PSU that has been running 24/7 for over two years, so it could be on its way out the door.  I ordered a 400W PSU last night figuring it can't hurt.

 

I'm willing to throw whatever money is necessary at this thing to get it back up and running.  If I have to replace every component in the case to get my array back up, so be it.

New PSU installed.  Checked BIOS and combined mode was indeed enabled.  Bizarre that it has worked just fine for over two years like that with no issues.  Disabled combined mode, rebooted, started the array, and the same disks are still "unformatted".  At least they're now "sd" disks instead of "hd" disks.

 

What do I do next?

Sorry, I didn't realize that's what "md1" was.  None of the drives are labeled md1 in the unraid menu, so I figured that was the designation for the whole array or something.

 

I tried the check again using hda and hdb, as well as md6 and md7 (figuring one of those had to be right) and this is what I got for all attempts:

 

 

 

Tried again for md8, just to verify, and that seems to be working.  The test is running now because I don't know how to interrupt it.

 

Grasping at straws:  The two drives are plugged into two on-board SATA ports that are physically next to each other.  They're the last two in a line of six ports.  These are the only two drives whose unraid designations start with "hd" - the others all begin with "sd".  Is it possible that they are on a separate SATA controller on the MB (a biostar A760G M2+) and that this controller has taken a dive?  I do have two spare ports on the supermicro RAID card that I could connect these drives to for testing.  Would this be worthwhile?

 

It wants to re build the superblock. See here: http://lime-technology.com/forum/index.php?topic=5072.msg47037#msg47037 Ignore the comments about HPA.

Here's the disk data.  Should I still go ahead with the rebuild?

diskdata.txt

That looks good. What partition format is indicated when you click on the disk links on unraid main? Now confirm where the partition actually starts. See here: http://lime-technology.com/forum/index.php?topic=15385.msg144723#msg144723

Both drives say "MBR: 4K-aligned".

 

dd output attached.

 

dd.txt

They don't look correct. See here: http://lime-technology.com/forum/index.php?topic=15385.msg144822#msg144822 

Use the -A option.

I was continuing to have trouble getting the machine to boot properly, so I replaced all the guts.  Asrock H61M-DGS, celeron G1620, and 2GB of DDR3 RAM.  During replacement I noticed a couple caps on the old board right next to the main power connector that were definitely puffy.  Now it's booting up cleanly every time.

 

Partition script output attached.

partition_test.txt

OK. Run with rebuild-sb. See here for correct answers: http://lime-technology.com/forum/index.php?topic=1483.msg9973#msg9973  The answers have not changed.

Ran on disk 6 using answers in the thread.  I wasn't sure if it worked or not, so as the program suggested, I ran it again with --check.  It does not appear to have gone well.

 

The post you linked continues on about something called spinrite.  Is this something else I have to run?

rebuild.txt

The next step is "reiserfsck –scan-whole-partition –rebuild-tree"

 

 

The reiserfsck report shows that it was run against /dev/sdg rather than /dev/sdg1.    That is highly likely to have mangled things.  I would wait until one of the experts can confirm this and tell you the best way to recover from this!

Good catch. This is very bad. Use the previous script to repair the MBR and then try again.

I have learnt the hard way that it is better to put the array into Maintenance mode and run reiserfsck against the /dev/md?? devices.  That way you can avoid the issue of forgetting to specify the partition, and also means you do not need to be aware of the Linux device name to uNRAID slot mapping.  It also has the benefit that if parity is enabled then parity is maintained.