Since it looks like New Config is probably going to be the way out of this anyway, you could also just disconnect parity, new config with just the data drives, and access your files that way until you could get your hardware squared.

 

Even though one drive has quite a few reallocated sectors, no drive has pending sectors, and none of them are failing, so that approach would probably be as safe as anything most people without any sort of fault-tolerance do every day to access the data on their PCs.

 

You should only consider this a temporary solution though.

 

And maybe wait and see if someone else has a better idea.

You know that a main feature of unRAID is, that it is not a real RAID and you can put the disk/disks that has your files in a different computer and access the files you need

yea i know i can do this but the files are spread all over the drives. i would have to remove and check each drive to find all the files.

 

Since it looks like New Config is probably going to be the way out of this anyway, you could also just disconnect parity,

So if i remove parity, and run new config that would allow me to remount the array with out parity and access all of my data. once i have that straightened back out and a new power supply i could then reinstall the parity drive and rebuild parity based off the data of the the array drives. Correct?

 

follow up question to that approach. Perhaps my understanding is incorrect but how can disk3 show emulation if the parity drive is not emulating?

 

You know that a main feature of unRAID is, that it is not a real RAID and you can put the disk/disks that has your files in a different computer and access the files you need

yea i know i can do this but the files are spread all over the drives. i would have to remove and check each drive to find all the files.

 

Since it looks like New Config is probably going to be the way out of this anyway, you could also just disconnect parity,

So if i remove parity, and run new config that would allow me to remount the array with out parity and access all of my data. once i have that straightened back out and a new power supply i could then reinstall the parity drive and rebuild parity based off the data of the the array drives. Correct?

 

follow up question to that approach. Perhaps my understanding is incorrect but how can disk3 show emulation if the parity drive is not emulating?

And that's why I said wait and see if someone else has a better idea. It doesn't make any sense to me that it thinks parity is new but disk3 is emulated. In fact, the indicator seems to mean that disk3 is being rebuilt, rather than that disk3 is disabled.

 

If everything was working, and assuming parity and disk3 were correct, then even if disk3 was only partially rebuilt it should still be OK. But since the whole system was teetering on the edge before it finally toppled I don't know if we can assume the partial rebuild would be right.

 

So, I don't know for sure whether all your files will be accessible since the state of disk3 is ambiguous, and the only way to remove the ambiguity that I know is to do New Config. That will make unRAID try to use the disks as they are, for better or worse. If there is a problem of some kind with any of them, then maybe filesystem repair would help.

 

Can you read the (maybe emulated) disk3 now?

 

Since all your drives are reiserfs (or so it says), another possibility would be to boot a fresh copy of v5 from a different flash drive and just assign a few drives (since FREE only allows a few) at a time to access the data you need.

 

Also

Seems odd that every drive is 0 bytes free?

That seems to be what your 2nd screenshot  (the "before" picture) is showing, every drive all used and none free. Is that even close to correct? Can you read any of your drives?

 

Do you have other copies of any of your unRAID data? Despite the fault-tolerance provided by parity, unRAID (or indeed any RAID) is not really a backup unless it is used to keep copies of files you have somewhere else. I don't have another copy of everything on my unRAID (not even close) but I do have multiple copies of the really important stuff.

 

Before proceeding with any approach, you should remember to disconnect that "straw that broke the camel's back" drive if you haven't already. But wait until you have a decision on what to do before you shutdown or reboot in case there are any better ideas or any other information that might be useful.

 

PM or email limetech and refer them to this thread. Maybe an invalidslot command or something could be used to make it continue with the rebuild of disk3.

 

 

i removed the "straw" drive when disk 3 initially showed as unmountable before the parity issues arose.

 

I can not access the disk data but then the array is stopped. I do not want to start it as it will initiate a parity sync.

 

unfortunately i do not have other copies of my data yet. I was in the middle of getting everything sorted and centralized with the intent of backing up the important stuff after i completed the house cleaning. Its sheer bad luck this happened when it did.

 

I have emailed support. lets see if they can chime in.

 

So this is my understanding so far.

 

Disk 3 had been marked as unmountable previously. I followed the restore procedure and Disk 3 was in the middle of a restore when the parity somehow got flagged as a new drive. so i am assuming because the disk was in the middle of a restore the data on it is probably compromised. Which means that without getting the parity drive back online there is no way to rebuild the contents of disk 3. Is that correct?

 

 

 

another question. If i simply remove the parity drive and reboot the server will that allow the array to start in its current condition with whatever state disk 3 was in when the restore was happening?

anyone have any additional information or input on this? How about someone from the unraid support team?

anyone have any additional information or input on this? How about someone from the unraid support team?

They may not notice the thread unless you contact them.

Just some corrections first -

* Disk 3 wasn't emulated, although it would have been if the array could be started, but it couldn't, because 2 drives were wrong.

* Parity drive wasn't reset out of the blue, it was dropped earlier, and if you restart before rebooting it would not have been found a second time, which would have dropped it completely from the array, making it a new drive after reboot.

* You indicate that adding the new drive was no big deal, not a concern, but it seems to have precipitated the problems.  The theory that it contributed to the lack of sufficient power seems to me to be a very plausible one.  It would have been nice though to see the syslogs when the various drives failed, to know better what sort of errors caused them to be dropped.  Do you happen to have any of those syslogs?  I assume that newest drive is disconnected now?

 

And one additional problem - on Aug 9 between 5:30pm and 6:20pm, you had 7 MCE's, Machine Check Events, indicating a significant hardware failure somewhere.  An MCE is generally always hard to diagnose, but indicates a defect in the hardware somewhere.  I cannot say that it is in any way related to your drive problems though.  It could be, if it's power related.

 

And a question - the screen showing all drives with 0 bytes free, were you in Maintenance mode then?  You never mentioned it, but seems the only rational explanation (except it should not show any bytes used either?).  (I assume you know it's not safe to completely fill a drive up.)

 

In case you're wondering, your drive data should all be fine, looks like drives being dropped, not drives failing.

 

The suggestion to run New Config seems like the best idea to me, re-assign all of your drives, start the array, and grab the files you need first.  Whether you build parity new again or run a correcting parity check doesn't seem more important either way to me.  Your choice.  If you can disconnect the Cache drive for now, that would be safer, it has a sizable power draw.  But it may have important files on it?

 

It's possible that the power supply is at fault for all issues.  Nothing about the name gives me any confidence in it.  It's also possible that it was working fine before, but is starting to fail.  You may want to consider starting with only the important drives assigned, no new drive, no parity drive, no cache drive, etc.  Once anything of importance is safe, then you can re-assign additional drives, one by one.

RobJ has given you the right advice.  If it were me, and I suspect bad power or other bad h/w I would go one step further:

 

First, backup you flash device.  Just drag it's contents to a folder on your PC.  Next, identify which drive is Parity and put a big sticky note on it that says, "parity".

 

Now physically disconnect all your drives except one of them (like disk1).  Boot server, and execute the 'New config' utility on the Tools page.  Then go back to Main and assign just this one disk.

 

So now you are in a state where you have just one disk in the array (no parity).  Start array and see if disk mounts and is exported on network.  You can spot check the files and check the syslog for any indication of controller/disk errors.

 

Once satisfied everything works with one disk, stop server, maybe power down, connect another drive.  You won't have to execute New Config in this case, just assign it on the Main page.  Start array and perform same spot checking.

 

I say "maybe power down" because it depends on your h/w.  If you are using hot swap drive cages you can just plug the next drive in.  If you are using individual cables, probably best to power down.

 

During this process avoid writing any new files to disk because if you run across a device which needs to be rebuilt, you will still be able to try that - but don't attempt this until you report back the results of this procedures.

 

If the idea of repeating this process seem like a pain in the neck, you can maybe add 2 drives at a time.  The idea is that you want to add devices slowly until either you hit the magic number where the PSU gives up, or maybe you end up with all of them working.  At that point you can add Parity and try a parity sync.

You indicate that adding the new drive was no big deal, not a concern, but it seems to have precipitated the problems.  The theory that it contributed to the lack of sufficient power seems to me to be a very plausible one.  It would have been nice though to see the syslogs when the various drives failed, to know better what sort of errors caused them to be dropped.

Dont get me wrong. Im not saying it was no big deal. And i agree that it may very well have pushed the power supply over the edge. What I am saying is that drive was removed from the array as soon as i booted up the server and saw that disk 3 had become unmountable. So it has not been part of the equation during most of the resulting problems.

 

Do you happen to have any of those syslogs? 

No i didnt not think to capture syslogs during each event.

 

And a question - the screen showing all drives with 0 bytes free, were you in Maintenance mode then?

Unless unraid had somehow entered maintenance mode on its own i never physically initiated it. When disk 3 originally showed as unmountable i shut down the server checked all cables, rebooted saw that disk 3 was still unmountable and immediately proceeded with the disk restore procedure. It was during the disk restore procedure that the parity drive suddenly changed its state to new drive and that disk 3 changed from unmountable to emulation mode that is shown in the second screenshot.

 

I will disconnect all drives and start with just adding one drive at a time and report back on the results.

Ok all drives have been reset and cache and parity drives are still disconnected. Spot checking it appears the files are there but its impossible to know for sure just by visually looking at files. The only thing that is really suspect is my two 1TB drives are completely empty. This could be correct, that no files have ever been written to those drives yet but it seems a little suspect. I am assuming though that i would have had to manually wipe the drives in order for them to be completely empty correct? This couldnt be an issue of the drive problems i was experiencing?

Ok all drives have been reset and cache and parity drives are still disconnected. Spot checking it appears the files are there but its impossible to know for sure just by visually looking at files. The only thing that is really suspect is my two 1TB drives are completely empty. This could be correct, that no files have ever been written to those drives yet but it seems a little suspect. I am assuming though that i would have had to manually wipe the drives in order for them to be completely empty correct? This couldnt be an issue of the drive problems i was experiencing?

 

Before a drive is added to the array, Two things must happen. 

 

1----  First,  It must be 'Cleared'.  This can either be done by the preclear script or done as a part of the process when unRAID adds a drive to the array.

 

2--- Second, it must be formatted with the file system that the drive is going to be using.

 

If you look at those two 1TB drives, they show up as having the reiserfs file system on them. Both show that this file system used up a bit 64MB of space on the drive.  Furthermore the drives were each read 8 times and written 29 times.  They are both shown as healthy, so they are usable by unRAID!  They may NOT have files on them yet BUT they are NOT empty.  They have all the information to allow files to be written to them whenever you, unRAID or some plug-in decides to write a file there. 

 

It appears that you added those three 1TB drives to the array recently.  (All the drives are the same make and model.)  If you notice, the third 1TB drive does have files on it.  It about 30% full and it would normally be written to half full before the other two new drives would have files written to them.  You could run a SMART test on those two drives to see if there are any issues but I would also suspect that there are none. 

Agreed. Guess its just coincidence that they haven't had anything stored on them yet. That said it appears everything is in order. Is the next step to add parity back in and let it rebuild?

 

Agreed. Guess its just coincidence that they haven't had anything stored on them yet. That said it appears everything is in order. Is the next step to add parity back in and let it rebuild?

 

Sure.  But earlier in the topic you say your PSU is split-rail with 12v1 = 18A, 12v2 = 20A.

http://lime-technology.com/forum/index.php?topic=42089.msg400256#msg400256

 

With 12 drives you are right on the edge man.  Bad/marginal power produces very strange issues.  A lot of people have spent a lot of time helping you out.  If you get into a situation where multiple drives start dropping out again, or other 'strange' issues such as random dma errors, etc. start happening, and you haven't replaced that PSU, then you're going to be on your own.  Ok?

 

With 12 drives you are right on the edge man.  Bad/marginal power produces very strange issues. 

 

Ive decided to pull three of the 1TB drives and just replace with a single 4tb Drive. That will eliminate two additional drives as well as the need for the rocket raid card. If i continue to get dropped drive issues or decide to add more drives in the future ill look at replacing the PSU.

So Problems still persist.

 

I removed the three 1TB drives and had everything up and working for a few days. (Screen shot 1)During a parity sync it dropped disk03 from the array again. Showed 444k write errors on disk 3. See screen cap 2(forgot to screen cap before reboot so its not showing the failed writes)

 

I then ordered a new seasonic 850w x-series gold power supply. Installed the new power supply, created a new config added all the drives back except disk 03. ran parity sync again and everything was good. I figured with the new power supply i could back in my my three 1tb drives.

 

Cleared and added the drives again everything was fine. Woke up this morning and it has now dropped the parity disk from the array. (screen shot 3). I have not rebooted the server yet so was able to grab the current diagnostics. also attached.

screen cap 2

screen cap 3

diagnostics

tower-diagnostics-20150904-0644.zip

Some stuff in your syslog that I'd not seen before, as unRAID handled the addition of 3 new drives. I think it's probably all normal though.

 

Of course, your parity drive was disabled due to a failed write, but there isn't a smart report for it. Maybe check your connections again.

Is there anything else I should grab before i shut down the server?