I have probably a mountain of issues here so hopefully all is not lost 

 

My unRAID box has been off for a few months while work has been getting done on the flat and I'm just at the point now of firing everything back up again.  I had a drive redball a few days ago however I know there is nothing wrong with the drive as I'd forgotten the backplane was faulty.  I've since run an extended smart test on the drive which came back fine (although I didn't realize it would take hours to complete lol).  So I set the "new configuration" and re-assigned all the drives (checking the parity is valid option) and everything started up again fine..  This is where I remembered the second issue the box was having before its long holiday.  A parity check started when I started the array and this was running between 800k-2MB and a loud clicking noise could be heard within the chassis.  I've got 16 drives (14 excluding parity/cache) and have no idea which one is failing.  I checked the syslog however this didn't give any answers on the bad drive.  What was even more worrying was the amount of errors about bad parity (the check box had somehow been selected to correct parity - something I don't recall having checked before I started the array.  I immediately cancelled the check and stopped the array.  While only a few minutes passed between starting and stopping the array I am now worried I have a potentially compromised parity drive.

 

 

So my questions are..

 

1.  What would be the easiest and quickest way to identify the failed drive?

2. Once the drive is identified what would be the least destructive way of replacing the drive assuming the parity drive has bad data?

3. Am I safe to upgrade to 12a before I try and fix the above or should I wait until everything is sorted.

 

 

 

System Specs:

 

unRAID 5.0 rc8a (running within ESXi 5.1)

Supermicro X9SCM

Xeon E1230

24GB RAM (3GB assigned to unRAID)

2x Dell H200 RAID cards

 

 

1. Power on each drive one at a time in a new system and perform some kind of read opperation if your not immediately able to hear the clicking.

 

2. Parity is toast (at least I wouldn't trust it now.) I would try to manually copy over a suspected dieing drive to a new one.

 

3. Don't upgrade until you get a working system again.

 

Sent from my SGH-I727R using Tapatalk 2

 

 

1.  What would be the easiest and quickest way to identify the failed drive?

2. Once the drive is identified what would be the least destructive way of replacing the drive assuming the parity drive has bad data?

3. Am I safe to upgrade to 12a before I try and fix the above or should I wait until everything is sorted.

 

Okay, the simplest question is #3 ==> Yes & No.  Yes, you should use RC12a;  but No, you shouldn't bother to upgrade.    Just start from scratch => revert to "stock" UnRAID if you have any plugins;  and update the files to the RC12a version.   

 

As for #1 ... just follow the process I'll outline below.

 

r.e. #2 ... you just replace it with a new drive.  Then copy the missing data from your backups.

 

As for getting your array back in order, do the following ...

 

=>  You need to know for sure which drives were parity and cache.    If not, if your system still boots, then note the serial #s of those drives before you start this process.  You do NOT want to add them as data drives while you're "building up" the new array !!

 

Now boot to UnRAID, and click the "New Config" button (on the Utils tab).  Assign ONE drive to your new configuration, with NO parity or cache drive.    Then start the array.  Now access the new disk from Windows Explorer [ \\Tower\disk1] and confirm that the disk you just added can at least be read okay (this doesn't guarantee it's good -- those checks will come later).

 

Now add ONE add-on to the system ... UnMenu.    I presume you know how to do this -- if not, just read the details in the Wiki.    Then, on UnMenu's Disk Management tab, select the disk you just added (the first time you do this it will be the only choice), and get a SMART Report and run a Short SMART test.  You COULD also run a long SMART test ... but this takes HOURS, so you probably don't want to do so at this stage.

 

Note:  I BELIEVE that UnMenu will also let you select drives that aren't yet part of the array, but don't have a convenient way to check that.    If that's the case, then you should do the following for the remainder of the drives:

 

(a)  Get a SMART Report; then do a Short SMART test.

(b)  If both are good, add the drive to the array -- and confirm it's accessible okay by starting the array and accessing the drive from Windows Explorer.

 

Repeat (a) & (b) until all drives are in the array.

 

If the drives have to be part of the array to run the SMART tests; then add them to the array first; then do the SMART checks; then test access.

 

If you find a drive that fails SMART, I would NOT add it to the array ... or would remove it if it's already been added.

 

If ANY drive that you add shows up as Unformatted, do NOT Format it ==> simply remove it from the array.    You can address that drive later.

 

Once all drives that pass SMART are part of the array, add the parity drive, and initiate Parity Sync.  Let that finish (many hours).    Once that completes, run a Parity Check (this will also take many hours)

 

... You will now have an up-to-date (RC12a) system with good parity, and all of the good drives in the array will be protected again.

 

At that point, I'd shut down;  add a new drive;  reboot;  and initiate a Pre-Clear on the new drive [This runs in the background and does not impact array operations ... read the Wiki for details on how to do this].    Once the drive is pre-cleared, you can add it to the array in just a few minutes.

 

Obviously this process will take several days [primarily for the parity sync & check, and the pre-clear].

 

Post a syslog. zip if needed. Examine all SMART reports for current_pending_sector RAW VALUE greater than zero. SMART tests are not needed; pull the SMART logs from each drive and post them.

Thanks for the reply's guys.  I've not been ignoring you, have just been trying to figure this out.  Parity is a write off - think there might actually be a bit of an issue with this drive but I've not run any tests on it yet.  Gives me a chance to swap it with a WD Red as they perform better than the green drives.  I've gone for the good old fashioned troubleshooting method of shoving the drives in an external esata caddy one by one.  I've got 4 drives making slightly more noise than I'd like, one wouldn't read two days ago but seems to be working quite happily again now.  Most of the failing drives are old 1TB ones that I was planning on replacing soon anyway so hopefully I should be able to recover the data from them before they fail outright.  I did notice something crazy earlier today with drive SDO (5YD6TH26) where it showed a few million reads/writes, had no temperature displayed next to it (but was showing as spun up) and had a few errors marked (this was all from the unRAID main page).  The logs showed a load of errors for the drive as well but yet it didn't redball it.  The drive showed when I connected to it through windows but when I went into it it was empty.  Re-booted the server and it was showing again as normal.  I thought I had a copy of that syslog but I don't seem to be able to find it now  . 

 

 

Hopefully the current syslog and smart logs for all the drives give some ideas.  I'm hoping to get 3 or 4 3TB WD REDS to replace all the 1TB dinosaurs and possibly a 2TB Seagate drive that I think may be starting to cause bother.  On a plus note I'm now on 12A and seem to have managed to get the box to join the domain which is an added bonus (although can only write to shares as the domain admin which is no good but that's a whole separate thing!).  I'm also running without any plugins now as it is a fresh install.  Only installed unmenu which I don't really see as a plugin.  Was wanting to move most my plugins to another VM so that's another sort of tick out of all of this.

 

 

unraid_logs_30-05.zip

Sounds like you're making progress ... and will end up with a nice new much-larger (and more reliable) array in the process.

 

If you have doubts about the data integrity on the drives, I'd just build your new array and populate it from your backups (assuming you have backups).    If you aren't backed up, then I'd still build the new array, and just copy the data across the network from your eSATA caddies one-disk-at-a-time from those that are still readable.    Since parity's "hosed", you'll only be able to get data from those disk you can read anyway, so there's no real reason to bother re-creating an array from the old, to-be-replaced, drives.    That's one of the nice things about UnRAID ... if this was a traditional RAID, you' HAVE to try to recreate it to read any data.  With UnRAID, it's much simpler.

 

... Of course you SHOULD have backups of all your data -- RAID is NOT a backup.  But I know a lot of folks simply don't do that.

 

Several drives show ATA errors about 100 hours ago. Was there some "event" about 100 power-on hours ago?

Several drives show ATA errors about 100 hours ago. Was there some "event" about 100 power-on hours ago?

 

I'm not aware of any issues however I did have to do a few "unclean" reboots as some of my plugins were miss-behaving so it could be something to do with that.  Although if a few drives show the same sort of thing at the same point I'd be more inclined to think one of my controllers started playing up.  I've not really had any issues with the H200's since I got them (other than my first post in this forum were I found out they were incompatible with 4.7 lol).  Bit irrelevant now seeing as I've got a few 3TB drives so would have had to upgrade anyway.

 

Both cards were on different firmware though.  Something I didn't realize until I moved my 3TB drives just after I had done the manual testing.  Seems the 3TB support was in a newer firmware so they are both running off the same version now which should hopefully cause less bother in future

 

 

... Of course you SHOULD have backups of all your data -- RAID is NOT a backup.  But I know a lot of folks simply don't do that.

 

 

I don't have any backups.  Most of the contents of my unRAID setup are media rips and while the thought of having to re-rip my collection is not a pleasant one it would not be the end of the world.  The whole point in me choosing unRAID was because there was the failsafe of the parity drive (assuming only one drive fails at a time) and of course the ability to still read individual drives in a failure, something which I'm now thankful for 

 

Well four 3TB drives have just arrived so just need to figure out the best way of setting this up.  Going to start a new array and ditch the 1TB drives (and I think my 2TB Seagate - 5YD6TH26).  Any other drives showing worrying signs of imminent failure?  Fortunately they all seem to be reading so I should be able to recover everything I need.  I've got an external E-SATA caddy that I can just slot the drives into but would it be quicker somehow mounting the drives outside the array and copying direct from one drive to another?

I don't have any backups.  Most of the contents of my unRAID setup are media rips and while the thought of having to re-rip my collection is not a pleasant one it would not be the end of the world.

 

No backups??  Ouch!!  I suppose the same thing is true for my collection ... I could always re-rip it all; but having spent a LOT of time over more than a decade building up this collection (approaching 4,000 movies) I really can't imagine doing it.  At perhaps 30 minutes/movie [rip, compress, copy to array, copy to backup, test at 32x playback]  that's ~ 2,000 hrs to redo.  I think a couple thousand bucks worth of backup disks safely "tucked away" in my fireproof/waterproof safe is well worth it to avoid any likelihood of ever having to do that again !!

 

I've got an external E-SATA caddy that I can just slot the drives into but would it be quicker somehow mounting the drives outside the array and copying direct from one drive to another?

 

The quickest copy will be to copy them to the new drives BEFORE you add a parity drive to the array.  You could either do this from within Linux (I'm not a "Linux guy", so I can't really help with that -- I THINK I know the right commands, but don't want to get it wrong, so I'll leave that advice to others  ), or (what I'd do) you can just connect them to a Windows box with the eSATA caddy and copy them to your array from there.  You'll need to install a Linux file system reader to do that ... this works well:  http://www.diskinternals.com/linux-reader/    If you create your new UnRAID system, but do NOT assign a parity disk, your file copies will be much quicker.  You WILL be running "at risk" until you finish the copies and assign a parity disk;  but as long as you don't destroy your old disks you'll still have all your data available if you had an infant mortality failure on one of the new disks (not likely, but it can happen).

 

 

Note:  r.e. infant mortality on your disks ==> You said you just got the new 3TB drives.  Do yourself a favor and either pre-clear the drives with Joe L's pre-clear utility; or at least run WD's Data Lifeguard and do a full "write zeroes" on the drives, followed by an extended test.    Either of these will take you a few days to do;  but they'll pretty much eliminate infant mortality issues.

 

So thought I'd resurrect my thread seeing as it's still a work in progress 

 

Turns out only one drive was causing issues although it looked like it was just starting to show signs of old age so I'm using it for something else that won't bother me if the drive suddenly dies I've retired the seven 1TB drives I was using and now have a few empty slots for bigger drives in the future. 

 

I should have probably mentioned the other reason I thought I had a few failing drives in my original post...  Parity speeds are slow, real slow.  Now I know I have one of the SuperMicro boards that has had issues but 50-60MB would be fine for the odd parity check tome (although 90-120 would be better ).  The slow I'm talking about is in the region of 200-300KB/s.  I decided to just start over when swapping the drives out so went from RC8a to 12a.  Everything has gone fine so far in copying all my data to the new disks although it took forever partly to do with a port somewhere in the network deciding to drop to 100MB.  Both the programs I used for reading the drives only had a progress bar and didn't actually show the transfer speed so I was "irritated" to put it nicely when I noticed the issue (which was about 6 drives in to the 8 I was copying over) 

 

Anyway...  All my data is back but not protected.  I've swapped the WD green drive that was the original parity drive for an existing WD red and that hasn't made a difference.  I'm also pretty sure the drive has been moved to another slot as well so that's not the issue either.  I've not got any read/write issues on any of the drives and can get over 70MB/s when I read and about 50-70MB/s when I write without a parity drive added.  Just tried doing a read with the parity check running in the background and the file transferred at about 50-60MB/s.  The system has 24GB of RAM installed but I've got 3GB assigned to the unRAID VM so I don't think the >4GB issue will apply to me.  Just upgraded to 15 (not 15a) and issue still the same.  I've not got any plugins running (haven't even got unmenu re-installed yet!).

 

Any ideas whats causing this?  syslog attached

syslog_20-06-13_2030.txt