Looks like I had a drive fail today. I've attached the syslog below. Maybe someone would be kind & review the last part of it & see if I'm right about the drive failing.

 

I couldn't see anything wrong in the smart reports. I completed the short test successfully. The long test would never complete.

 

I went crazy & went to Best Buy & picked up a Seagate 2TB drive.   I have it pre-clearing right now to make sure the drive is good. If it is OK then I will do a parity swap with it & use the current parity drive as the replacement drive for the failed drive.

 

I have already pulled the bad drive from the system. If anyone requests smart reports on the failed drive I can always put the drive in a different machine.

 

Thanks in Advance for the help.

 

P.S. I had a bunch of entries in the Syslog for duplicate files. I deleted these from the syslog to reduce its size.

 

Looks like I had a drive fail today. I've attached the syslog below. Maybe someone would be kind & review the last part of it & see if I'm right about the drive failing.

 

I couldn't see anything wrong in the smart reports. I completed the short test successfully. The long test would never complete.

 

I went crazy & went to Best Buy & picked up a Seagate 2TB drive.   I have it pre-clearing right now to make sure the drive is good. If it is OK then I will do a parity swap with it & use the current parity drive as the replacement drive for the failed drive.

 

I have already pulled the bad drive from the system. If anyone requests smart reports on the failed drive I can always put the drive in a different machine.

 

Thanks in Advance for the help.

 

P.S. I had a bunch of entries in the Syslog for duplicate files. I deleted these from the syslog to reduce its size.

 

It sounds as if you know what you are doing...

 

These lines in your syslog would have had me buying a new drive too:

Sep 18 13:53:41 Tower kernel: ata4.00: ATA-8: WDC WD5000AAJS-00YFA0, 12.01C02, max UDMA/133

Sep 18 13:53:41 Tower kernel: ata4.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 31/32)

Sep 18 13:53:41 Tower kernel: ata4.00: max_sectors limited to 256 for NCQ

Sep 18 13:53:41 Tower kernel: ata4.00: Drive reports diagnostics failure. This may indicate a drive

Sep 18 13:53:41 Tower kernel: ata4.00: fault or invalid emulation. Contact drive vendor for information.

 

Once you swap the drive assignments on the devices page, make sure you use the "Start" button to start the parity swap process.  This parity swap process will take a while.  Please keep us informed of its progress.  (I did it once on my array when I updated from a 750Gig to a 1TB parity drive.) Whatever you do, DO NOT use the button labeled "restore"  It is poorly named, and has nothing to do with data restoration.

 

Joe L.

Would be useful to see the smart report for the failed drive.  It would help you to know if it is RMA-able.

Thanks for the replies. I figured it was time to remove that drive. The drive is less than 2 years old. Bought it from Newegg. Every drive I have lost in the last year have been from them. Makes me kinda nervous. I've been buying retail since.

 

The 2TB drive is in step 2 of 10 on the pre clear script. It's copied 1.2TB in 10 hours. This is going to take a while. 

 

Once it's done I'll shut down & install the old drive & get the smart reports & run the WD diagnostic software on it. Can't turn off my personal machine until the clearing is done as I'm running putty on it to the unRAID box.

 

A little off topic but Newegg had the Seagate Retail Box 2 TB drive for 229.00 with free shipping. Best Buy had the same exact model for 219.00. I had a 5.00 dollar reward certificate so after tax I paid about a dollar less than if I had bought the drive at the Egg. Worked for me. 

Finally the 2TB drive has finished the pre-clear procedure. It took 20 hours 46 minutes. Everything looked fine in the smart reports.

 

I have started the parity swap & rebuild of the bad drive. I'll let everybody know how long it takes.

 

I should have a replacement drive for the bad drive in a few days. I opted for the advance replacement from WD.. It only had 15,000 power on hours on that drive.. 

Now I'm getting a little nervous. The new drive is starting to make clicking sounds. Parity is still being copied to it so I'm afraid to stop it. Guess I'll let it finish & let it parity swap. Hopefully the new drive will last long enough to get a successful rebuild out of it.. 

I think the new drive died. I'm going to need some advice on this one.

 

My old parity drive is now replacing disk 10.

Parity was copied to new 2 TB drive.

This drive lost communication.

 

Is there anyway to start the process over again with a new parity drive. I can take the new drive back & get it replaced today since I bought it locally. Any help would be appreciated. I have attached a screenshot below showing Top & the unRAID admin page. This is when the server is having problems. I can telnet in at this time.

 

Thanks

 

 

Here's the screenshot..

 

This is where the server froze or the drive stopped responding..

Well it finally looks like I got bit in the butt.

 

When the new parity drive locked up it locked up the whole system. The old parity drive got corrupted so drive 10 is now gone.. Drive 15 which was a 1.5TB got corrupted. It's gone. I could try & fix it but not worth it.. It was only Blu-ray rips which I own.. I'll just rerip.

 

So I lost 2 TB of data.

 

I removed the old parity drive & drive 15, hit restore & started from scratch again. I'll go get another 2TB Seagate & start over. Run the pre-clear script on it & then generate parity & then run a parity check to verify. If all is well & good then I'll reinstall the other 2 drives & run the pre-clear script on them & re-add them to the array. At least that way I will know all corruption is gone..

 

Don't know if I'll try a Parity Swap again. If a drive fails during it sure can mess things up.. 

Well it finally looks like I got bit in the butt.

 

When the new parity drive locked up it locked up the whole system. The old parity drive got corrupted so drive 10 is now gone.. Drive 15 which was a 1.5TB got corrupted. It's gone. I could try & fix it but not worth it.. It was only Blu-ray rips which I own.. I'll just rerip.

 

So I lost 2 TB of data.

 

I removed the old parity drive & drive 15, hit restore & started from scratch again. I'll go get another 2TB Seagate & start over. Run the pre-clear script on it & then generate parity & then run a parity check to verify. If all is well & good then I'll reinstall the other 2 drives & run the pre-clear script on them & re-add them to the array. At least that way I will know all corruption is gone..

 

Don't know if I'll try a Parity Swap again. If a drive fails during it sure can mess things up.. 

 

That is the one reason I do not do the parity swap method.  I always take the old parity out, rebuild parity with the new drive, then run the verify to make sure all is good, run the old parity through preclear, then add it back to the array.  That way if something was to happe during a parity build the old parity drive is still around to save me, as much as it possible can.

 

I always run the prelcear for three rounds also, I know this can take a very long time for the newer large capacity drives but running the new drive through 3 clears should be enough to work the drive completely and thoroughly.

Well it finally looks like I got bit in the butt.

 

When the new parity drive locked up it locked up the whole system. The old parity drive got corrupted so drive 10 is now gone..

 

Don't know if I'll try a Parity Swap again. If a drive fails during it sure can mess things up.. 

You would be in the same shape regardless of which second drive fails when you have a degraded array.     

 

If you had not done anything yet... and if the old parity drive had not had much written to it before the failure,  you might have been able to

put the old parity drive back in the parity slot,

put a new drive in the failed drive slot

force unRAId to rebuild the failed drive....  Parts of it might not be correct because of the overwritten portion before the corruption occurred, but other parts might be still there.

Use resrfsck to try to rebuild the superblock and rebuild the file-tree

 

Best case... you get back some files, worst case, you do not.

 

Joe L

Well it finally looks like I got bit in the butt.

 

When the new parity drive locked up it locked up the whole system. The old parity drive got corrupted so drive 10 is now gone.. Drive 15 which was a 1.5TB got corrupted. It's gone. I could try & fix it but not worth it.. It was only Blu-ray rips which I own.. I'll just rerip.

 

So I lost 2 TB of data.

 

I removed the old parity drive & drive 15, hit restore & started from scratch again. I'll go get another 2TB Seagate & start over. Run the pre-clear script on it & then generate parity & then run a parity check to verify. If all is well & good then I'll reinstall the other 2 drives & run the pre-clear script on them & re-add them to the array. At least that way I will know all corruption is gone..

 

Don't know if I'll try a Parity Swap again. If a drive fails during it sure can mess things up.. 

 

That is the one reason I do not do the parity swap method.  I always take the old parity out, rebuild parity with the new drive, then run the verify to make sure all is good, run the old parity through preclear, then add it back to the array.  That way if something was to happen during a parity build the old parity drive is still around to save me, as much as it possible can.

 

I always run the prelcear for three rounds also, I know this can take a very long time for the newer large capacity drives but running the new drive through 3 clears should be enough to work the drive completely and thoroughly.

Some times there is no choice but to do a parity swap... (The new drive is the only one available, and it is larger than the existing parity disk)

 

And... there is a huge trade-off.  The time to preclear a drive, when the array is degraded vs. the need to get it back being protected.    zero additional hours (if you already have a drive you can use) vs. 21 hours (as in this case) vs. 63 hours (as you prefer)

 

Best practice is to have a working drive, installed in the array, as big as parity.  (might be a good argument for a parity-sized cache drive)

 

Hate that drives tend to fail so early in their lives.  Be consoled that in a normal RAID array you would would have lost all your files, not just those on the one drive.

 

Joe L.

Some times there is no choice but to do a parity swap... (The new drive is the only one available, and it is larger than the existing parity disk)

 

And... there is a huge trade-off.   The time to preclear a drive, when the array is degraded vs. the need to get it back being protected.    zero additional hours (if you already have a drive you can use) vs. 21 hours (as in this case) vs. 63 hours (as you prefer)

 

Best practice is to have a working drive, installed in the array, as big as parity.  (might be a good argument for a parity-sized cache drive)

 

Hate that drives tend to fail so early in their lives.  Be consoled that in a normal RAID array you would would have lost all your files, not just those on the one drive.

 

Joe L.

 

Yup, I understand that and it is the reason I have kept one of the drives in my array completely empty.  It is a 1.5TB drive that I test periodically (plus parity checks are done once a month to it) with smart and the like to make sure that it is still in good shape.  If something should happen I can swap it for the parity drive and be ok.  When funds permit (here in the next month or so I will be buying another 1.5TB drive so that I can use it as a "hot spare" in case of a drive failure.

If the second drive hadn't of gone belly up I would have put the old parity drive back in place & asked for help.

 

I was telling the wife in line at Best Buy that I was thinking about putting the 2TB back & picking up another 1.5TB so I wouldn't have to do the parity swap.. It would have rebuilt the damaged drive & everything would have been fine. No need to touch the parity drive. I could have bought a bigger drive later & then just regenerated parity. Would have been much safer.

 

Oh well.. Live & learn.. Time to start re-ripping movies.. By the way I'm exchanging the Seagate 2TB for a WD 2TB.. Kind of scared of the Seagates now... lol

Went to Best Buy to return that Seagate 2Tb drive. I paid 219.00 for the drive Friday night. Today 2 days later the same drive is on sale for 189.00. The WD Green 2TB drive is 249.00.

 

I told the Geek Squad guy at customer service that I didn't want the drive exchanged for another Seagate. I didn't care if they were on sale for 30 dollars cheaper. He laughed at me for that & said he would make me a deal & let me have the WD drive for the same price as the Seagate.. I didn't complain.. 

 

It's pre-clearing now. We'll see how this one turns out. I'll let everybody know.

Went to Best Buy to return that Seagate 2Tb drive. I paid 219.00 for the drive Friday night. Today 2 days later the same drive is on sale for 189.00. The WD Green 2TB drive is 249.00.

 

I told the Geek Squad guy at customer service that I didn't want the drive exchanged for another Seagate. I didn't care if they were on sale for 30 dollars cheaper. He laughed at me for that & said he would make me a deal & let me have the WD drive for the same price as the Seagate.. I didn't complain.. 

 

It's pre-clearing now. We'll see how this one turns out. I'll let everybody know.

A $60 discount on the WD drive is pretty decent of them...  I pray this drive works longer...

 

Joe L.

Went to Best Buy to return that Seagate 2Tb drive. I paid 219.00 for the drive Friday night. Today 2 days later the same drive is on sale for 189.00. The WD Green 2TB drive is 249.00.

 

I told the Geek Squad guy at customer service that I didn't want the drive exchanged for another Seagate. I didn't care if they were on sale for 30 dollars cheaper. He laughed at me for that & said he would make me a deal & let me have the WD drive for the same price as the Seagate.. I didn't complain.. 

 

It's pre-clearing now. We'll see how this one turns out. I'll let everybody know.

A $60 discount on the WD drive is pretty decent of them...  I pray this drive works longer...

 

Joe L.

 

I did say that wrong. He let me have the WD for the same price as the Seagate for what I paid for it Friday night. We did a even exchange. It would be nice if it would have been for the sale price, but it was still 30.00 dollars cheaper. 

I took a look at both syslogs, and I'm not sure I agree with part of the analysis.  I think there is a very good chance that the original WD 500GB drive is fine, and that fact changes a lot of your planning here.  If it is fine, then you don't need to rebuild anything, except for parity on whichever drive you decide.  And your new parity drive (sdk, Seagate 2TB) appeared completely fine in the second syslog, no issues with it at all.  The problems were just with 2 of your Seagate 1.5TB drives (sdl and sdm), and they were actually somewhat minor, relatively speaking.  They looked like the normal little timeout frozen errors that that drive model is known for, no real errors reported, just the occasional timeout when it takes too long to respond.  For unknown reasons that I could see (no actual errors), the system quickly slowed both drives down to UDMA/33.  At various times they were continuously reporting timeout/frozen errors, but the second syslog ends without anything otherwise wrong.  I don't know how you proceeded afterward, but it looked to me that if you gave it long enough, it would finish the rebuild successfully.  The parity had been copied successfully from sdm to sdk, and the rebuild of Disk 10 had begun.  Obviously, the performance is terrible, with 2 drives at quarter speed.

 

My suggestion would be to obtain and post the SMART report for that WD 500GB, before anything else, since its status governs your next move.

 

The reason I think it may be fine, is that it had been running a couple of months without any errors, then suddenly had major communications errors, particularly related to the SATA link.  The SATA link was recovered, then lost again, and very shortly after, the drive was disabled, and I don't trust any messages afterward for drives that the kernel has disabled.  The drive was subsequently recovered again, only this time it was assigned to sdo, previously had been sde which was what unRAID knew it as.  And then it was lost and recovered multiple times afterward, plus there was a strange crash related to it.  That crash makes me wonder if something had gone wrong with the system itself, or the driver, or the card itself, or just that specific SATA port.  The crash indicates a probable instability in the system somewhere, so the drive may have been just the first casualty of that instability.

 

One small possibility is a memory glitch, so I would recommend a long memory test, just in case, over night.  Your memory is probably fine, but that is one possibility that is easy to eliminate, by a long memory test with no errors.

 

I have to agree with others here, about the Parity Swap method. It is a good tool to have available, and convenient since it does 2 things at once, but the drive operations involved are so important, that I would always prefer to do them one at a time, testing carefully after each step.  The risks are more than I find acceptable.

By the way, I just noticed that you said that Disk 15 was corrupted and gone.  Are you sure?  Disk 15 is sdl, one of the 2 Seagate 1.5TB drives that was spewing all of the timeouts, and very very slow.  But there did not appear to be anything seriously wrong, as in lost or corrupted data.  I suspect that if you power off and back on, then check the contents of that drive, that you will find them still there.

Disk 15 showed the directories but they were empty. I telneted in & cd to the directories & used the command ls -al. It showed only a couple of files with a ? for the last letter & the file size as zero (0). I also tried MC & it showed the same info. I have only disconnected that drive & have not done anything with it.. For that matter I haven't touched the old parity drive either. I just disconnected it. I did check this drive after several attempts at powering down the server.

 

The 500gig drive I tried to put it into my windows box to get smart reports & it would lock up Win XP SP3. It would state found new hardware & then just sit there. I hit the start button & nothing happened. I let it sit for about 30 minutes & the machine was still locked up. Had to turn the power off. I tried to put that drive back into the unRAID box & during boot up when unRAID looked at that drive everything stopped. After a few minutes (timed out I guess) then unRAID continued. I contacted WD through their web site & started the RMA process. I chose the advance replacement so I should have a replacement in a few days. I would love to get the info off of this drive but I can't get Win XP to play with it. Same thing from the 1.5TB drive. Between the two I lost 2 TB of movies..

 

Since I own all of the Blu-rays that were on it I figured it would just be easier to re-rip everything than take the chance of the files being corrupt after recovery. If that drive had the home movies of my 4 year old then that would have been a different story.

 

I definitely appreciate all of the replies & the help. If anyone has any more ideas I'm all ears... 

 

P.S. The Seagate 2TB drive made me nervous. By the time it was through copying parity to it, it was clicking up a storm. Not a quiet click, but LOUD clicking. Being only a couple of days old I said no way... Best thing to do was take that monster back... lol

I took a look at both syslogs, and I'm not sure I agree with part of the analysis.  I think there is a very good chance that the original WD 500GB drive is fine, and that fact changes a lot of your planning here.  If it is fine, then you don't need to rebuild anything, except for parity on whichever drive you decide.  And your new parity drive (sdk, Seagate 2TB) appeared completely fine in the second syslog, no issues with it at all.  The problems were just with 2 of your Seagate 1.5TB drives (sdl and sdm), and they were actually somewhat minor, relatively speaking.  They looked like the normal little timeout frozen errors that that drive model is known for, no real errors reported, just the occasional timeout when it takes too long to respond.  For unknown reasons that I could see (no actual errors), the system quickly slowed both drives down to UDMA/33.  At various times they were continuously reporting timeout/frozen errors, but the second syslog ends without anything otherwise wrong.  I don't know how you proceeded afterward, but it looked to me that if you gave it long enough, it would finish the rebuild successfully.  The parity had been copied successfully from sdm to sdk, and the rebuild of Disk 10 had begun.  Obviously, the performance is terrible, with 2 drives at quarter speed.

 

My suggestion would be to obtain and post the SMART report for that WD 500GB, before anything else, since its status governs your next move.

 

The reason I think it may be fine, is that it had been running a couple of months without any errors, then suddenly had major communications errors, particularly related to the SATA link.  The SATA link was recovered, then lost again, and very shortly after, the drive was disabled, and I don't trust any messages afterward for drives that the kernel has disabled.  The drive was subsequently recovered again, only this time it was assigned to sdo, previously had been sde which was what unRAID knew it as.  And then it was lost and recovered multiple times afterward, plus there was a strange crash related to it.  That crash makes me wonder if something had gone wrong with the system itself, or the driver, or the card itself, or just that specific SATA port.  The crash indicates a probable instability in the system somewhere, so the drive may have been just the first casualty of that instability.

 

One small possibility is a memory glitch, so I would recommend a long memory test, just in case, over night.  Your memory is probably fine, but that is one possibility that is easy to eliminate, by a long memory test with no errors.

 

I have to agree with others here, about the Parity Swap method. It is a good tool to have available, and convenient since it does 2 things at once, but the drive operations involved are so important, that I would always prefer to do them one at a time, testing carefully after each step.  The risks are more than I find acceptable.

 

I tried to answer your questions in the post above..

 

The answer to the question on how I procedded afterwards was simple. I got MAD...  OK just a little mad but had to say it..

 

I unplugged both of the Seagate 1.5TB drives. The old parity drive & drive 15.. Hit restore & started the array.

 

I knew what that would do & it did what I wanted. Get rid of those 2 drives & start fresh when my new WD 2 TB drive gets through pre-clearing. If it passes ok then I'll put it in as the new parity drive. I'll generate parity & then run a parity check to make sure it can be read from with no errors. Then I was going to run both 1.5TB Seagates thru the pre-clear script. If they passed ok then I was going to reasign them to the array & re-rip the Blu-rays that were lost back to the drives. I figured that way there shouldn't be any chance of corruption. At least on the drives that had been freshly cleared.

 

Phil

If anyone is interested when I pre-cleared the Seagate 2TB it started the copying process (step 2 of 10) at 95 MBs. The WD 2TB started the same process at 102 MBs. Not a whole lot faster but a little faster.. At least it's quiet so far with no clicking...  Just hope I didn't just jinx myself..

The WD 2TB finally finished clearing. It finished a little slower than the Seagate. It took 21 hours & 22 minutes. A little over 30 minutes longer. When it finished & it compared smart reports the only thing that came up was the error reports.. Example seek error.. Everything came up with zero's.. The Seagate had several hundred seek errors. I should have noticed that & never had tried the parity swap with it. Hope someone learns from my mistake..

 

Time to generate parity.. Be back after..

Do you currently have the array protected  by a parity drive?

 

If so, and if you can keep from writing to the array while you build the new parity drive, it will be available if anything else unexpected happens...

 

I'll keep my fingers crossed everything goes smoothly.

 

Joe L.

Do you currently have the array protected  by a parity drive?

 

If so, and if you can keep from writing to the array while you build the new parity drive, it will be available if anything else unexpected happens...

 

I'll keep my fingers crossed everything goes smoothly.

 

Joe L.

 

Not yet.. I wasn't able to generate parity. It was crawling along.. About 1,343k..

 

Turns out one of my other 500gig drives developed a problem cable. I believe this is what caused everything to slow to a crawl during the parity swap procedure. I swapped out the cable & parity started to generate at about 11,000k. That speed is normal on my machine until the ide drives get passed.

 

Talk about dumb luck.. RobJ noticed the slowdown & I think I found the culprit.

 

I just looked & parity sync is going at 11,155k. Normal speed now.

 

It was my fault for loosing so much data.. When the cable went bad I just pulled the plug... Everything seems to be back to normal..

 

I'll pre-clear the remaining 2 1.5tb Seagates & re add them later..

 

Phil