I recently had a flood in my basement and the stand that I had my server on was knocked over due to the immense amount of water we had in our basement.  From what I can tell the server was never fully submerged in water.  I was using a HP Proliant N40L for my server and the motherboard did get wet, but I am not sure if the HDD ever had water on them.  After removing the HDD and letting them sit outside the server for a day I hooked each drive up to my windows 8 computer and used FTK Imager to pull all the data off the drives.  I had no issue doing this and each drive performed without any issues.  I never heard any ticking sounds that would leave me to believe the header or platters had been damaged.  Before I rebuild my server and buy new drives, is there any way to stress test these drives to see if they still are good?  I have five 3tb drives so if I could avoid the cost of replacing the drives I would really like to save the money.  With that being said, I do not want to put in drives that are going to fail on me causing me to loose data.  Any advice you could offer would be appreciated.  Thanks.

The badblocks program can read the drives from start to finish and do so with multiple passes.

capture a SMART log before hand.

Sometimes I like to run a SMART Long test at this point, then examine the SMART log.

run badblocks in readonly mode on the drive.

capture a SMART log after.

diff the 2 logs.

Check the syslog for read errors that may have been successfully retried, but never passed on to badblocks.

 

 

I suppose at the very least, I would invest in a new parity drive once all the drives have passed the test.

If you already have all the data backed up, then running preclear would provide a pretty good stress test.

 

I suppose at the very least, I would invest in a new parity drive once all the drives have passed the test.

 

Can you explain why you feel it would be important to replace the parity drive?

I would at least replace the parity drive because if drive health is in question, it's the easiest drive to replace at the current moment without loss.

 

If you were to preclear a new parity, then re-create/generate new parity from the current drives it would prove all drive sectors on all drives are readable.  Then if any of the drives happen to fail in the near future, you would have confidence that at least the parity drive would survive and could rebuild the failed drive.

 

I would probably back up the super.dat file on the flash key.

If unraid failed to recreate new parity because of a drive failing while creating the new parity, it may be possible to recover the failed drive by restoring the super.dat file and old parity try to rebuild the failed drive on the newest drive.  and so on and so forth.

 

Your other choice is to do the smart long tests, the badblocks in readonly mode. Then do a no correcting parity check.

 

That will prove.

1. the firmware can read all sectors (check smart logs)

2. the kernel and application can read all sectors.

3. All drives and all sectors are readable by unRAID.

 

Do this on the parity or any data drive that you feel you could loose first.

 

Then again, if you have all the data already backed up somewhere else, as suggested you can do a preclear and double check the drive, then put data back or do the badblocks test in read/write mode which does 4 passes of read/write tests.

 

1.2.3. above is easiest without overwriting data.

 

I would probably still replace the parity drive because you said the machine was "knocked over" and the motherboard was wet.

Agree the parity drive is a good candidate to replace with a new, pre-cleared (i.e. well tested) drive.

 

But before doing anything, I'd simply check that all the drives are fully readable in all sectors.    The simplest way to do that is just run a parity check.

 

I would at least replace the parity drive because if drive health is in question, it's the easiest drive to replace at the current moment without loss.

 

But replacing any drive involves reading from each of the old drives, and writing to the new one - I don't see that it matters whether the new one is parity or data.

 

If you were to preclear a new parity, then re-create/generate new parity from the current drives it would prove all drive sectors on all drives are readable.  Then if any of the drives happen to fail in the near future, you would have confidence that at least the parity drive would survive and could rebuild the failed drive.

 

... but having one new drive, whether it be data or parity, counts for nothing if a second drive were to fail.  Simple parity protection of n drives relies on the other n-1 drives continuing to work irrespective of the designation parity or data.  The only significant difference between data and parity is that a data drive can be mounted, and read, stand-alone, whereas the parity drive is meaningless without at least n-2 of the other drives.

 

I think that my reasoning leads to the advice to replace the drive with the most important data on it (or, perhaps, the one which is fullest?).

 

I'm not trying to be argumentative here, but merely understand why the parity may be considered to be the most important drive.

 

It's inevitable these drives are going to fail, it's just when.

 

If the parity is the newest and possibly the most reliable drive, then the chances of that one failing are smaller then the drives that were knocked over. It's a peace of mind exercise.

 

It's a given to save the drive with the most important data or fullest drive. From what I saw on the OP that was done already.

According to the OP

letting them sit outside the server for a day I hooked each drive up to my windows 8 computer and used FTK Imager to pull all the data off the drives.

 

So the most important data should be safe.

 

My first option would be to save data I could and put that on a reliable location. Which seems to have been done.

 

Frankly, Swapping the parity drive for a known reliable drive would be what I would do as my second step.

The parity drive is not the most important drive.

It's the easiest one to sacrifice, It's the easiest one to rebuild onto a new drive. If two drives are lost, there is just no way around it. The data is lost.

 

No matter what choice is made, Every drive needs to be read from start to finish and validated for reliability.

 

If new parity were created reliably. I would test the hell out of the old parity drive. Really give it a grueling stress test. If this proved reliable. It would become my spare as I test each of the other drives individually till I had peace of mind.

 

It actually doesn't matter if it's the most important data drive or the parity drive. I would systematically replace drives over time.

This can be done with the most important data drive also.  It does make sense to replace this drive first.

 

By replacing at least one drive, and using the old one as a benchmark for grueling stress tests, you could have some level of confidence over the other drives.

 

I suppose the reason I would replace parity first is because it is involved in every single write operation on the array.

Parity isn't "the most important drive" => but it is definitely the most used drive for writes, since it's involved in all writes.    Of course it's also never used for reads (unless you're running at risk)

 

As for a "most important data" drive -- in general I'd say there's no such thing, assuming you're allowing all user shares to be allocated by UnRAID.  ALL of your data should, of course, be backed up somewhere else -- and in this particular case, that's already been done (as noted by the author).

 

One could argue that there's no need to replace ANY drive if a parity check is successful.    But in addition to the parity check, I'd also run S.M.A.R.T. reports on all the drives to see if any are reporting pending failures.    If those are also good, then the drives are likely all fine.    Nevertheless, I'd at least buy a spare (or two) and have them handy, just in case 

The most important drive you own is the one with the most important and difficult (or impossible) to recreate data. If you're not backing that up offsite, you're doing something wrong.

Of course if you're properly backed up there's no such thing as "... difficult (or impossible) to recreate data ..."  ==>  you simply restore it from your backup 

 

I suppose the reason I would replace parity first is because it is involved in every single write operation on the array.

 

Yes, that was the main reason I could come up with (the number of writes on the parity should be approximately the sum of writes to all the data drives) but, on the other hand, in a fully functional system the parity drive barely sustains any reads (except during parity checks).  A typical unRAID installation will be used as 'write once, read many', so the data drives may still have more accesses.

One could argue that there's no need to replace ANY drive if a parity check is successful.    But in addition to the parity check, I'd also run S.M.A.R.T. reports on all the drives to see if any are reporting pending failures.    If those are also good, then the drives are likely all fine.    Nevertheless, I'd at least buy a spare (or two) and have them handy, just in case

 

In addition to a parity check, I would also.

 

Capture the smart logs.

 

SMART LONG Test.

 

Capture the smart logs.

Compare.

 

badblocks 4 pass readonly test.

 

Capture the smart logs.

Compare.

 

But this is me. I've been working with hard drives since the early 80's and have seen all sorts of issues crop up.

 

There was a reason I used to run monthly badblocks tests on each drive in addition to the monthly parity check.

In my hosting company we used to run SMART long tests monthly on each drive.

We saw all sorts of failures.

 

I suppose the reason I would replace parity first is because it is involved in every single write operation on the array.

 

Yes, that was the main reason I could come up with (the number of writes on the parity should be approximately the sum of writes to all the data drives) but, on the other hand, in a fully functional system the parity drive barely sustains any reads (except during parity checks).  A typical unRAID installation will be used as 'write once, read many', so the data drives may still have more accesses.

 

>> in a fully functional system the parity drive barely sustains any reads (except during parity checks).

The parity drive is always read whenever a write occurs.

 

read data, read parity, xor new data block, write data, write parity. (Not sure if that's the order)

 

Keep in mind even regular filesystem maintenance causes writes to the filesystem.

That's why they are mounted noatime,nodiratime. So that access times are not updated all throughout any form of read.

 

I also think a typical unRAID installation has evolved over the years.

People are using unRAID more heavily.

There's a reason people are adding plugins, cache drives and are virtualizing.

 

My use is pretty heavy. In fact so heavy. I had a cron job to insure my parity stays spinning from 6am till 9pm.

I wanted to say thank you to everyone for all of the information and insight.  I am still figuring out what server option I want to go with (HP Proliant N40L or N54L) and hoping one goes on sale soon.  WeeboTech, per your advice I have ordered a new parity drive just to be on the “safer” side.  Hopefully the rest of the drives are healthy and this will be a less expensive replacement than I initially thought.  Thanks again for everyone’s advice.

I wanted to say thank you to everyone for all of the information and insight.  I am still figuring out what server option I want to go with (HP Proliant N40L or N54L) and hoping one goes on sale soon.  WeeboTech, per your advice I have ordered a new parity drive just to be on the “safer” side.  Hopefully the rest of the drives are healthy and this will be a less expensive replacement than I initially thought.  Thanks again for everyone’s advice.

 

Before you build/rebuild your server, do a preclear on the replacement drive.  This way, should any drive prove unreliable, you already have a replacement drive handy and already tested/precleared.

 

FWIW, I have the N54L. I was able to get 16GB of ram in it and run ESX. However, newegg sometimes has these pretty awesome rebate deals on the older models, so if it's only going to be a fileserver. You may want to consider that.

It will just be a file server.  The N40L worked perfectly for our needs in the past, so whatever goes on sale first I think we will be purchasing.