The array currently has 10x data drives, 2x parity drives, and 5x unassigned drives that have been precleared. All drives are 3TB. 

 

I have several WD Green drives in my array that are from the same batch. One of them failed recently. I believe it was an electronic failure, not a mechanical (wear) one. However, as a precaution, I'd like to replace the other 3 drives from the same batch with new(er) drives I picked up.

 

I want to reiterate that I have confidence that this one drive had problems that the others don't have. It's been flaky with interface errors for nearly a year before it finally failed. None of the other drives exhibit this problem, so I don't expect any more failures from them before they are swapped out. They are even going to still be in use, but getting moved to another server for bulk VM storage with mirrored redundancy.

 

I've already swapped out the 1 bad drive and rebuilt it with a replacement.

 

What is the recommended process to replace the 3 older data drives and add the other 2 new ones with minimal parity check/rebuild stages?

Edited July 14, 20178 yr by DarkKnight

Adding the precleared drives can be done at any time and won't need a sync, for the 3 replacements and since you have dual parity, you can replace 2 at the same time and the third one after, 2 rebuilds total (doing one rebuild at a time is safer in case another disk fails during the rebuild).

After a successful preclear, I typically mount the new drives outside the array, copy the data over, and then do a single parity build, including the new and excluding the old.

 

You need to partition the new drives to unRaid spec, but then can format and mount them with unassigned devices, and then copy the data over.

 

This has a couple advantages:

 

- It is fast, because the copies are not involving parity. You can even do them in parallel. 

 

- You can reorganize or even combine the contents of two or more drives onto one larger replacement drive. (When I am doing a mass replacement, I often am replacing smaller drives with larger ones and reducing the drive count.)

 

- A lesser concern, but by copying the data, you are also eliminating disk fragmentation.

 

I always make sure to do a parity check soon before building parity (after installing any new drives), and carefully examining the smart reports. The biggest risk is in adding the new drives, as you can knock a cable loose. If you have drive cages, this is well mitigated.

 

There are some techniques to provide recovery from a disk failure (even after threse precautions) during the parity build. I can share those of you want to proceed in this manner. 

 

As @johnnie.black says, replacing them one at a time the safest, but once the risk gets below a very low threshold, I am less concerned.

 

But if you are going to rebuild parity, this is a way to only do that once, and copy the data over as fast as possible.

 

One other comment. Although I understand and appreciate the desire to avoid data loss, preemptively replacing perfectly good disks without a sign of problems might be a bit over the top. Drives can and do fail, but normally you get some warning. Just because one of the same vintage failed is not a reason to panic. And you have dual parity. You can even argue that a drive that is burned in had lesser risk of failure than a new drive (read about the bathtub curve). Now if they are old and small, and one has already failed, and you'd like to move the others into backup duty and put some fresh, possibly larger, drives in your primary array, that makes perfect sense. (Just my $0.02)

Thanks for the advice.

 

Keep in mind they while they are functioning well now, I'm replacing ~5 year old drives with a mix ~2 year old & new drives, one of the ones getting replaced is about 15% slower than any other drive in the array as well. Based on my personal experience, the drives I've historically used for my storage (in different cases and locations over time, but still) last around 5 years before they start having unrecoverable issues. I had 2 out of 5 Hitachi 1TB drives that I bought in 2011 for my first RAID array develop bad sectors in the same area of the disk, and a third that failed the same way my current 3 TB drive has all within 2016. I pulled all 5 out of the array and replaced them with 3TB drives and haven't had any disk issues for almost a year.

 

BTW, that's not to say Hitachi drives aren't reliable. The Rosewill storage case I had those in for a long time had a fan failure that went unnoticed, and since there was no temperature monitoring the whole case cooked when it was syncing parity (Flexraid) until half the backplane failed and that's when I noticed there was a problem. Back then it wasn't run 24/7 and it had a really crappy design for keeping drives cool, but it was a lot less expensive than the Norco 24 bay that replaced it, as well as the power costs of running that.   

 

Edited July 14, 20178 yr by DarkKnight

OK, so I replaced the one slowest drive, one bad drive and added 4 new drives in two steps, rebuilding parity twice. At some point well after that process was completed and everything was okay, another parity check just spontaneously started. I had assumed I must have triggered it with an old browser session because it was only a few minutes in, so I cancelled it. Later on, another parity check started *again*. I let this one complete, and it looks like it found just over 1k errors. 

 

What's is going on? Why did the check run twice on it's own (normally scheduled monthly, on the 1st) and why would I have any errors with all good disks? The last 14 parity runs all completed without any problems, in addition to 2 disk replacements.

Grab and post the diagnostics, though if there were reboots between the steps you did they may not show much.

Sounds like maybe you are losing power and the BIOS is configured to restart on power on. Then you get a parity check for an unclean shutdown.

 

What is the exact model of your power supply?

 

24 minutes ago, trurl said:

Sounds like maybe you are losing power and the BIOS is configured to restart on power on. Then you get a parity check for an unclean shutdown.

Or, adding the new drives pushed the PSU over the edge for clean power required and is overheating under stress.

25 minutes ago, trurl said:

What is the exact model of your power supply?

This, exactly. A PSU near capacity, or marginal before you added drives, could definitely cause this.