Hey!

TLDR: don't know how to go about replacing 4 x 2TB drive array (1 drive failed) with 2 x 12TB drives.

I'm in a weird spot and I'm trying to figure out the best way to go about this, please tell me what you'd do or even perhaps what not to do?

So I have a 4 drive array, 1 parity drive and 3 data drives, all are 2TB in size. I have cloud backups, and physical backups at home of all my important data and have tested pulling them back down - if it all goes horribly wrong, I'll be alright.

One of my data drives (disk2) had issues and failed and has subsequently been automatically (not sure how) ejected from the pool. The SMART data says the drive is fine but I don't really trust it. It still shows up as unmounted in unassigned drives plugin. But that's besides the point.

I have ordered 2 12TB drives to replace the drives in my array. Now I need to figure out how to swap them all out. I have I think a few options:

1 - Try remount the failed drive and then swap parity, then data drives. (bad idea)

2 - Swap parity and then data drives and then replace the data from the missing drive (not sure this would work because it might destroy parity?).

3 - Pull all the drives and Rsync all the data back onto the new array (nice idea but slow due to internet bottleneck).

4 - Put one of the new drives into the system and copy across all the data, remove it, replace the drives (will parity be able to sync if the data drive already has data when creating a new array?).

5 - Replace the failed drive with the 2TB USB HDD I have and then replace parity, then rebuild from parity (Can you add a USB drive into the main pool?)

6 - Just buy another 2TB HDD and do it properly (meh)

Anything I may have overlooked? I realise some ideas are more stupid than others but they're all options. I haven't had to do this before so any guidance, tips/tricks would be greatly appreciated!

Post the diagnostics first, but you should be able to use the parity swap procedure.

@JorgeB Some of the issues were caused by a bad SATA PCIe expansion card. I'm trying to resolve all of this by upgrading from what I have now to two large data drives instead four small drives. Additionally I'm replacing my three small cache drives with a 2TB NVMe SSD and a 2TB 2.5" SSD. This will leave me with one spare SATA port (my board only has 4 SATA ports) for swapping the drives around etc.

I'm open to suggestions. Also, rebuilding the array from scratch could potentially mean an opportunity to switch to ZFS, although that would require an upgrade which I haven't done due to the janky drive situation.

tower-diagnostics-20251006-1606.zip

Last lines of the log show issues with a device on the onboard SATA controller; it also looks more like a power/connection issue.

@JorgeB I have replaced SATA cables multiple times, power is off the same string of SATA power connectors as the other drives (I guess could still be an issue). Would it still be safe to try the parity swap? Short of replacing the motherboard and PSU I don't really know what else to do.

I also don't understand how the parity swap would work for multiple data disks and single parity. Would it be a case of moving the parity drive to the data pool and syncing parity to the newer, larger drive? Would that mean the smaller, old parity drive assumes the roll of replacing the defective drive and then the pool rebuilds parity to the new parity drive?

5 hours ago, deano_southafrican said:

power is off the same string of SATA power connectors

How is this string? Single cable? any power splitters?

Parity swap with single parity allows you to replace only one drive initially, but after new parity is installed, you could replace the other ones.

Thank you for your help, I greatly appreciate it.

The drive is attached to the third (of four) SATA power connectors directly attached to the PSU, no splitters or anything. So a bit strange. I've removed the drive an tried to connect it to my PC in one of those SATA HDD USB docks in an attempt to check the drive info in Crystal Disk Info but the drive won't read at all despite spinning up, other drives do work in this way, I've tested 3 other drives today.

So I've moved all my data from my other cache pools onto a single new NVMe drive and removed the PCIE to SATA adapter that had all of my SSD's connected. The HDD's in my array were all connected to the SATA ports directly on the motherboard.

I've removed the failed drive and connected in its place the 14TB drive that will be my new Parity drive. I disabled Docker and turned off VMs before all of this maintenance work. Now I'm running pre-clear on the new drive. The array is offline.

So obviously the parity drive is now emulating the failed disk which is why I could still see and access my data. So I'm trying to figure out if I can use parity swap to replace the failed disk, then rebuild parity to the new 14TB drive. Then I'm guessing I'll have to remove one HDD, replace it with the new 12TB drive, and rebuild from parity, at which point I can move the data from the other two drives to the new 12TB drive and then remove those?

I have backups as mentioned so I'm not overly stressed if it goes badly. But just trying to avoid having to rebuild from backups as I'm bottlenecked by my internet speed.

Really appreciate you taking the time to read this and give me advice.

6 hours ago, deano_southafrican said:

So obviously the parity drive is now emulating the failed disk which is why I could still see and access my data. So I'm trying to figure out if I can use parity swap to replace the failed disk, then rebuild parity to the new 14TB drive. Then I'm guessing I'll have to remove one HDD, replace it with the new 12TB drive, and rebuild from parity, at which point I can move the data from the other two drives to the new 12TB drive and then remove those?

That should do it.

Hey @JorgeB, so the preclear of the new 14TB parity drive took forever, and then finally, after nearly 30 hours it managed to do the parity swap. So followed my plan and I'm letting it rebuild disk 2 (the old parity drive is now disk 2 after the swap). But I'm getting read errors on disk 1 and it's causing the rebuild to go at like KB/s and I can hear the drive struggling.

At this point I just have a feeling everything is going to fail. I've attached the diagnostics.

I can't see anything obvious besides some errors due to swapping my cache drives and not having changed the mapping yet.

I'm wondering, can I not just take out the drive that's being rebuilt (disk 2) and put in my new 12TB data drive now already and attempt to rebuild? Also wondering if there's anything else that could be causing slow-downs or issues? Or perhaps it's messed up and I should be pulling from backups?

Appreciate any advice here...

tower-diagnostics-20251014-2351.zip

RIP, I think Disk1 has failed now after the read errors. So Disk2 is being emulated but obviously now Disk1 data will be lost. I'm wondering now if I should just pull the old drives and put in the new 12TB drive and then pull from backups and resync parity...

Updated diagnostics...

tower-diagnostics-20251015-0024.zip

Looks more like a power/connection issue, replace both cables for disk1 and try again.

tower-diagnostics-20251015-1903.zip

@JorgeB Don't know what happened to my response that accompanied the above diagnostics...

Anyway, disk1 is still unmountable after swapping to a brand new SATA cable and putting it onto the other PSU SATA string which is now the same as disk2 and disk3. However, the rebuild for disk2 is still going ahead... Does that mean that I might be able to rebuild disk 1 afterwards? Or will the data be corrupt? I'm very confused...

Initially it slowed down to 25MB/s for the rebuild and then ramped back up to full speed.

Oct 15 19:01:30 Tower kernel: XFS (md1p1): Corruption warning: Metadata has LSN (88:463746) ahead of current LSN (88:463731). Please unmount and run

xfs_repair (>= v4.3) to resolve.

Once the rebuild finishes, check filesystem on disk1