Hey everyone. I've been running my 30 drive server for years without any issues. Today, I decided to swap an 8TB drive for a new 18TB drive. New drive was previously successfully pre-cleared. So, I went ahead and swapped it out and started a data rebuild. I've upgraded over a dozen drives in the last year the same way, without ever having an issue.

 

Today my luck ran out  and Disk1 (1.5 year old Seagate 14TB with XFS) threw 2048 read errors and went offline (red X) about 30 mins into the rebuild. It is now showing up in the Unassigned devices area. I checked the file system and it shows corruption. Short SMART was perfect. Long SMART is not able to be run currently (greyed out). All attributes look normal.

 

 

 

 

SO, just because it has been a VERY LONG time since I've dealt with this, and because I have seen a lot of people lose data and make the situation worse by panicking and making mistakes, I just wanted to run my plan of action by everyone. Open to any advice, and any help is appreciated to keep me on track for a successful outcome.

 

1. The data rebuild is continuing and has about 50+ hours left. I will not stop it as disk1 is emulated and the data is accessible. Since i have dual parity, I am only going to be in a world of hurt if a 3rd disk craps out. Fingers crossed... 

 

2. Once the data rebuild is successful for the new disk I just swapped, I can either try to repair disk 1, or just swap it for another new 18TB disk. I have plenty of spares on standby.

 

Questions on #2. If I did want to attempt a XFS file system repair, and if that was successful, how do I get it back into the array? I was planning on a full shutdown, re-seating disk1, then attempting the repair via the "RUN WITH CORRECT FLAG" button. I've never ran this before, so have no idea what to expect. If it is successful, will the disk just pop back into the array like normal, or after a reboot?

 

I could also just throw this disk into my backup server, run a new pre-clear, extended test, and see if it is stable. If it is, then at least it is serving some purpose in expanding my backup server's space, and I don't have to worry about it blowing up again in my production server.

 

For now, I am not going to touch anything until the data rebuild is finished. I'll refrain from posting diags, and just looking for overall advice.

 

Thanks and Happy Thanksgiving!

 

 

Edited November 27, 20241 yr by falconexe

Welp, I just read something that I was not aware of, but it makes total sense:

 

 

 

So, should I just stop the data rebuild if the rebuild is going to be corrupted anyway? Or am I fine BECAUSE I HAVE DUAL PARITY? I could throw back in my original 8TB disk and try to get disk1 resolved? I am now REALLY thinking that I should not do anything until someone with experience chimes in. If the data rebuild is going to be corrupted on my new 18TB disk because disk1 had 2048 read errors, what is the best protocol to NOT LOSE DATA at this point? I always thought if you had dual parity, even if a disk failed during a rebuild, everything would remain intact so long as you didn't lose a 3rd.

 

Currently, everything is still accessible. I have the original disk I am expanding from, and I have spares to replace disk1 if needed.

 

Thanks again!

 

 

Edited November 28, 20241 yr by falconexe

Please post the diagnostics.