I have a 6 data disk + 1 parity disk setup and recently noticed disk 3 (serial number ended in 1RJ8) was disabled and running in emulated drive mode. Candidly, I don't know how long it was running in that mode due to atrophy in me keeping my notification settings configured correctly, a mistake I won't make again. I ran some SMART tests on disk 3 but the test reported no errors found so I brought the array back online until it got marked disabled again a few days later. I then bought two new drives to replace disk 3 and execute a parity swap procedure with larger drives while replacing the drive. Before shutting down the array to execute the changes I decided to do one last backup to my external drive to grab the 4 files I'd put onto the server that morning and to my dismay it reported disk read errors on disk 6 (6GCC). I have not written much to the array recently and have original copies of pretty much everything (certainly all the important files) though it'd be arduous and time consuming to restore it all from backup.

I may have made my first error at this point by deciding to proceed with the replacement process instead of seeking guidance here first given the circumstances. I can't change that now, so I'll continue telling what I did do.

I executed the parity swap using the instructions found at (https://docs.unraid.net/legacy/FAQ/parity-swap-procedure/).

Disk 3 was disabled already so I started from Step 3. I removed disk 3 (1RJ8) and installed a new drive (ZNBL). As part of this, I re-cabled the system and moved some drives around to try and address some of the heat issues I've noticed by putting the larger, hotter drives in the better case airflow paths, so the logical mapping to sdX changed for some of the drives.

When I continued the parity swap procedure from step 8, I moved the previous parity drive (WXWJ) to the disk 3 location and setup ZNBL as the new parity drive since it was now the largest drive in the system. The parity copy went smoothly with no errors reported. 

During Step 15, the data drive rebuild, I received disk read errors from disk 6 (6GCC), and that's where I'm now stopped and debating how to proceed.

I've attached a screenshot of the main screen prior to starting this effort as well as the new setup after what I've done. I have also attached the Diagnostics zip file. 

I still have the original disk 3 sitting on my desk. I also have another 8 TB hard drive in the box, I was going to replace disk 6 with it but had hoped to get through the disk 3 rebuild first. Is there any path through here to use the old disk 3 to rebuild a (possibly old) image of disk 6 and get the system into a coherent state and then just upload the recent files that were affected since things went south? 

If that's not going to work, is there some way to identify which files on the drive(s) are affected by the bad sectors on disk 6, replace disk 6 and then perform a partial restore of only the affected files? I'm not sure if there is any tooling available to construct a file list based on bad sectors or not?

Any other ideas on possible next steps?

My system does no overclocking or other optimizations and is running Unraid 6.12.6. I run plex in a docker but no VMs, and docker has been shut down since I started the process. The array was online for the data rebuild operation but I have not attempted any actual file writes to the array, although I do see some write activity that may be related to one of our PCs connecting to the shares when the PC booted up, which I shut back down asap. 

I hope I have included the pertinent information to get the conversation started. Many thanks for any help you can offer.

jensen-nas-diagnostics-20240130-0616.zip

Are you sure the disks have failed?  Disks being disabled due to factors other than the disk itself failing are very common.    Running an extended SMART test on the drives is probably the best health check.

 

BTW:   You were using obsolete documentation.    The official documentation for parity swap is here in the online documentation accessible via the ‘Manual’ link at the bottom of the GUI or the DOCS link at the top of each forum page. The Unraid OS->Manual section in particular covers most features of the current Unraid release.

Thank you for the quick reply!

 

I will perform some extended tests and see what comes back from that.

I have completed SMART tests (both short and extended) for all drives in the system. They all completed both tests without error except disk 6 (6GCC). I've uploaded the SMART test results here for that disk. It looks like it does indeed need to be replaced, but this is my first rodeo and I'm trying to learn from the experts here. If I'm being shortsighted please enlighten me. 

I've been reading up on SMART tests and I confess I'm confused. The SMART test descriptions I've been reading online generally indicate that if the VALUE is above the WORST and the WORST is above the THRESH then the drive would be considered doing ok still, yet I'm seeing these read errors popping up. Is the drive perhaps good but the cabling is bad? When I moved the drives around as part of the recent parity swap sequence I also moved the cables around a bit to let longer ones reach the farther drives, etc. I didn't keep track of which cable was attached to disk 6 so I have no way of knowing if it was the same cable or not. It is definitely connected to a different SATA connector on the motherboard (sdb) versus the original port (sdg, a port on a peripheral card).
 

My original disk 3 is still on my desk, next I will plug it in and run SMART tests on it to assess whether it was experiencing errors or may have been disabled for some other reason.

 

 

jensen-nas-smart-20240201-2041.zip

5 hours ago, jeffreyjensen said:

It looks like it does indeed need to be replaced

Yep.

 

5 hours ago, jeffreyjensen said:

My original disk 3 is still on my desk, next I will plug it in and run SMART tests on it to assess whether it was experiencing errors or may have been disabled for some other reason.

Post a SMART report after an extended test.

original disk 3 (1RJ8) completed the short and extended test without error. I'm now thinking I was hasty in replacing disk3 before disk6, but since it was disabled (twice) I figured that was the first priority. Regardless I need to plan a path forward from current state, which is:

• New Parity should match the expected system state (parity swap copied without error)
• New Disk 3 rebuild completed but with read errors on disk 6, leading me to assume disk 3 has errors in it now. 
• Disk 6 needs replaced.

I see two possible paths forward - if you know of others I would welcome them. Else, can I get feedback on the wisdom or pitfalls of these?

Option 1:

• Replace disk 6 and rebuild using the standard procedure.
• If disk 3 contains errors, they'll be propagated to disk 6.
• Once disk 6 is rebuilt, do an extensive file compare/repair between the newly rebuilt system and my backups.
• Replace whatever damaged files I find on the server.
• Should be green to go from here.

Option 2:

• Put the original disk 3 back into service and replace disk 6 as a simultaneous action.
• I was thinking this could (in theory) use the (believed good) parity and the current physical state of disk 3 to rebuild disk 6 in a clean state, accepting there may be missing files for anything copied to the array after disk 3 went into emulation mode.
• After a few reads of the emulated disk documentation in the manual, I'm now not sure this would work.
• If I understand correctly, the parity was updated while disk 3 was running in emulation mode and the physical disk state fell behind, so if I bring disk 3 online in the older physical state, the parity would be incorrect for disk 3's older state and a rebuild of disk 6 would then be incorrect. Is that right?
• If that understanding is correct, this option is really not a good option after all and could lead to unknown results depending on how far off the emulated disk 3 went from the physical state.

Sounds like option 1 is really the best option for proceeding, unless someone has a better idea?

 

Important Question:
Since the parity is driven sector-by-sector, this means the disk read errors and possible bitrot that occurred during rebuild would impact filesystems as well. Should I check my disk 3 filesystem BEFORE I do the disk 6 rebuild? Or will that risk making parity changes that could render disk 6 even worse off after it gets rebuilt? 

Thank you so much for your insights.

 

 

 

jensen-nas-smart-20240203-1008-1RJ8.zip

I am using XFS so I started the array in maintenance mode and ran the filesystem check tests. 

I'm not sure how to interpret these results - does this somehow indicate FS corruption or not and indicate a need for repair? It appears identical to my drives that have not had any problems.... 

I'll keep researching this but figured I'd add this to my collection of knowledge in case any of you are online and can short-circuit my research. Thank you!
 

I reran the tests via the command line. 

xfs_repair -nv /dev/md3p1

xfs_repair -nv /dev/md6p1

echo $? after each returned 0, which I believe indicates no corruption was detected. Am I understanding correctly?

 

14 hours ago, jeffreyjensen said:

echo $? after each returned 0, which I believe indicates no corruption was detected. Am I understanding correctly?

That is correct.

JorgeB, thank you for confirming. I will commence implementing Option 1.