To preface I have dual parity. I was rebuilding a data disk that I had swapped out for a bigger drive (the original disk is fine). During the rebuild process another disk in the array was disabled by Unraid. I canceled the rebuild process, stopped the array and shut the system down but forgot to grab the diagnostics. Also after I stopped the array both parity disks were marked as "missing" by Unraid which I don't understand. After shutting the machine down I checked all the cables for each disk and started the machine again but did not start the array. Once the machine started I was warned about CRC errors on both parity disks with a count of (1) for each. To fix the disabled disk I un-assigned the disk and started the array in maintenance mode then stopped the array again and reassigned the drive. That brings me to now where I'm not sure what to do next. Currently I am running an extended SMART test on the disk that was disabled by Unraid. That said, Unraid is now telling me there are 79071814 parity errors... I don't know what to do next as I do not want to lose my data. Any help is appreciated!

Please post the diagnostics.

Hi @JorgeB thanks for taking a look! Attached are the current diagnostic logs.

 

Edited December 17, 20241 yr by pr0x1ma
removed diagnostics attachment

There's an issue with the syslog server setting spamming the log, fix or disable that and post new diags after a reboot, with the array started.

@JorgeB I’ll post once the SMART test finishes which could be a while. It’s at 40% right now.

@JorgeB Quick question -- will it be an issue if I start the array given the parity errors and my data disk that was disabled? I suspect I will be forced to start the rebuild process when starting the array?

You can cancel the rebuild, we need to see if the emulated disks are mounting.

@JorgeB attached are the updated diagnostics. 

 

 

Edited April 7, 20251 yr by pr0x1ma
removed logs

Unassign disks 3 and 5, start the array in maintenance mode and Check filesystem on both emulated disks, run it without -n

 

@JorgeB Here are the results after un-assigning disk 3 and 5:

Output for Disk 3:

Quote

Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... writing modified primary superblock Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.

Output for Disk 5:

Quote

Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... writing modified primary superblock sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 resetting superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 resetting superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 resetting superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.

 

Use -L

@JorgeB here's the output with -L:

 

Disk 3:
 

Quote

 

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_fdblocks 1083356393, counted 1093576033 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 6 - agno = 8 - agno = 4 - agno = 1 - agno = 5 - agno = 7 - agno = 3 - agno = 9 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (1:1250654) is ahead of log (1:2). Format log to cycle 4. done

 

Disk 5:

Quote

 

Phase 1 - find and verify superblock... sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 resetting superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 resetting superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 resetting superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_icount 0, counted 3328 sb_ifree 0, counted 524 sb_fdblocks 2441087415, counted 42425848 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 4 - agno = 6 - agno = 5 - agno = 3 - agno = 8 - agno = 9 - agno = 7 clearing reflink flag on inodes when possible Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (1:147554) is ahead of log (1:2). Format log to cycle 4. done

 

You should now restart the array in Normal mode and post new diagnostics.     Hopefully the emulated drives are now mounting OK.

@itimpi I restarted the array in normal mode and it's now reconstructing both disk3 and disk5--attached are the diagnostics.

 

Edited April 7, 20251 yr by pr0x1ma
removed logs

Both are mounting, hopefully contents look OK, also look for a lost+found folder.

@JorgeB I don't see a lost+found folder on either disk... should one be there? For now it seems I should let the rebuild happen for both disk3 and disk5?

24 minutes ago, pr0x1ma said:

 I don't see a lost+found folder on either disk... should one be there?

That's a good sign, just let them rebuild.

4 hours ago, pr0x1ma said:

I don't see a lost+found folder on either disk... should one be there?

That is a good sign as it suggest the repair went very well.   You get a lost+found folder created when the repair process finds a file for which it cannot find the directory entry.  The rebuild process should now be making the physical disks match the emulated ones.

@JorgeB @itimpi Hey folks this worked! Thanks so so much!