February 17, 20251 yr Hello! I've been migrating off my Drobo 5C to an Unraid system, and thought I was on the final straight: Drobo is empty, I have dual parity set up, and I just swapped the 8TB that has some SMART errors for a 14TB that doesn't, all that's left is to rebuild the data. Reached this point 4 hours ago. Aaaaand then this happens. Disk 4 drops out, though honestly seemed to be doing just fine. Dual parity keeps things alive, except... Wait, that a LOT of errors! Parity 2 still reads as being connected, but if I go to the attributes it shows the error (Smartctl open device /dev/sdg failed), which is the same error as with Disk 4. After checking the cables, I have confirmed that both drives are going into the same PCIe SATA card, so seems like THAT is the one that's borked. What I'm asking for right now: What should I do right now?! I have paused the data-rebuild because it was throwing so many errors. Array is still running, because stopping it would cancel the data-rebuild. Potential things I can think of (ordering a new SATA PCIe card is a given, but in the meantime): If I remove the Cache 2 drive, that would enable two SATA ports on the motherboard that are currently disabled (the M.2 slot and the SATA ports share lanes). Can the array recognize this even though the SATA controller they're going through has changed?
February 17, 20251 yr Author Welp, I ended up stopping the array, powering down, taking out the cache M.2, and plugging the two drives into the mobo. At first it looked like all was fine? All the drives were found and went into their correct slots in the listing, looked as if all I had to do was hit Start Array and it'd be good. But after starting it, it's now showing the drives still in the same states, with the addition that they're listed as unmountable. Despite this it wants to do a data-rebuild on a 14TB drive, looks like Disk 2, even though it's unmountable? Kinda lost right now to be honest.
February 17, 20251 yr Author Screenshot of the situation. I've paused the data-rebuild again. Also stopping the array isn't available because there's a BTRFS operation going. Though I have no idea what exactly it's doing (probably some cleanup from removing that one cache drive)...
February 17, 20251 yr Community Expert Solution Check filesystem on both unmountable disks, run it without -n
February 17, 20251 yr Author 6 minutes ago, JorgeB said: Check filesystem on both unmountable disks, run it without -n Disk 4: Quote Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... writing modified primary superblock sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 resetting superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 resetting superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 resetting superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. Disk 2: Quote Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... writing modified primary superblock Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.
February 17, 20251 yr Community Expert 3 minutes ago, jubuttib said: If you are unable to mount the filesystem, then use the -L option Unraid has already determined the disks can't be mounted, so you have to use -L
February 17, 20251 yr Author 1 minute ago, trurl said: Unraid has already determined the disks can't be mounted, so you have to use -L So run it without -n, but with -L?
February 17, 20251 yr Author Disk 4: Phase 1 - find and verify superblock... sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 resetting superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 resetting superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 resetting superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_icount 0, counted 10624 sb_ifree 0, counted 332 sb_fdblocks 2441087415, counted 332646411 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 5 - agno = 8 - agno = 2 - agno = 4 - agno = 6 - agno = 9 - agno = 3 - agno = 7 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (1:106489) is ahead of log (1:2). Format log to cycle 4. done Disk 2: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_icount 128, counted 320 sb_ifree 122, counted 280 sb_fdblocks 3403949121, counted 3413601993 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 3 - agno = 8 - agno = 10 - agno = 1 - agno = 4 - agno = 6 - agno = 5 - agno = 9 - agno = 12 - agno = 7 - agno = 11 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (1:1616054) is ahead of log (1:2). Format log to cycle 4. done
February 17, 20251 yr Community Expert Restart the array in Normal mode and I would now expect the drives to mount fine.
February 17, 20251 yr Author 1 minute ago, itimpi said: Restart the array in Normal mode and I would now expect the drives to mount fine. Hmm, seems like they did, but Disk 4 is still marked as disabled for some reason. =/
February 17, 20251 yr Community Expert That was to repair the filesystem, once a disk gets disabled it needs to be rebuilt, if the contents look correct for both emulated disks you can rebuild on top, disk2 looks mostly empty, though. Recommend updating to 6.12.15 or 7.0.0 first, since there's a bug in previous release that can disabled a 3rd disk with dual parity.
February 17, 20251 yr Author Right, so update first, I think 6.12.15 is probably safer than straight to 7... What after that? Is it something like: Set Disk 4 to nothing Start in maintenance mode Stop Set Disk 4 to the drive Start in normal mode Something like that? EDIT: Update to 6.12.15 done. Edited February 17, 20251 yr by jubuttib
February 17, 20251 yr Author OK, did that, data-rebuild going on with the two drives. Thanks for all the help @JorgeB, @trurl & @itimpi! Would love to mark all you guys as solutions, but alas that's not possible. Let's hope the other SATA PCIe card doesn't shit the bed before the rebuild is done... Edited February 17, 20251 yr by jubuttib
February 17, 20251 yr Author Just now, trurl said: Do you have a lost+found share after repair? Not that I can see?
February 17, 20251 yr Community Expert Any errors during the rebuild post new diags before rebooting.
February 17, 20251 yr Author 5 minutes ago, JorgeB said: Any errors during the rebuild post new diags before rebooting. Will do!
February 17, 20251 yr Community Expert 10 minutes ago, jubuttib said: Not that I can see? That is good. You only get a lost+found share created if the recovery process finds any files for which it cannot find the directory entry to give the files their correct name.
February 18, 20251 yr Author On 2/17/2025 at 9:17 PM, JorgeB said: Any errors during the rebuild post new diags before rebooting. Rebuild just finished without any errors, many thanks once more for all the help!
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.