November 4, 2025Nov 4 Hi,I have a twelve disk array with dual parity (14 disks in total), plus two caches (2 TB and 525 GB). The two parity drives are 8 TB each, disks in the array are a mix of sizes up to that capacity.I had one relatively old 1 TB disk fail a few days ago, ordered an 8 TB replacement which arrived today.I then did the following steps:removed the failed 1 TB disk (disk2) from the serverremoved another failed 4 TB disk from the server (this one failed months ago and had been replaced by a disk in another bay, I just forgot to remove it originally)ran "Force Smart + DB" in the Disk Location plugin, added locations for three new disks I had added/replaced a few months ago but forgot to update in Disk Locationadded the new 8 TB drive into the original slot of the failed 1 TB diskran "Force Smart + DB" in the Disk Location plugin again, as I was promptedWas about to start to pre-clear the new disk, when I noticed a total of three unassigned devices on the dashboard - the new 8TB Seagate disk I was about to pre-clear and two disks that are also part of the array. Those still showed up in the array, but were also shown here, using different device designators (disk9 as sdj and disk 12 as sdn in the array, but as sdk and sdt in unassigned devices).I stopped the array, set disk2 to empty (I planned on replacing disk2 with the new 8 TB drive after pre-clearing and rebuilding the contents there) and noticed that disk12 now also was disabled. I left the failed disk in the disk12 slot for now and restarted the array. The array right now is running but obviously is fully depending on the two parity disks it seems. Maybe this was just some sort of controller snafu that re-initialized the disks?Besides not understanding what happened with the disks getting new device designations I am somewhat nervous about the state of my array right now. It is working for now and all data seems to be there.I plan on preclearing the new disk and replacing the original failed disk with it, then getting another new drive ASAP and replacing the reportedly failed other disk (I believe this is one of the newer ones I got a few months ago, so I should be able to RMA it). Can I easily copy the contents off the emulated disks, or should I leave anything as is? The Unbalanced plugin shows disks 2 and 12 as empty right now, but I guess it bypasses the array layer?Addition info:Case/backplanes: Inter-Tech IPC 4U-4724Controller: Broadcom SAS 9305-24iCables: SilverStone CPS06, Slimline miniSAS HD SFF-8643 to miniSAS SFF-8087View of the array with the three unassigned devices:View of the array now:diagnostics-20251104-1138.zip Edited November 4, 2025Nov 4 by Darkguy
November 4, 2025Nov 4 Community Expert Disks dropped offline; it was right after spin up, so if it happens again to those disks, it may be worth disabling it, or trying a different controller. For now, reboot the server and post new diags after array start.
November 4, 2025Nov 4 Author diagnostics-20251104-1339.zipDiagnostics after reboot Edited November 4, 2025Nov 4 by Darkguy
November 4, 2025Nov 4 Author I put the array in maintenance mode, ran the XFS Check on disk12 with this output:Phase 1 - find and verify superblock...bad primary superblock - bad CRC in superblock !!!attempting to find secondary superblock....found candidate secondary superblock...verified secondary superblock...writing modified primary superblocksb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128resetting superblock root inode pointer to 128sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129resetting superblock realtime bitmap inode pointer to 129sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130resetting superblock realtime summary inode pointer to 130Phase 2 - using internal log- zero log...ERROR: The filesystem has valuable metadata changes in a log which needs tobe replayed. Mount the filesystem to replay the log, and unmount it beforere-running xfs_repair. If the filesystem is a snapshot of a mountedfilesystem, you may need to give mount the nouuid option. If you are unableto mount the filesystem, then use the -L option to destroy the log andattempt a repair. Note that destroying the log may cause corruption --please attempt a mount of the filesystem before doing this.XFS check on disk2 with this output:Phase 1 - find and verify superblock...bad primary superblock - bad CRC in superblock !!!attempting to find secondary superblock....found candidate secondary superblock...verified secondary superblock...writing modified primary superblocksb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 97resetting superblock realtime bitmap inode pointer to 97sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 98resetting superblock realtime summary inode pointer to 98Phase 2 - using internal log- zero log...ERROR: The filesystem has valuable metadata changes in a log which needs tobe replayed. Mount the filesystem to replay the log, and unmount it beforere-running xfs_repair. If the filesystem is a snapshot of a mountedfilesystem, you may need to give mount the nouuid option. If you are unableto mount the filesystem, then use the -L option to destroy the log andattempt a repair. Note that destroying the log may cause corruption --please attempt a mount of the filesystem before doing this.Should I stop the array and start it in regular mode or press the "Zero Log" button next? Edited November 4, 2025Nov 4 by Darkguy
November 4, 2025Nov 4 Author So press the "Zero Log" button and do not attempt to mount the disks first? I always like to understand what is happening - so what exactly happens when pressing the Zero Log button and how does this correspond to "Mount the filesystem to replay the log, and unmount it before re-running xfs_repair"?
November 4, 2025Nov 4 Community Expert Since the disks cannot be mounted, there's no other option other than zeroing the log. This is typically safe, though it will always depend on the amount of filesystem corruption that exists.
November 4, 2025Nov 4 Author Thanks for your help so far.I hit "Zero Log" on both devices, let it finish, then stopped the maintenance-mode-array and restarted it properly. Both disks/emulated disks mount fine now.I can browse the emulated disk2 and see its contents. disk 12 shows up as 153GB used and 7.85TB free; however, when I browse disk 12 (in the Unraid GUI or via mc in the terminal), there it shows up as completely empty. Unfortunately I can not quickly determine if anything is missing - the disk has been in use for about 8 months, so I would guess something should have been on there.Should I stop the array, remove it and try to restore anything on it with Linux tools on another machine?Also, Disk12 still shows up as disabled/content emulated. Can I reactivate it or is it safer to RMA it and swap it out for a new one?
November 4, 2025Nov 4 Community Expert If disk12 had any actual data that xfs_repair could not correctly recover, it typically would move it to a lost+found folder, so most likely it was already emptied, but you should be able to confirm by mounting the actual old disk. For that, stop the array, unassign disk12, and then mount it with the UD plugin and check its contents.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.