Configuration

Version: 7.1.4

Parity	ST4000VN008 - 4 TB (sde)
Disk 1	ST4000VN006 - 4 TB (sdc)
Disk 2	ST2000VN004 - 2 TB (sdf)
Disk 3	ST4000VN006 - 4 TB (sdb) - NEW

Timeline

1. Old disk 3 (3TB) was seeing some read errors. Old drive. I was purchasing a replacement drive

2. Unraid disabled disk 3 all of a sudden and started emulating it

3. New disk 3 (4TB) arrived

4. Stopped the array

5. Removed old disk 3 & replaced it with new disk 3 (4TB)

6. Ran pre-clear on the new disk 3

7. Started the array

8. Unraid successfully rebuilt the new disk 3 (4TB)

9. No data loss. All okay. Array is green. Parity check is green.

10. All of a sudden today the new disk3 went into Unmountable: Unsupported or no file system state

What I have done after

1. Stopped the array and started it

2. Stopped that array and started it in maintenance mode

3. /sbin/xfs_repair -n /dev/md3p1 in progress. It has been running for a couple of hours now.

Any help and guidance to solve this will be appreciated :)

Diagnostics attached.

wednesday-diagnostics-20250807-1528.zip

Edited August 7, 2025Aug 7 by niranjan94

Aug 7 02:51:40 WEDNESDAY kernel: XFS (md3p1): Metadata corruption detected at xfs_buf_ioend+0xa5/0x380, xfs_inode block 0x575580d8 xfs_inode_buf_verify

Aug 7 02:51:40 WEDNESDAY kernel: XFS (md3p1): Unmount and run xfs_repair

Typically you fix the filesystem before rebuilding. What is xfs_repair outputting?

8 minutes ago, JorgeB said:

What is xfs_repair outputting?

This is what the xfs_repair is doing

Phase 1 - find and verify superblock...
couldn't verify primary superblock - not enough secondary superblocks with matching geometry !!!
attempting to find secondary superblock...
..............................................................

10 minutes ago, JorgeB said:

Typically you fix the filesystem before rebuilding

But there were no issues reported before. The array rebuilt successfully with the new disk and was running okay.

The emulated disk would already have been unmontable.

1 hour ago, niranjan94 said:

couldn't verify primary superblock - not enough secondary superblocks with matching geometry !!!
attempting to find secondary superblock...

This is not a good sign, and if no backup superblock is found after one hour, you can cancel.

To rule out any md driver issue, stop the array and post the outout from:

xfs_repair -n /dev/sdb1

3 minutes ago, JorgeB said:

This is not a good sign, and if no backup superblock is found after one hour, you can cancel.

No superblock was found after more than 2 hours. I have cancelled this now

3 minutes ago, JorgeB said:

To rule out any md driver issue, stop the array and post the outout from:

That has a similar output like xfs repair on md3p1

# xfs_repair -n /dev/sdb1
Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!
attempting to find secondary superblock...
warning: device is not properly aligned /dev/sdb1
.............................................................

9 minutes ago, niranjan94 said:

warning: device is not properly aligned /dev/sdb1

This is not normal, post the output from fdisk -l /dev/sdb1

12 minutes ago, JorgeB said:

This is not normal, post the output from fdisk -l /dev/sdb1

The disk size is 4TB. But seems to be showing a 2TB in this

# fdisk -l /dev/sdb1
Disk /dev/sdb1: 2 TiB, 2199023255040 bytes, 4294967295 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Alignment offset: 3584 bytes

Sorry, wrong command, it shoudl be:

fdisk -l /dev/sdb

But yeah, the partition doesn't look correct.

5 minutes ago, JorgeB said:

fdisk -l /dev/sdb

# fdisk -l /dev/sdb
Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: ST4000VN006-3CW1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x00000000
Device     Boot Start        End    Sectors Size Id Type
/dev/sdb1           1 4294967295 4294967295   2T  0 Empty
Partition 1 does not start on physical sector boundary.

That's not an Unraid standard partition layout. Do you have old diags when the old disk was still connected? Or is the old disk still working?

Taking a second look at the diags, the disk was partitioned correctly before the rebuild. Not sure what happened. Try this: reboot first to make sure the partition info is up to date, start the array, if the disk still doesn't mount, stop the array, unassign disk3, start the array, post new diags

And keep the current disk3 intact for now.

• I still have the old disk. I'm able to still read the data from it via an external linux machine.

• I rebooted and the disk still did not mount (same error: Unmountable: wrong or no file system)

• I unassigned disk3 and have started the array now.

• Unraid says device is missing(disabled), contents emulated for disk3. But when checking the shares, I see that data that was stored in disk3 missing (So the emulation isn't working as expected)

• Both current and old disk3 drives are still intact

New diagnostics attached

wednesday-diagnostics-20250807-1941.zip

Try running a filesystem check on the emulated disk, but don't think it's going to work.

If it doesn't, you can try recreating the correct partition on the current/new disk3, with the disk still unassigned, type:

sgdisk -o -a 8 -n 1:32K:0 /dev/sdX

Replace X with the correct letter. Then see if the disk mounts with the UD plugin (don't assign it again to the array for now)

43 minutes ago, JorgeB said:

Try running a filesystem check on the emulated disk, but don't think it's going to work.

If it doesn't, you can try recreating the correct partition on the current/new disk3, with the disk still unassigned, type:

sgdisk -o -a 8 -n 1:32K:0 /dev/sdX

Replace X with the correct letter. Then see if the disk mounts with the UD plugin (don't assign it again to the array for now)

• Filesystem check on the emulated disk resulted in the same error as before. bad primary superblock - bad magic number !!! attempting to find secondary superblock...

• Ran sgdisk -o -a 8 -n 1:32K:0 /dev/sdb and got this error. Should I run with -g?

  Found invalid GPT and valid MBR; converting MBR to GPT format
  in memory.
  ***************************************************************
  Non-GPT disk; not saving changes. Use -g to override.

First run

wipefs -a /dev/sdb

then

sgdisk -o -a 8 -n 1:32K:0 /dev/sdb

Assuming it's still sdb

1 hour ago, JorgeB said:

First run

wipefs -a /dev/sdb

then

sgdisk -o -a 8 -n 1:32K:0 /dev/sdb

Assuming it's still sdb

# wipefs -a /dev/sdb
/dev/sdb: 2 bytes were erased at offset 0x000001fe (dos): 55 aa
/dev/sdb: calling ioctl to re-read partition table: Success

# sgdisk -o -a 8 -n 1:32K:0 /dev/sdb
Creating new GPT entries in memory.
The operation has completed successfully.

Then tried mounting with the UD plugin and the mount failed.

Aug  7 22:43:01 WEDNESDAY unassigned.devices: Mounting partition 'sdb1' at mountpoint '/mnt/disks/WW675B6T'...
Aug  7 22:43:01 WEDNESDAY unassigned.devices: Mount cmd: /sbin/mount -t 'xfs' -o rw,relatime '/dev/sdb1' '/mnt/disks/WW675B6T'
Aug  7 22:43:01 WEDNESDAY kernel: XFS (sdb1): Mounting V5 Filesystem e28b8fc3-0300-49ce-be43-85460e8a8b4c
Aug  7 22:43:01 WEDNESDAY kernel: XFS (sdb1): totally zeroed log
Aug  7 22:43:01 WEDNESDAY kernel: XFS (sdb1): Corruption warning: Metadata has LSN (4:33) ahead of current LSN (1:0). Please unmount and run xfs_repair (>= v4.3) to resolve.
Aug  7 22:43:01 WEDNESDAY kernel: XFS (sdb1): log mount/recovery failed: error -22
Aug  7 22:43:01 WEDNESDAY kernel: XFS (sdb1): log mount failed
Aug  7 22:43:02 WEDNESDAY unassigned.devices: Mount of 'sdb1' failed: 'mount: /mnt/disks/WW675B6T: wrong fs type, bad option, bad superblock on /dev/sdb1, missing codepage or helper program, or other error.        dmesg(1) may have more information after failed mount system call.'
Aug  7 22:43:02 WEDNESDAY unassigned.devices: Partition 'WW675B6T' cannot be mounted.

Now I see the xfs signature; run xfs_repair again.

xfs_repair -v /dev/sdb1'

Unfortunately it failed too

# xfs_repair -v /dev/sdb1
Phase 1 - find and verify superblock...
couldn't verify primary superblock - not enough secondary superblocks with matching geometry !!!
attempting to find secondary superblock...
.........................................................................................................................................................................................................................................
..................................................................................................................................................................................Sorry, could not find valid secondary superblock
Exiting now.

Not excaçlty sure what happened to that disk, something very strange, because it was correctly partitioned during the rebuild, but in the end it reverted to a partition like if the disk was just precleared.

Best bet is to try and recover the data from old disk3, but I would keep the new one intact for now. If not all data can be recovered from the old one, you could be able to recover some data from the new one using a file recovery app, like UFS explorer

Yes. This was a weird behavior. Thankfully, I did not have any critical data on disk3 that I would be sad about loosing. Just some entertainment stuff.

So, I'm gonna go ahead reformat the new disk3, add it to the array and copy over whatever data I can from the old disk three.

So, I pre-cleared the new disk3 again (no errors in the pre-clear report), added it back to the array. Now it says data rebuild is in progress.

But I still see Unmountable: wrong or no file system against disk3 while the rebuild is happening.

I still also see Unmountable disk present: while the rebuild is happening.

I don't remember this being the case during a disk replacement + rebuild. Is this expected ?

wednesday-diagnostics-20250809-1003.zip

You will need to format it before or after rebuilding; preclear was unnecessary.

On 8/9/2025 at 12:29 PM, JorgeB said:

You will need to format it before or after rebuilding; preclear was unnecessary.

Yes. that's what I ended up doing. Also finished copying over all the data successfully from the old disk3. As of now, looks like there was no data loss.

Still a little concerned about what happened since I don't yet know what went wrong where.

But anyway, thanks a ton for all your help @JorgeB in debugging this. 😄