I recently rebuilt/upgraded a 4TB drive to 12TB without issues and am doing the same thing with an 8TB drive to 12TB, but after assigning the new drive and starting the array, the previous drive says "Unmountable: wrong or no file system". Wondering on next steps as the new drive is being re-built and the Array Operation section says Unmountable disk present: Disk 2 and gives me the option to Format. I believe my best option is waiting for the re-build to finish on the new disk, then format the "Unmountable" disk which was just re-built last week but looking for options and what this really means and why it might have happened.

 

Prior to adding the newest disk, the newly built disk had no issues.

tower-diagnostics-20250324-1943.zip

Edited March 25, 20251 yr by BillPlott
more info

First check filesystem on disk1, since there are errors spamming the log, then reboot and post new diags after array start.

Disk 1 is the one being rebuilt from 8TB to 12TB currently - has a couple more days til it's finished before I can reboot. I can wait til then, then post the logs and determine next steps.

18 hours ago, BillPlott said:

I believe my best option is waiting for the re-build to finish on the new disk, then format the "Unmountable" disk which was just re-built last week

NO! You must not format any disk that has data you want to keep.

 

Emulated disk1 does have some corruption that will have to be repaired, but it is mounted and shows 7.1TB data.

 

Disk2 is too corrupt to mount so we will have to try to repair it to make it mountable.

 

Since you rebooted before getting diagnostics can't see what happened before boot.

 

Might actually be better to stop rebuild since you are rebuilding the corrupt filesystem on emulated disk1.

 

12 hours ago, JorgeB said:

First check filesystem on disk1, since there are errors spamming the log, then reboot and post new diags after array start.

 

11 minutes ago, trurl said:

NO! You must not format any disk that has data you want to keep.

 

Emulated disk1 does have some corruption that will have to be repaired, but it is mounted and shows 7.1TB data.

 

Disk2 is too corrupt to mount so we will have to try to repair it to make it mountable.

 

Since you rebooted before getting diagnostics can't see what happened before boot.

 

Might actually be better to stop rebuild since you are rebuilding the corrupt filesystem on emulated disk1.

 

 

If Disk 1 finishes rebuilding, can't Disk 2 (Rebuilt last week) be rebuilt again without issue?
Should I replace the new disk with the old one and let a parity check happen, then try replacing it again?

Rebuild won't fix filesystem corruption. It needs to have filesystem repair. Click the check filesystem link.

 

19 minutes ago, trurl said:

Emulated disk1 does have some corruption that will have to be repaired, but it is mounted and shows 7.1TB data.

 

Disk2 is too corrupt to mount so we will have to try to repair it to make it mountable.

 

19 minutes ago, trurl said:

You must not format any disk that has data you want to keep.

If you format a disk in the array, the only thing parity can rebuild is a formatted disk.

20 minutes ago, trurl said:

Might actually be better to stop rebuild since you are rebuilding the corrupt filesystem on emulated disk1.

 

13 hours ago, JorgeB said:

First check filesystem on disk1, since there are errors spamming the log, then reboot and post new diags after array start.

 

 

So, My steps would be to:
- Stop Current Rebuild

- Perform File System Check

- Upload new Diagnostics

I'm also a bit thrown off that Disk 2 became corrupt when it was rebuilt/replaced last week and I only touched Disk 1 last night to replace it.

 

I do still have the option to replace Disk 1 with the old 8TB drive if that helps my scenario.

Just now, BillPlott said:

Perform File System Check

Post the output from that and the diagnostics.

 

3 minutes ago, BillPlott said:

I do still have the option to replace Disk 1 with the old 8TB drive if that helps my scenario.

You would have to New Config and rebuild parity.

 

Probably better to just keep that disk handy and maybe copy data from it if necessary depending on how filesystem repair goes.

Just now, trurl said:

Post the output from that and the diagnostics.

 

You would have to New Config and rebuild parity.

 

Probably better to just keep that disk handy and maybe copy data from it if necessary depending on how filesystem repair goes.

ok - Cancelling Rebuild and will do a File System Check

 

How long should I expect the FileSystem Check to take?
Currently it's been over 90 minutes.

Status:

Phase 1 - find and verify superblock...
couldn't verify primary superblock - not enough secondary superblocks with matching geometry !!!

attempting to find secondary superblock...
.found candidate secondary superblock...
unable to verify superblock, continuing...
unable to verify superblock, continuing...

Did you use the webUI to check filesystem? Easy to get the command wrong if you try to use the command line.

8 minutes ago, trurl said:

Did you use the webUI to check filesystem? Easy to get the command wrong if you try to use the command line.

Oh I sure did. Button clicking is harder to mess up - KISS :)

Finished - File System Corruption Detected. Going to initiate the Fix now.

18 hours ago, trurl said:

Post the output from that and the diagnostics.

 

Out of curiosity, I ran a FileSystem Check on Disk 1 and received the following output - is this expected since I stopped the data rebuild on this disk? or is this a different issue? Should I run a Fix on this as well?

Check Filesystem Status-Disk1.txt

Edited March 26, 20251 yr by BillPlott
additional question

9 minutes ago, BillPlott said:

Should I run a Fix on this as well?

Yes, that's the one I asked to fix first.

Just now, JorgeB said:

Yes, that's the one I asked to fix first.

ok - fixing that one as well now

Just now, BillPlott said:

ok - fixing that one as well now

Received this after hitting Fix:
Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.
 

Hit Zero Log and received the thousands of rows of output again. Can't scroll to the bottom so I'll wait for it to finish.

Just wait for it to finish, no need to post the output, then start the array and post new diags.

I believe it finished this morning, but still said it was corrupted for Disk 2 so I hit Fix again. Is this expected and will this actually get fixed? what's the "fix" if this is unable to repair the filesystem?

Just now, BillPlott said:

but still said it was corrupted for Disk 2

You only fixed disk1 right?

 

21 hours ago, JorgeB said:

then start the array and post new diags.

 

1 minute ago, JorgeB said:

You only fixed disk1 right?

 

 

Disk 1 was fixed yesterday without issue. I was concerned about Disk 2 as that one was rebuilt last week and was working fine until I shut down a couple days ago to replace Disk 1. If you are saying to cancel the repair on Disk 2 and try starting the Array, I can do that.

If you already started, let it finish, but I would have preferred to see diags before that, post new ones when done, after array start.

Disk 2 finished trying to fix the filesystem but failed again.

Logs:

2308 - before reboot

2313 - after reboot

2315 - after array start

 

I received the message about a unclean shutdown after the reboot which is what I received when this original issue with Disk 2 started.

I have paused the data rebuild for now until I know what to do next.

tower-diagnostics-20250327-2313.zip tower-diagnostics-20250327-2308.zip tower-diagnostics-20250327-2315.zip

Start the array in maintenance mode and post the output from:

 

xfs_repair -v /dev/md2p1