I had a drive that died, I put it in to start the parity rebuilt but one of my other drives is not wanting to connect. I've tried swapping SATA power and data cables with other known good ones and that didn't resolve it. Unraid sees the drive but wants to emulate it, either way I can't start the rebuild because this one drive isn't reporting correctly. I did a smart test on it last night and that showed it fine. 

 

diagnostics-20230910-1134.zip

Do you have the original disk4?

8 hours ago, trurl said:

Do you have the original disk4?

No, that is the disk that died and I sent back to Seagate. The new one that is precleared is the replacement that I need to parity repair. 

You can't rebuild from single parity when you have 2 disks disabled. Don't understand how another got disabled when one already was. Was disk10 already disabled when you decided to replace disk4?

4 minutes ago, trurl said:

You can't rebuild from single parity when you have 2 disks disabled. Don't understand how another got disabled when one already was. Was disk10 already disabled when you decided to replace disk4?

No, Disk10 was fine when I had to take out disk4. Its still in there, I never disabled it. Disk10 is the same disk it's always been, that is my question, why it won't recognize it. 

It recognizes it, it just considers it disabled.

 

I guess we could New Config it back into the array. New Config will accept all disks assignments so none are disabled.  Then we can re-disable disk4 so it can be rebuilt (after making sure emulated disk4 is mountable).

 

Before following these instructions, wait a few hours to see if @JorgeB has any other ideas.

 

This process isn't documented, but we have used it many times. It is very important to follow the instructions precisely.

1. Go to Tools - New Config, Retain All, Apply. Not entirely sure it will keep the disk4 assignment since it thinks it is wrong. If it doesn't, assign disk4 before continuing, leave all other assignments as they are.
2. In Main - Array Operation, check BOTH Maintenance mode and Parity valid checkboxes, then start the array.
3. Stop the array, unassign disk4, then start the array. This will disable and emulate disk4.

Then post new diagnostics so we can see if emulated disk4 is mountable.

 

 

Thanks! I'll wait for @JorgeB and try that tomorrow unless they have another idea. 

7 hours ago, trurl said:

has any other ideas.

I think that's the best option for the current situation.

11 hours ago, trurl said:

It recognizes it, it just considers it disabled.

 

I guess we could New Config it back into the array. New Config will accept all disks assignments so none are disabled.  Then we can re-disable disk4 so it can be rebuilt (after making sure emulated disk4 is mountable).

 

Before following these instructions, wait a few hours to see if @JorgeB has any other ideas.

 

This process isn't documented, but we have used it many times. It is very important to follow the instructions precisely.

1. Go to Tools - New Config, Retain All, Apply. Not entirely sure it will keep the disk4 assignment since it thinks it is wrong. If it doesn't, assign disk4 before continuing, leave all other assignments as they are.
2. In Main - Array Operation, check BOTH Maintenance mode and Parity valid checkboxes, then start the array.
3. Stop the array, unassign disk4, then start the array. This will disable and emulate disk4.

Then post new diagnostics so we can see if emulated disk4 is mountable.

 

 

I followed these steps, this is what I'm seeing. 

diagnostics-20230911-0848.zip

Check filesystem on the emulated disk4, run it without -n

16 minutes ago, JorgeB said:

Check filesystem on the emulated disk4, run it without -n

Okay. I think I might be in business now. I went into maintenance mode without Disk4 added and corrected the Disk10 to read xfs, stopped the array, added Disk4 back, started again in maintenance mode and its currently reconstructing Disk4!

Did you check if it's mounting now? Rebuild can be in normal mode.

27 minutes ago, JorgeB said:

Did you check if it's mounting now?

You don't mention if emulated disk4 was mountable. If not, you are rebuilding an unmountable filesystem, which you will have to repair after rebuild (assuming it is repairable).

 

Since you are rebuilding in maintenance mode, diagnostics taken now won't tell us whether disk4 is mountable since maintenance mode doesn't attempt to mount anything.

 

You could stop rebuild, start array in normal mode to see if emulated disk4 is mountable, and post diagnostics.

 

Or you could just let rebuild complete and we can deal with the consequences after.

9 minutes ago, trurl said:

You don't mention if emulated disk4 was mountable. If not, you are rebuilding an unmountable filesystem, which you will have to repair after rebuild (assuming it is repairable).

 

Since you are rebuilding in maintenance mode, diagnostics taken now won't tell us whether disk4 is mountable since maintenance mode doesn't attempt to mount anything.

 

You could stop rebuild, start array in normal mode to see if emulated disk4 is mountable, and post diagnostics.

 

Or you could just let rebuild complete and we can deal with the consequences after.

This is what its currently doing in normal mode. I stopped the parity in maintence and started this, its rebuilding but also looked like disk10 and 4 are unmountable.

Are you sure both of these disks should be xfs?

 

You will have to check filesystem on both disks 4 and 10. Be sure to capture the output so you can post it.

I'll post and check Disk10 when the rebuild is done. i don't think I can stop the array now that the rebuild is started, should be about 12 hours for the rebuild. I'll post the diag then, or tomorrow.

 

33 minutes ago, BlakeB said:

I'll post and check Disk10

Yes, that disk also needs a filesystem check, missed it before.

 

Sep 11 05:45:35 COLDHEART kernel: XFS (md10): metadata I/O error in "xfs_read_agf+0x6d/0xa3 [xfs]" at daddr 0x27fffffd9 len 1 error 117
Sep 11 05:45:35 COLDHEART root: mount: /mnt/disk10: mount(2) system call failed: Structure needs cleaning.

 

Also see here to see if it fixes a PCIe error that's constantly spamming the log.

https://forums.unraid.net/topic/118286-nvme-drives-throwing-errors-filling-logs-instantly-how-to-resolve/?do=findComment&comment=1165009

 

14 minutes ago, JorgeB said:

Yes, that disk also needs a filesystem check, missed it before.

 

Sep 11 05:45:35 COLDHEART kernel: XFS (md10): metadata I/O error in "xfs_read_agf+0x6d/0xa3 [xfs]" at daddr 0x27fffffd9 len 1 error 117
Sep 11 05:45:35 COLDHEART root: mount: /mnt/disk10: mount(2) system call failed: Structure needs cleaning.

 

Also see here to see if it fixes a PCIe error that's constantly spamming the log.

https://forums.unraid.net/topic/118286-nvme-drives-throwing-errors-filling-logs-instantly-how-to-resolve/?do=findComment&comment=1165009

 

Ah, so you're seeing what fills up my log all the time. I've wanted to solve that too. 

Just to make sure this is right before applying. The thread has a lot of sub-topics it seems.

Yes, that's correct.

Parity rebuild of Disk4 was successful. Disk10 is still showing unmountable. I ran a file system check on it and it didn't look like there were any issues. 

diagnostics-20230912-0954.zip

2 minutes ago, BlakeB said:

Disk10 is still showing unmountable. I ran a file system check on it and it didn't look like there were any issues. 

Not sure what you mean, since those diagnostics still shows disk10 unmountable.

 

Did you capture the output of check filesystem on disk10 so you can post it? Did you actually do the repair (without -n)?

10 minutes ago, trurl said:

Did you capture the output of check filesystem on disk10 so you can post it? Did you actually do the repair (without -n)?

I just tried without the -n

 

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... sb_ifree 5382, counted 5482 sb_fdblocks 2328188686, counted 2345233769 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 4 - agno = 3 - agno = 7 - agno = 8 - agno = 9 - agno = 6 - agno = 5 - agno = 2 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (2:1818096) is ahead of log (2:1817289). Format log to cycle 5. done

Should be OK now.

Yes, Disk10 looks like its mounted now, but Disk4 is still showing unmountable even after the parity rebuild.

diagnostics-20230912-1120.zip

The log is incomplete due to the PCIe errors spam, reboot and post new diags after array start.