One of my drives failed in the array.   (showed up as unavailable).  I followed these steps to replace the disk:

 

• I stopped the array
• shutdown
• replaced the faulty drive with a new one
• booted and went into the array devices and replaced the missing drive with a new one
• The system rebuilt overnight

 

Now, it's showing that my new drive is unmountable.  It also appears that the array is healthy?  Not sure what's going on here?

 

homer-diagnostics-20211214-0854.zip

Unfortunately, you rebooted after rebuilding, so syslog in diagnostics doesn't show anything about that. Do you have earlier diagnostics or maybe syslog from syslog server?

 

Do you still have the original disk? Possibly nothing wrong with it. Connection problems are much more common than disk problems, And connection problems are often a reason for rebuilds going bad.

 

We can try to repair the filesystem on the rebuilt disk, but I will wait on your answers.

Handling of unmnountable drives is covered here in the online documentation available via the 'Manual' link at the bottom f the GUI.

6 minutes ago, trurl said:

Unfortunately, you rebooted after rebuilding, so syslog in diagnostics doesn't show anything about that. Do you have earlier diagnostics or maybe syslog from syslog server?

 

Do you still have the original disk? Possibly nothing wrong with it. Connection problems are much more common than disk problems, And connection problems are often a reason for rebuilds going bad.

 

We can try to repair the filesystem on the rebuilt disk, but I will wait on your answers.

 

I Don't have earlier diagnostics, or syslogs.  The drive that's failing is one of my oldest drives and started showing DMA errors prior to no longer showing up in the array.  I'm sure I can get it active long enough to extract any data on it.

 

I was able to mount the disk and fix the filesystem.   But  I'm still at a loss about the raid itself.  Is it healthy?  What happened to the data on the missing disk?  Should I just copy the data from the failed disk back to the array, format the new drive and all is good?

 

 

Just now, Groo said:

I was able to mount the disk and fix the filesystem. 

Post new diagnostics

1 minute ago, Groo said:

What happened to the data on the missing disk?

What missing disk? You rebuilt and repaired. You didn't also FORMAT did you?

Just now, trurl said:

Post new diagnostics

homer-diagnostics-20211214-0923.zip

4 minutes ago, Groo said:

I was able to mount the disk and fix the filesystem. 

Those diagnostics show disk1 still not mounted. Are you sure you took them after the repair? Is disk1 still unmountable?

2 minutes ago, trurl said:

What missing disk? You rebuilt and repaired. You didn't also FORMAT did you?

 

Nothing was formatted.

 

In the slot where the missing disk was (slot 1), it now shows as "Unmountable: not mounted"

 

 

5 minutes ago, Groo said:

I was able to mount the disk and fix the filesystem

You should post the output from the repair.

1 minute ago, Groo said:

In the slot where the missing disk was (slot 1), it now shows as "Unmountable: not mounted"

Then what exactly did you mean by this?

7 minutes ago, Groo said:

I was able to mount the disk and fix the filesystem. 

Actually, I should have questioned that statement to begin with. The reason it needed to be repaired was because it was unmountable. So obviously saying you mounted it before fixing it couldn't have happened. Maybe you don't understand the word "mount".

 

Just now, trurl said:

You should post the output from the repair.

The disk in slot 1 is the new drive that appears to have been rebuilt, but shows up as unmountable.

 

The old drive is mounted on /mnt/temp after I ran an xfs_check -L 

 

# xfs_repair -L /dev/sdm1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
destroyed because the -L option was used.
        - scan filesystem freespace and inode maps...
sb_fdblocks 500481120, counted 500481113
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
data fork in ino 4298782893 claims free block 805568742
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
Maximum metadata LSN (1:3286931) is ahead of log (1:2).
Format log to cycle 4.
done

 

1 minute ago, Groo said:

The old drive is mounted on /mnt/temp after I ran an xfs_check -L 

Do you mean you mounted the original using the command line? Unassigned Devices is the recommended way to mount unassigned disks.

 

If the original needed repair, then of course the rebuild would need repair also since rebuild is going to rebuild what was on the original.

is it possible that after it rebuilt the party on the new drive, the file system just needs to be repaired and it will be good?  i.e. the array is healthy, but the filesystem on that particular disk isn't.  i.e. if I run xfs_repair on the unmountable disk, it should recover?

Does the original look like it has all its contents after the repair? Is there anything in lost+found folder?

Just now, Groo said:

is it possible that after it rebuilt the party on the new drive, the file system just needs to be repaired and it will be good?  i.e. the array is healthy, but the filesystem on that particular disk isn't.  i.e. if I run xfs_repair on the unmountable disk, it should recover?

 

2 minutes ago, trurl said:

If the original needed repair, then of course the rebuild would need repair also since rebuild is going to rebuild what was on the original.

 

1 minute ago, trurl said:

Does the original look like it has all its contents after the repair? Is there anything in lost+found folder?

 

it looks good, no lost+found

17 minutes ago, Groo said:

one of my oldest drives and started showing DMA errors

Not clear that means there is anything wrong with the original disk. If it is still attached post new diagnostics and we can take a look at its SMART report.

 

Probably simplest to see how repair of the replacement goes and if it looks like anything is missing get it from the original. Make sure you repair the md device and not the sd device or you will invalidate parity.

 

7 minutes ago, trurl said:

Not clear that means there is anything wrong with the original disk. If it is still attached post new diagnostics and we can take a look at its SMART report.

 

Probably simplest to see how repair of the replacement goes and if it looks like anything is missing get it from the original. Make sure you repair the md device and not the sd device or you will invalidate parity.

 

# smartctl -H /dev/sdm -d sat
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.10.28-Unraid] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

 

Looks promising, but already have a replacement, so I'd rather not trust this disk.

 

How do I identify which md device refers to my unmountable disk?

The diagnostics would have given a much more detailed SMART report.

32 minutes ago, Groo said:

How do I identify which md device refers to my unmountable disk?

That information is in the wiki link provided earlier, but it is a bit wordy just for this simple answer.

 

disk1 is md1, disk2 is md2, etc. That doesn't account for possible encrypted disks though.

 

But, the best approach is to click on the disk in Main - Array Devices to get to its page, and do the test and repair from the webUI. That way it will use the correct device for the disk.

ok, stopped array, and restarted in maintenance mode.  Ran a (successful) repair on the unmountable disk (had to use -L) and started the array.

 

It appears to be happy.  Thanks for your help!

 

 

homer-diagnostics-20211214-1037.zip

2 minutes ago, Groo said:

It appears to be happy.  Thanks for your help!

👍