Last night I noticed that my shares were not responding. After looking into it, all the drives attached to one of my LSI controllers were having issues.  My log partition was full due to samba and subsequently syslog errors, and I was unable to generate a diagnostics.  Any command which accessed the filesystem would just hang.  I shut down the system (after being unable to do it gracefully after trying multiple ways, I did the power button hold of shame.  

 

I reseated both of my controller cards, and restarted.  As the system was spinning back up, I heard the telltale clicking sound of a dead 6TB drive.  Once the system rebooted, I was greeted with my 2 parity drives disabled, and another "unmountable: unsupported of no filesystem" drive.  The drive which it thought needed to be formatted was indeed the 6TB drive making the dying sounds.  I ran an extended smart test on both parity drives and they turned out OK.  A SMART test on the questionable 6TB drive resulted in an error.  I do have a replacement for the 6TB in hand.

 

Here is my dilemma.  The common fix for disabled parity drives is to remove/readd/rebuild parity.  I don't want to do rebuild parity because I have a bad drive I need to rebuild.  Any thoughts on:

 

1. Should I just assume that the controller card unseated itself?  Are there any telltale signs of a failing card?  I've been running Unraid for over 13 years and have never had a card unseat itself like this.
2. It seems like a pretty big coincidence that I had a drive die at the same time as the controller malfunction.  One couldn't lead to the other would it?
3. Is there a way to tell why my 2 parity drives were disabled (and only those 2 drives)?  I know that might be tough/impossible w/o the logs from before my restart  
4. Is it safe to / is there a way to re-enable my 2 parity drives without having to rebuild what is on them, so I can rebuild the dead drive?
5. If my best bet is to abandon the dead drive, rebuild parity without it, then replace it, is there a way to see at least the filenames which were on it so I know what I need to replace?
6. I know about the LSI/Ironwolf drive disabling issue, but my 2 parity drives are Exos, and my card is a SAS3 card, which I believe doesn't suffer from the issue, correct?
7. What would be the best way to make sure this doesn't happen again?  Could I have gotten an alert as soon as the problem started?

 

I'll attach my diagnostics. Unfortunately it's from after the system reboot as they wouldn't generate before (they hung trying to query the drive controller), but should include the extended SMART test of the parity drives.

 

Thanks in advance!

 

 

jabba-diagnostics-20231028-2240.zip

Edited November 5, 20232 yr by tjiddy

You can try force-enable parity and see if they can emulated disk6, it will only work if parity is still valid:

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disk6
-Start array (in normal mode now) and post new diags.

Thanks!

I followed your steps and the parity drives are now enabled.  The missing disk is being emulated.  Here are the new diagnostics.  Just curious, why start in maintenance mode the first time??

jabba-diagnostics-20231029-0935.zip

3 hours ago, tjiddy said:

Just curious, why start in maintenance mode the first time??

To keep parity in sync as much as possible.

 

Check filesystem on disk22, run it without -n.

I must have missed a step.  disk22 is currently emulated.  That is the drive which was making a bad sound, had smart errors, and when I tried to attach the drive it showed up as "unmountable: unsupported of no filesystem".   Now when I set the device into disk 22 the slot It warns me "All existing data on this device will be OVERWRITTEN when array is Started"

 

Currently checking the filesystem on that device isn't possible because Unraid doesn't recognize the filesystem on the device.  (Looks like it hasn't completely died yet, maybe just corrupted to the point where the fs is unrecognized due the controller errors/hard shutdown? )   Should I re-add it, and let Unraid rebuild it from parity before checking the filesystem?  If so, since I'm planning on replacing that drive, would it be better to put the replacement in, let it rebuild, then check the fs?

 

 

12 hours ago, tjiddy said:

I must have missed a step.  disk22 is currently emulated.  That is the drive which was making a bad sound, had smart errors, and when I tried to attach the drive it showed up as "unmountable: unsupported of no filesystem".   Now when I set the device into disk 22 the slot It warns me "All existing data on this device will be OVERWRITTEN when array is Started"

Don't add a replacement drive for now.

 

12 hours ago, tjiddy said:

Currently checking the filesystem on that device isn't possible because Unraid doesn't recognize the filesystem on the device. 

Change the filesystem from auto to xfs then check it.

 

Thank you for all of your help so far.  It's really appreciated.   

 

I'm a little confused how to check the filesystem for disk 22.  This is what my UI is showing when the array is started in maintenance mode.

 

When I go into the drive settings, I don't even see the check filesystem section, even though I'm in maintenance mode.  (I do see it for the other drives).

 

Do you want me to rebuild the drive from parity, then check the filesystem?

 

17 minutes ago, tjiddy said:

Do you want me to rebuild the drive from parity, then check the filesystem?

No.

 

6 hours ago, JorgeB said:

Change the filesystem from auto to xfs then check it.

or just type in the console:

xfs_repair -v /dev/md22p1

 

11 minutes ago, JorgeB said:

xfs_repair -v /dev/md22p1

Phase 1 - find and verify superblock...
        - block cache size set to 717200 entries
sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128
resetting superblock root inode pointer to 128
sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129
resetting superblock realtime bitmap inode pointer to 129
sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130
resetting superblock realtime summary inode pointer to 130
Phase 2 - using internal log
        - zero log...
zero_log: head block 114006 tail block 114002
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

 

Use -L

Done.  Results attached.

checkfs_output.txt

Now start the array in normal mode, emulated disk22 should mount now, check contents, also look for a lost+found folder.

Sorry to be overly paranoid, but it's still ok to start with with this message?

I didn't say to assign the old disk, start the array with disk22 still unassigned.

Array is started.  disk 22 only contains a lost+found directory with what looks like most the old contents of the drive spread throughout numeric folders.  I've noted the filenames in case they need to be replaced.   Does this mean the data is lost, or will parity be able to rebuild this disk once replaced?

 

Thank you so much for getting me this far!  What would be my next step?

Parity will rebuild the same data you are already seeing on the emulated disk, unless the old disk can still be read that's what can be recovered with parity.

Thank you.  How could this have been avoided?  I thought the whole point of parity was to be able to rebuild the contents of failed drives?

Possibly because parity wasn't 100% in sync due to the parity disks becoming disabled after the disk, don't forget that you had to force enable parity, and I did mention:

 

On 10/29/2023 at 10:20 AM, JorgeB said:

it will only work if parity is still valid:

 

Note that you may be able to recover most data from the lost+found folder, also make sure taht the old disk was really dead, it may have been a power/connection issue, you can post a SMART repost so we can take a look.

 

I ran an extended SMART test.  Completed: read failure

I'll attach the report

WDC_WD60EFRX-68L0BN1_WD-WX31D17DJK65-20231030-1343.txt

Edited October 31, 20232 yr by tjiddy

The disk is failed and needs to be replaced, but you may be able to recover most of the data with ddrescue, with most cases like that it can recover 99%+