Earlier today I had a 10TB drive fail out with multiple i/o write errors.  I had a larger 14TB drive on standby, parity drives are both 14TB, so I stopped the array, replaced the failed disk and started the array.  After doing this, I noticed that the new drive reported as unmountable, but not until after I had started the array in attempt to rebuild the failed disk.

 

So, I stopped the array, which may have been a mistake.  I attempted a few different things including starting the array again without a disk present and now it reads the failed disk, whose contents are supposedly emulated as unmountable.  I'm afraid to do anything further, because even with some googling I honestly don't know what I should do to troubleshoot this issue.  Also a bit frustrated because this same disk failed before, but with a different drive in its place.

 

I also clicked on the checkmark in unassigned devices next to the drive I tried to use as a replacement, and it started a file system check on that? so I guess it's going to do that for the next 20ish hours or so.  Anyways, any help would be appreciated.

syslog 5.14.25

Attach Diagnostics ZIP to your NEXT post in this thread. 

Done.  I currently have the array in maintenance mode with the new 14TB drive in the failed disks place, but I haven't restarted the sync/rebuild process on it.

tower-diagnostics-20250514-2006.zip

Check filesystem on emulated disk13 from the webUI and post the output.

6 minutes ago, trurl said:

Check filesystem on emulated disk13 from the webUI and post the output.

I went to Disk 13's settings per the instructions and clicked "Check" under Check Filesystem Status, however, Unraid seems hung up.  Is it normally supposed to take a while for anything to happen after clicking that button?

Did you get any output at all?

3 minutes ago, trurl said:

Did you get any output at all?

No, it's still showing the loading icon.

Alright, I refreshed the page after waiting a while and this is what's displayed:

 

This is checking the file system on the emulated disk and not the new drive I put in the failed disks place correct?

Edited May 15, 20251 yr by chris206

9 hours ago, chris206 said:

Alright, I refreshed the page after waiting a while and this is what's displayed:

 

This is checking the file system on the emulated disk and not the new drive I put in the failed disks place correct?

Sadly this is still running on the disk, should I cancel the process?  It's been roughly 11 hours without any apparent progress.

If it hasn't found a backup superblock by now, it won't, upgrade to 7.1.2, and try again, since it has newer xfs progs, if the same, the best bet is to restore from the old disk, or a backup, or use a file recovery app like UFS explorer.

2 hours ago, JorgeB said:

If it hasn't found a backup superblock by now, it won't, upgrade to 7.1.2, and try again, since it has newer xfs progs, if the same, the best bet is to restore from the old disk, or a backup, or use a file recovery app like UFS explorer.

Alright, I cancelled the check, but that seemed to hang on something.  So, I went and stopped the array and got two errors regarding the drive being busy.  It won't even let me take the drive out of the array.  How can I see what process is accessing the drive and stop it?  The server had been in maintenance mode.

 

Edit:  It also says the array is started and it seemingly won't let me do anything...

Edited May 15, 20251 yr by chris206

Type reboot in the CLI, if it doesn't reboot in 5 minutes, you will need to force, but since the array was in maintenance mode, there shouldn't be any issues.

Rebooting from the CLI worked.  However, this brought in a whole new host of issues...

 

Server booted seemingly normally, found the disks and then it started freaking out over a drive that failed previously, but was only in unassigned devices.  Now, it seems every drive has been dropped out of the array and there are issues with my HBA card?  

 

Edit: no drives are detected and LED lights on the card are off...

 

tower-syslog-20250515-1659.zip

Edited May 15, 20251 yr by chris206

16 hours ago, trurl said:

Attach Diagnostics ZIP to your NEXT post in this thread. 

 

I may have screwed myself further.  Rebooted and it's taking a while to come back...

 

Edit: the LEDs on the card are back and are active, but no idea how to read them...

Update: server came back up, recognized the drives and it's attempting to read that disk again.  I have a feeling I need to shutdown and remove it entirely before moving forward.

 

Update 2: It went through the same pattern, so I've shutdown and plan to remove the offending disk and will try to work from there.  I should've removed it over a month ago in hindsight...

tower-diagnostics-20250515-1210.zip

Edited May 15, 20251 yr by chris206

Alright, offending disk removed, server seems to be stable now, array not started.

 

Here's where I'm at:

 

The original disk that started this all with read errors remains in unassigned devices (sds).

I have the replacement disk that highlighted filesystem errors in Disk 13's place.  This disk seemed to have issues with its format previously?  I'm not entirely sure, that information should be in the original syslog and/or diagnostics.

I have NOT updated to 7.1.2, still on 7.0.1

 

Should I proceed with updating as previously instructed or do other things first?

tower-diagnostics-20250515-1247.zip

The old disk13 looks OK, see if it mounts with the UD plugin.

Attempted it twice, both times seemed to fail, do I need to remove the replacement drive first? 

 

Edit: attached full diagnostics to this post.

 

tower-diagnostics-20250515-1504.zip

Edited May 17, 20251 yr by chris206

Disk is 4Kn, try this:

 

mkdir /x
mount -t xfs -o offset=0 /dev/sds1 /x

 

If it works, browse /x to confirm all goods good, then unmount with

 

unmount /x

 

10 hours ago, JorgeB said:

Disk is 4Kn, try this:

 

mkdir /x
mount -t xfs -o offset=0 /dev/sds1 /x

 

If it works, browse /x to confirm all goods good, then unmount with

 

unmount /x

 

 

So, I tried this, but I'm a bit uncertain about what I should be looking for?  I used the webterminal to enter the first two commands, but as far as browsing the created directory, assuming it mounted, I type "browse /x" into the same terminal?  Or do I need to have the array started in maintenance mode before doing any of this?

tower-diagnostics-20250516-1147.zip

Edited May 16, 20251 yr by chris206

I meant browse using your favorite tool, or the CLI, post the output from

 

ls -l /x

 

5 minutes ago, JorgeB said:

I meant browse using your favorite tool, or the CLI, post the output from

 

ls -l /x

 

 

Ah sorry, if it's not already obvious, I'm quite inexperienced with using linux at this level.

 

 

Output:

Quote

total 0
drwxrwxrwx 3 nobody users 31 Jun 30  2024 backups/
drwxrwxrwx 5 nobody users 72 Aug  5  2022 media/
drwxrwxrwx 5 nobody users 58 Feb 18  2022 private/

 

Those seem to be the correct folders for the drive.

If you want to browse it more, you can use midnight commander for example, mc on the console, then got to /x and you can enter all the folders, but all should be there.

 

Don't forget to unmount when you are done:

 

unmount /x

 

You can then do a new config and use the old disk instead, then resync parity., or since it should be mostly valid, check "parity is already valid" and run a correcting check.

 

Any doubts please ask.

1 minute ago, JorgeB said:

If you want to browse it more, you can use midnight commander for example, mc on the console, then got to /x and you can enter all the folders, but all should be there.

 

Don't forget to unmount when you are done:

 

unmount /x

 

You can then do a new config and use the old disk instead, then resync parity., or since it should be mostly valid, check "parity is already valid" and run a correcting check.

 

Any doubts please ask.

 

By new config, you mean placing it back into the array where it was originally?  As for parity checking, I should start the array in maintenance mode and run a check through the CLI?  If I can check parity before doing corrections I would probably prefer that.

 

Also thanks, I greatly appreciate the help I've received.

 

https://docs.unraid.net/unraid-os/manual/storage-management/#reset-the-array-configuration