Hi all, diags attached below.  I checked my array today only to find two disks with unmountable file systems.  I have a dual parity array. 

 

I stopped the array and removed the device from the slot to see if I could at least browse the emulated contents but there was no option to do so, it just continued to say unmountable file system. 

 

I stopped the array again and then reattached the drives into their proper spot but now they say "new device." and if I try to start the array, it says it will do a parity sync/build.

 

What is my next move...if I start the array, is it saying it will rebuild those two missing disks?  Or is it saying it's going to rebuild the entire array and treat those disks as new and everything on them is lost?  FS is XFS encrypted if it it matters.

 

I'm worried that when I set the disks unassigned in the first instance, it is now unaware of their previous existance, and now parity won't be able to rebuild then anymore.

 

However, if im wrong and it's going to try to rebuild parity on those two disks, im ok with that, I'd just like to be sure I understand what unraid is saying is going to happen when I start the array.

 

I'm also wondering why when I unassigned it, it was still saying unmountable...possibly the parity also has some corruption on those drives?  

nas-diagnostics-20241118-2012.zip

Edited November 19, 20241 yr by oliver

One of the parity drives is showing a 100% error rate during the rebuild - I highly suspect a cable issue with the drive.

 

Can I stop the rebuild, fix the cable, and then restart from the beginning?  

 

Or should I let the current rebuild continue and presumably only one of the drives will be fixed, then I can 'redo' the rebuild to fix the other? I would prefer the former so I only have to rebuild once.

Unassign both disks, start the array in maintenance mode and check filesystem on both unmountable disks, run it without -n, then post new diags after array start.

1 hour ago, JorgeB said:

Unassign both disks, start the array in maintenance mode and check filesystem on both unmountable disks, run it without -n, then post new diags after array start.

thanks for the reply, unfortunately i stupidly was not aware of the filesystem check and i started the array rebuild process.  one of the parity drives immediately also failed during the rebuild.  i suspect that one is a cable issue.  (new diag and picture attached)

 

Can I stop the rebuild, fix the cable, and then restart from the beginning?  

 

Or should I let the current rebuild continue and presumably only one of the drives will be fixed, then I can 'redo' the rebuild to fix the other? I would prefer the former so I only have to rebuild once.

 

 

 

nas-diagnostics-20241119-0548.zip

Cancel the rebuild, check/replace cables for parity, then try to check filesystem on the emulated disks before rebuilding.

20 minutes ago, JorgeB said:

Cancel the rebuild, check/replace cables for parity, then try to check filesystem on the emulated disks before rebuilding.

 

 

here's the output of xfs repair on both drives without the -n option (attached via screenshot because paste format was not cleanly pasting).  should i still proceed with a rebuild once all cables are checked?  or is this filesystem level corruption

 

 

Edited November 19, 20241 yr by oliver

Use -L

11 minutes ago, JorgeB said:

Use -L

 

Both failed with input/output error, along with the parity drive again suddenly showing as failed again (diags attached).  I need to double check the hardware.  All of this nightmare started when I did a motherboard upgrade.  I'll see if there's any common connections between the three drives but also probably just go back to my old motherboard and see what happens.  I just find it really unlikely all 3 of these would fail at the same time like this, especially if they are not all part of the same connection and have no issues prior to mobo change.  Thank you very much for the assistance so far.  Attached diags if there's any more info to give.

 

The first one got a little further before i/o:

 

 

2nd one failed immediately. 

nas-diagnostics-20241119-0657.zip

Edited November 19, 20241 yr by oliver

Parity is still dropping offline, did you replace both cables?

3 hours ago, JorgeB said:

Parity is still dropping offline, did you replace both cables?

 

so i tried switching the cable's and i've removed the bad data drives for the moment.  the parity device still says disabled.  attached diagnostics.  should i assume the parity drive is toast and cannot be used to rebuild the two other disks? 

 

can i at least rebuild one of them with the good parity drive?

 

assuming parity is toast, for the two data disks, im thinking about trying to put them into a backup array as an unassigned disk and run a filesystem repair and see if i can salvage data from the lost/found.

nas-diagnostics-20241119-1039.zip

Edited November 19, 20241 yr by oliver

Reconnect the data drives and try force enabling parity to see if it can still emulate the disabled disks:

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed, including the disabled disks
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign both disks you want to rebuild
-Start array (in normal mode now) and post new diags.
 

4 hours ago, JorgeB said:

Reconnect the data drives and try force enabling parity to see if it can still emulate the disabled disks:

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed, including the disabled disks
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign both disks you want to rebuild
-Start array (in normal mode now) and post new diags.
 

 

sorry for the delay.  i installed the old motherboard and replaced all cables while i was at it.  after following the procedure, i get the parity drive back but bad disks will not emulate, filesystem error...im guessing the parity drive has a bad filesystem for those drives.  should i try to repair emulated FS? 

 

attached diags.  

nas-diagnostics-20241119-1525.zip

Edited November 19, 20241 yr by oliver

12 hours ago, oliver said:

should i try to repair emulated FS?

Yep.

3 hours ago, JorgeB said:

Yep.

 

hm im not sure what im doing wrong, now it just says 'not installed' when i unassign the two disks and start, there's no option for me to repair the FS.   

 

in maintence mode, it just says 'not installed'.  

 

in normal mode, it says 'not installed' and on the right it says 'unmountable or missing file system' but when i go to the disk settings there is no option to repair the FS.

 

we want to try to repair the emulated version not the original drive at this point, is that correct?  looking at the checkfilesystem procedure, im not sure why i don't have the option. 

 

diags attached

 

 

EDIT: do i need to follow the procedure you listed above again with new array config?

 

nas-diagnostics-20241120-0620.zip

Edited November 20, 20241 yr by oliver

1 hour ago, oliver said:

now it just says 'not installed' when i unassign the two disks and start,

That's normal, click on the disks and change the fs from auto to xfs, then start in maintenance mode and check filesystem

18 minutes ago, JorgeB said:

That's normal, click on the disks and change the fs from auto to xfs, then start in maintenance mode and check filesystem

 

done, i ran the check for a single disk (it seems to have kicked it off for both drives though, as the other one also shows a check file system run)  and i can see it did some repairs and it looks like ill have some cleanup in lost/found, which is fine.  

 

what would my next steps be?  diag attached.  

 

and also, i really want to say, thank you very much for all the help given so far. 

nas-diagnostics-20241120-0752.zip

Start the array in normal mode, see if the emulated disks mount and contents look correct, if yes, you can rebuild them.

1 hour ago, JorgeB said:

Start the array in normal mode, see if the emulated disks mount and contents look correct, if yes, you can rebuild them.

 

success!  thank you so much.  everything is in lost/found but that's understandable.  i did have to go back and run the file system check on the other bad disk, there is a bug in the UI where it was showing the result for disk 9 in the output for disk 12.  

 

to confirm the rebuild proces:

 

1.  set disk type back to auto (not required, but for good measure.  currently it's on xfs-encrypted)

2.  reassign disks (they will show up as 'new')

3.  start in maintenance mode (i want to avoid any activity during the process)

4. click 'sync' button to start data-rebuild

 

does that sound right?

Edited November 20, 20241 yr by oliver

That should do it, no need to change the fs. to auto, it will just change back to the correct one.

15 minutes ago, JorgeB said:

That should do it, no need to change the fs. to auto, it will just change back to the correct one.

 

thank you, now the long 24 hour or so wait.  but unlike last time, this one is proceeding without any errors so far.

 

i'll make a new thread to get some feedback on the root cause of this.  it was something with the new hardware not playing nice instead of the drives themselves i think.  

Edited November 20, 20241 yr by oliver

17 hours ago, JorgeB said:

That should do it, no need to change the fs. to auto, it will just change back to the correct one.

well unfortunately it seems to have made it to 78% and a bunch of read errors on the original parity disk started again.  the rebuild appears to be stuck, nothing is moving.  i assume at this point i should replace parity 1 and have it checked/rma'd separately.  

 

is there any last ditch point in retrying to salvage a single disk from parity 2 or should i just consider both data disks lost at this point?  considering the entirety of both drives was lost+found, im tempting to just write off the data.  

 

diag attached

nas-diagnostics-20241121-0238.zip

Edited November 21, 20241 yr by oliver

It still looks more like a power/connection issue, it may be worth trying to swap cables/slot with a different disk and retrying, with just parity2 you cannot recover any of the disabled disks, would lose the data from both.

24 minutes ago, JorgeB said:

It still looks more like a power/connection issue, it may be worth trying to swap cables/slot with a different disk and retrying, with just parity2 you cannot recover any of the disabled disks, would lose the data from both.

the UI works but stopping or pausing the rebuild isn't doing anything.   i do seem to hear some drive noise though so it sounds like something is still happening with drives?  parity progress is completely stalled on the UI though

 

i can't cancel the parity rebuild and shutdown to swap cables.  anything to be done with command line or should i force shutdown via power button?

Edited November 21, 20241 yr by oliver

Type powerdown in the CLI, if it doesn't shutdown after 5 minutes force it, no point in trying to continue the rebuild

 

 

4 hours ago, JorgeB said:

Type powerdown in the CLI, if it doesn't shutdown after 5 minutes force it, no point in trying to continue the rebuild

 

 

 

almost immediately on restarting the build the two data drives failed with some kind of odd message about swapping drives, but they basically became inaccessible.  at this point im swapping out the entire HBA card and PSU, along with all cables.  

 

nas-diagnostics-20241121-0648.zip

Edited November 21, 20241 yr by oliver