Powerloss / but clean shut down with UPS

[Gsusking2]

Hi guys,

This morning my server lost power.

I believe the UPS saved it and put it into a safe shutdown.  The same cant be said for the JBOD attached to my unraid server.

Basically i have 2 drives that needed i file system repair which i did and restarted the array.  The 2nd parity drive started to show errors

I am not amazing at unraid and fixing disk errors etc. as you can see from my post history. But i do have a basic understanding

I have attached my diagnostics. if anyone can chime in as to what i should do next it would be appreciated 

tower-diagnostics-20230330-1317.zip

[Gsusking2]

I should add it is currently rebuilding right now.

[JorgeB]

Parity2 dropped offline, cancel the rebuild, check/replace cables/slot and post new diags after array start.

[Gsusking2]

hi, 

I have done so and the parity seems ok now. Though disk 8 is now having read errors. i did not touch this disk.  This must be coming from wire/card.  I will investigate this further and post a new diagonostic.

It has started rebuilding disk 5 and 14. Should i allow this? even tho they are in an 'unmountable' state?

 

tower-diagnostics-20230330-1351.zip

Edited March 30, 2023 by Gsusking2
Clarityx2

[JorgeB]

Not seeing any errors on disk8, but cancel the rebuild and check filesystem on both emulated disks first.

[Gsusking2]

Yeah, Its asking for -L repair on the disk (5) im doing so now.

Should i do the same for disk 14 if it asks for the -L repair.

 

Once i finish this i should reboot or just start array and let it rebuild?

[Gsusking2]

hmmmm disk 8 seems to be having read/write errors on data rebuild.

could just be a power issue.  should i let the rebuild run its course and figure out disk 8 afterwards?

 

tower-diagnostics-20230330-1505.zip

[JorgeB]

Disk8 dropped offline, similar to parity2 before, cancel the rebuild, if you let it continue it will rebuild corrupt disks.

 

13 hours ago, Gsusking2 said:

Yeah, Its asking for -L repair on the disk (5) im doing so now.

Should i do the same for disk 14 if it asks for the -L repair.

Yes, if is asks for -L use it.

 

[Gsusking2]

Ok this makes sense.

During rebuild a random disk will start throwing read errors and will completely kibosh the rebuild process.

first it was disk 8 now its disk 1.

 

[JorgeB]

Could be a power problem.

[Gsusking2]

in this scenario, if im unable to rebuild because of another disk failing.  is the array toast? i have 2 parity drives. 

Should i run the sync (data rebuild) in maintenance mode?

 

Edited March 31, 2023 by Gsusking2
additional info

[JorgeB]

Not the complete array, but the rebuilt disks will likely have some corruption if you continue after read errors on a different disk.

[Gsusking2]

ok so basically disk 1 has become unmountable now.

So currently i have disk 1 - 5 - 14 down.

I started the array it started the rebuild process and now im not able to access my webgui.

 

I shut down the server blind and will post new diagonostics

IF anyone needs to shut down their server blind 

1 Plug in a keyboard

2 put in root (enter) password (enter)

3 enter command  "shutdown -h now"

 

Edited March 31, 2023 by Gsusking2

[Gsusking2]

here is the diagnostics of 3 emulated disks.

1 5 14

1 is a red X

5 and 14 each have the Triangle exclamation point.

In this case what do i do. Data preservation on these disks isn't crucial (all replaceable data)

tower-diagnostics-20230331-1238.zip

[JorgeB]

With 3 invalid disks Unraid won't be able emulate any of them, I can give you instructions to re-enable disk1 and you can try another rebuild, but if other disks keep dropping it will be the same issue.

[Gsusking2]

Ok i would be interested in this information. (the re-enable of disk 1)

If disks were to keep dropping could i just remove and replace said disks loose a lot of data and keep going?

 

I really appreciate your help with this. Let me know if there is a link i can use to buy you a beer.

Edited March 31, 2023 by Gsusking2
to give thanks

[JorgeB]

16 minutes ago, Gsusking2 said:

(the re-enable of disk 1)

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disks 5 and 14
-Start array (in normal mode now), ideally the emulated disks will now mount and contents look correct, if they don't you should run a filesystem check on the emulated disks
-If the emulated disks mount and contents look correct stop the array
-Re-assign disks 5 and 14 and start array to begin.

 

27 minutes ago, Gsusking2 said:

If disks were to keep dropping could i just remove and replace said disks loose a lot of data and keep going?

Problem is if different disks keep dropping, though reducing the array size might help, but to rebuild those two disks you need all the other ones, you could only remove after the rebuilds finish.

[Gsusking2]

49 minutes ago, JorgeB said:

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disks 5 and 14
-Start array (in normal mode now), ideally the emulated disks will now mount and contents look correct, if they don't you should run a filesystem check on the emulated disks
-If the emulated disks mount and contents look correct stop the array
-Re-assign disks 5 and 14 and start array to begin.

 

Problem is if different disks keep dropping, though reducing the array size might help, but to rebuild those two disks you need all the other ones, you could only remove after the rebuilds finish.

Ok so i got to the step where i stop the array unassigned the disks, then restart the array in normal mode (it says Disks missing please replace asap).  I stop the array reassigned them. and now its doing a data rebuild and disk 1 is immediately throwing errors. and its saying disk 5 and 14 are unmountable file system or no file system. when i start in maintenance mode i am no longer able to check the file system on these disks.

 

[Gsusking2]

I have since stopped the rebuild and restarted. here are the diagnostics after restarting after what happened above.

 

tower-diagnostics-20230331-1426.zip

[JorgeB]

9 minutes ago, Gsusking2 said:

when i start in maintenance mode i am no longer able to check the file system on these disks.

click on both disks and change the filesystem from auto to xfs, then you will be able to check filesystem again.

[Gsusking2]

Thank you,

That worked.

Now disk 1 is constantly throwing CRC errors.

Im gonna try to reseat it and put it into my JBOD on a different power supply etc.

 

If this doesnt work. How do a move forward with 2 disks needing a rebuild and disk 1 being a jerk.

 

[JorgeB]

We cannot rebuild the two disks if another disk has issues, so would need to fix that first.