server crash while 2 hdd failed :-/

[Nexius2]

Hello, bad morning for me today.

I started rebuilding a disk early this week to upgrade. the data rebuild has not finished yet.

during the night, 2 other HDD failed... no luck...

I have 2 parity HDD but with the one that was rebuilding, I guess that takes the count to 3.... so my array doesn't like it.

my shares don't all show up, for example, I can't see the appdata (wich is on a cache so???)

 

can someone help to get this back up with minimum data loss?

thanks

nostromo-diagnostics-20231202-1001.zip

[Nexius2]

what I was thinking for now.

 

- get 2 new hdd

- preclear them

- copy data from crached HDD to new one (don't know exactly how for now)

- replace failed hdd with the new ones

- replace the hdd that was rebuilding with the old one

- reset the config to loose my 2 parity disks

- let it rebuild the parity

- cross my fingers very hard

 

this methode will take a very long time (couple days / weeks) and doesn't explain the loss of shares (and i think data on disks that are fine)

Edited December 2, 2023 by Nexius2

[JorgeB]

Looks more like a power/connection problem with disks 6 and 19, cancel the rebuild, check/replace cables for both disks and post new diags after array start.

[Nexius2]

doesn't seem like I can cancel it... nothing happens and i don't see it in the logs.

is there a command line to stop rebuild?

 

 

logs are full, it might explain why i can't cancel rebuild or do anything.

df -h /var/log
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           128M  128M     0 100% /var/log

nostromo-diagnostics-20231203-0935.zip

Edited December 3, 2023 by Nexius2

[JorgeB]

Type reboot in the CLI.

[Nexius2]

the server rebooted. but it says rebuild is finished (with lots of errors) I think it was disk 25. I didn't start the array yet but I guess i can't swap back since it is a 18To and the old was an 8To.

 

Disks 6 and 19 where moved but bad power or connexion problem, they are now disabled.

don't know what to do 😕

nostromo-diagnostics-20231203-1736.zip

Edited December 3, 2023 by Nexius2

[JorgeB]

There's a possibility the rebuilt disk has some corruption, disks 6 and 19 look healthy, one option that may work would be to force enable them and try the rebuild again, but I would at least replace the cables first.

[Nexius2]

how can i force enable them?

 

and how can i launch the rebuild again since it thinks the rebuild is over?

 

thanks

[JorgeB]

This will only work if parity is still valid:

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disk25
-Start array (in normal mode now) and post new diags.

[Nexius2]

as parity can't be valid after the reboot (rebuild disk marked as ok but has not even half the data on it), I decided to try copying data from failing disks to new ones.

after a preclear, I can format and mount new disks. but is there a way to mount disks from a stopped array that are disabled? I have hear of Unassigned device that could do it, but couldn't figure out how.

thanks

[JorgeB]

2 hours ago, Nexius2 said:

as parity can't be valid after the reboot (rebuild disk marked as ok but has not even half the data on it)

It could still be mostly valid, it was worth a try since it wouldn't cause any harm.

 

2 hours ago, Nexius2 said:

but is there a way to mount disks from a stopped array that are disabled?

Not sure what you mean, disable disks are emulated, if you mean mount the actual disk you can do that with UD, or do a new config to re-enable the disk (this will prevent rebuilding).

 

[Nexius2]

ok, I'll give it a try.

but when you say unassign disk 25, I need to put the old disk instead right?

 

I was searching how to mount a disk without starting the array and on a disk that is disabled. how can it be done with UD? can't find out 🙂

thanks

[JorgeB]

54 minutes ago, Nexius2 said:

ok, I'll give it a try.

but when you say unassign disk 25, I need to put the old disk instead right?

It can be a new one, as long as it's same or larger capacity

[Nexius2]

I must be missing something.

if I replace disk 25 with wathever, have disk 6 and 19 emulated, and reset the config, how is it supposed to emulate and repair 3 disks?

[JorgeB]

The new config with the instructions posted will force enable disks 6 and 19 and then you can see if disk25 can still be emulated.

[Nexius2]

disk 25 was unusable, rebuild not complet.

I reset the config, put back the old disk, changed cables for 6 and 19.

checked all disks individually then rebuilt both parity.

everything is ok now, going to cahnge again for disk 25

thanks