recover from double drive issues (again)

[korro]

Hi.

Unfortunately, I'm having the same issue I had a couple of months back, but this time it looks even worse. 

You can find the original thread here: https://forums.unraid.net/topic/138169-recover-from-double-drive-issues/#comment-1254907

 

To sum it up the problem the last time was a drive having issues, (probably cables or the hard drive cage backplane). after trying to rebuild the drive the parity drive also started to throw errors. Luckily everything went well and I recovered all the data.

 

This time the problem is similar, drive 6 started throwing errors, like last time, and yes, it's the same drive, in the same cage but with a different cable, so I'm starting to think it's the backplane of the cage. This time the drive was also unmountable due to file system corruption. I solved it following the official guide and started the rebuild. Like last time the parity drive started throwing errors during rebuild. Again, same drive, same cage, different cable. And again, this time it's worse than last time, after a reboot the parity is also disabled, so now the array has parity disabled and disk 6 emulated as you can see in the image attached (how? is it possible to have a disabled parity but still use it for drive emulation?)

 

Unraid gives me the option to start the array and rebuild, but I don't know if it will keep all the data because of the 2 issues. Can I recover from this?

glados-diagnostics-20230629-1932.zip

[JorgeB]

If a disk is invalid (not disabled) Unraid can disable another one, it shouldn't, but it does, you can try to force enable parity and see if disk6 can still be emulated:

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disk6
-Start array (in normal mode now) and post new diags.
 

 

 

[korro]

thank you for the very quick reply.

I will check as soon as i get home

 

EDIT:

Quote

If a disk is invalid (not disabled) 

just a quick note on this, i'm quite sure the parity does say disabled when i hover on the red X

Edited June 29, 2023 by korro

[JorgeB]

12 hours ago, korro said:

just a quick note on this, i'm quite sure the parity does say disabled when i hover on the red X

Disk6 is and was invalid, not parity.

[korro]

Sorry for the delay, it has been a rough couple of days.

I've done as you said and you can find the diags attached.

I've also noticed an error on drive 2 smart, it's a very old drive, i think it's time to replace it after everything else is resolved.

glados-diagnostics-20230630-1956.zip

[JorgeB]

Forgot that there's a bug with this release, stop array, click on disk6, set filesystem from auto to xfs, start array, post new diags.

[korro]

ok, new diags attached

glados-diagnostics-20230630-2023.zip

[JorgeB]

Check filesystem on disk6, if after doing that the emulated disk mounts and contents look good you can rebuild.

[korro]

the check detected a corruption. Running it again with writing enabled gave me the following message:

Quote

ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed. Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair. If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

 

shoud i try to mount and replay the log as suggeted? how do i do that?

[itimpi]

You simply need to rerun without -n and supplying the -L option.   

[korro]

unfortunately it failed.

 

Quote

xfs_repair: Releasing dirty buffer to free list!

cache_purge: shake on cache 0x50c440 left 1 nodes!?

xfs_repair: Refusing to write a corrupt buffer to the data device!

xfs_repair: Lost a write to the data device!

fatal error -- File system metadata writeout failed, err=117. Re-run xfs_repair.

 

[JorgeB]

Try again, but if xfs_repair cannot repair the filesystem not much more you can do, other than for example usinging a file recovery util like UFS explorer to see if it can recover something.

[korro]

ok, the second try did it.

The content still looks quite good so i'm rebuinding, let's hope this goes smoothly.

[korro]

Ok, data rebuild completed with 33 errors but everything seems to be working correctly to me. 

Is this normal?

 

[JorgeB]

Post new diags.

[korro]

sorry for the delay.

new diags attached

glados-diagnostics-20230703-1859.zip

[JorgeB]

Not seeing any errors during the rebuild and check, where did you see the errors? If it was a notification they are known to some times give info from a previous check/rebuild.

[korro]

yes, it was a notification. I received serveral notifications that looked like notifications from previous runs and i ignored them, but this was the notification about the finished data rebuild so it scared me a bit.

If you say there's nothing to worry about then i think i can close the thread. Thank you very much, you have been very helpful, as always!