"Error 0x8000FFFF: Catastrophic failure' when attempting to write to server

[sheppp]

I have had my Unraid server for many years with no major issues. Today, I found that I cannot write to my the server from any of my Windows machines. I get a "Error 0x8000FFFF: Catastrophic failure" whenever I attempt to do so - with my cache drive turned off on my shares (I can write to the server when the cache drive is on, however, my cache drive is not moving data onto the shares. I could, however, move files from my cache drive to a share via Midnight Commander. I checked the syslog (see attached) and found the following:

 

"Sep 15 12:38:25 UR1 kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 15518 does not match to the expected one 2 Sep 15 12:38:25 UR1 kernel: REISERFS error (device md10): vs-5150 search_by_key: invalid format found in block 903401453. Fsck? Sep 15 12:38:25 UR1 kernel: REISERFS (device md10): Remounting filesystem read-only Sep 15 12:38:25 UR1 kernel: REISERFS error (device md10): vs-2140 finish_unfinished: search_by_key returned -2"

and

"Sep 15 12:56:14 UR1 root: Fix Common Problems: Error: Unable to write to disk13 Sep 15 12:56:14 UR1 kernel: REISERFS error (device md14): reiserfs-2025 reiserfs_cache_bitmap_metadata: bitmap block 753664 is corrupted: first bit must be 1"

 

I have rebooted the server and NONE of the disks are red-balled.

 

If you have any ideas re how to possibly resolve this, I would greatly appreciate it. Thanks.

 

ur1-syslog-20190916-0321.zip

[itimpi]

Looks like you have file system corruption on disk10 so it is being set read-only.    To fix such an issue you should stop the array; start it in Maintenance mode: click on disk10 on the Main tab and use the option to run a file system check.    Running the check from there will confirm any corruption and suggest what action is needed to fix it.

 

BtW:   You have a USB mouse that is continually spamming the syslog with disconnect messages making it very hard to spot genuine problems.

[sheppp]

Thanks. I'll do that.

[sheppp]

I ran the check and received this message:

Quote

 

"reiserfsck --check started at Sun Sep 15 22:19:56 2019

###########

Replaying journal:

Replaying journal: Done.

Reiserfs journal '/dev/md10' in blocks [18..8211]: 0 transactions replayed

Checking internal tree..  finished

Comparing bitmaps..Bad nodes were found, Semantic pass skipped

1 found corruptions can be fixed only when running with --rebuild-tree

###########

reiserfsck finished at Sun Sep 15 23:29:34 2019

###########

block 903401453: The level of the node (15518) is not correct, (2) expected

 the problem in the internal node occured (903401453), whole subtree is skipped

vpf-10640: The on-disk and the correct bitmaps differs."

 

I have never run --rebuild-tree before. Does anybody have any suggestions re how to do/fix this? Thanks.

[JorgeB]

30 minutes ago, sheppp said:

Does anybody have any suggestions re how to do/fix this?

Run reiserfsck --rebuild-tree

[sheppp]

I attempted to do this via telnet/putty with the command, "reiserfsck --rebuild-tree /dev/sdq"  (where "sdq" is the Unraid-assigned name of the drive), however, I received the message, "Failed to open the device '/dev/sdq'".  What am I doing wrong?  Thanks.

[trurl]

1 minute ago, sheppp said:

I attempted to do this via telnet/putty with the command, "reiserfsck --rebuild-tree /dev/sdq"  (where "sdq" is the Unraid-assigned name of the drive), however, I received the message, "Failed to open the device '/dev/sdq'".  What am I doing wrong?  Thanks.

Instead of sdq you should be using md#, where # is the assigned number of the disk. You should never try to use the sdX for a disk in the parity array or you will invalidate parity. And you can't just use sdX anyway since the command works on the filesystem (partition), not the disk. It would typically be sdX1 in that case, but as I said, you shouldn't be working with the sd anyway.

[sheppp]

Thanks!! That worked. For those like me that don't work much in Linux and can't remember anything about what was previously "learned" about using it, I used Putty to telnet in and used the command: ' reiserfsck --rebuild-tree /dev/md10 '  to repair the file system on my 10th drive. I really appreciate the help from you (itimpi, johnnie.black, and Constructor)! 

[JonathanM]

7 minutes ago, sheppp said:

 (itimpi, johnnie.black, and Constructor)! 

ROTFL @trurl

 

[itimpi]

20 minutes ago, sheppp said:

Thanks!! That worked. For those like me that don't work much in Linux and can't remember anything about what was previously "learned" about using it, I used Putty to telnet in and used the command: ' reiserfsck --rebuild-tree /dev/md10 '  to repair the file system on my 10th drive. I really appreciate the help from you (itimpi, johnnie.black, and Constructor)! 

FYI:  This could have been done from the GUI by clicking on the drive in the Main tab and using File System check/repair from there.    Less chance of error and less Linux knowledge required.