Cache disk corrupted (and a possibly another disk)

[JohnGAG]

I'd like advice on how best to approach recovery?

 

The cache shows a BTRFS corruption. 

BTRFS error (device nvme0n1p1: state EA): parent transid verify failed on logical 119125622784 mirror 1 wanted 4234875 found 4233357

...

BTRFS critical (device nvme0n1p1): corrupt leaf: root=7 block=118272999424 slot=2, bad key order, prev (18446744073709551606 128 99098980352) current (18413109569270616310 255 99098984448)

...

 BTRFS: error (device nvme0n1p1) in btrfs_commit_transaction:2494: errno=-5 IO failure (Error while writing out transaction)

Jan  2 09:18:30 Tower kernel: BTRFS info (device nvme0n1p1: state E): forced readonly

 

( I ran check filesystem status --report & started to run --repair, but aborted when I saw it suggested that should only be done "under advice")

 

Diagnostics attached (nb edited out some file copy errors which showed content)

 

Possibly connected

- It is possible that this was caused UPS failure (late Christmas day)

- I have noticed app (like Plex) failing as permissions have changed to read only ( I can now see this in the logs)

- initially recovered by variously re-installing Plex, restarting the server, running scrub (fix corruptions).

 

Data on cache

I have Appdata Backup installed

the cache has (had?)

• appdata & directories on it,
• and some files from a backup process - these were reported as having failed
• I've edited the filenames out (which looked like text content but were from a macintosh backup app called Carbon Copy Cloner

 

 

The "appdata" share with all the docker configs is no longer visible

Parity check has been running since the reboot (no obvious issues yet - but I see syslog notes a XFS corruption)

 

XFS (md2p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0xc0004408 dinode
Jan  1 23:01:34 Tower kernel: XFS (md2p1): Unmount and run xfs_repair

 

Should I run "BTRFS --repair" on the cache?

should I "unmount andrun XFS_repair"

 

or are there some other checks or steps recomended?

thanks

John

 

tower-diagnostics-20240102-1303.zip

[JorgeB]

For the cache I would recommend backing up and re-formatting, for the disk you can check filesystem, run it without -n

[JohnGAG]

Thanks Jorge!

 

I appear to have backed up, reformatted and restored ok.

It's now running without errors showing on the log

 

I struggled a little over the BTRFS reformatting.

Not sure if I did it the right / best way. 

• although it was a BTRFS "single" pool I couldn't find a GUI option to reformat (to either BTRFS or XFS). I had expected to find this from the FAQ text
• I couldn't find the right command from the terminal (within Unraid) - this might be my insufficient command line knowledge...
• eventually shut the system down and restarted it in safe mode (directly on the server) and "reformatted" there
• not sure if I truly reformatted it, but this was sufficient to persuade Unraid that it was the cache pool was unformatted, which then permitted formatting within the Unraid GUI
• I tried to set it to XFS (which the FAQ suggested for single cache devices), but it nonetheless restarted as "auto" and changed "auto"matically to BTRFS

 

[itimpi]

6 minutes ago, JohnGAG said:

I couldn't find a GUI option to reformat (to either BTRFS or XFS). I had expected to find this from the FAQ text

There is this in the online documentation accessible via the Manual link at the bottom of the Unraid GUI.

[JohnGAG]

yes

point 7 was where I discovered that XFS was recommended. 

tried starting and stopping as per other lines but "file system type" remained greyed out.

 

I'm not claiming expertise here, or criticising... but I've sometimes learnt valuable detail from the experiences of the less knowledgeable when searching forums to see if someone has had similar experience (also why my warning about might not be right)