Unmountable: No file system / Issue started with read only filesystem

[schnub]

Oh dear...

 

I´ve just created a new VM dedicated to the cache array share. 

 

Then the ubuntu install showed me an error. I went and check the other vm that runs on the same "share". It indicated a read-only file system.

 

Ok so i rebooted the whole server to check if all will be allright then.

 

Now the share is gone and the cache disks says: Unmountable: No file system.

 

Cache is setup as RAID1 with 2x 256gb SSD´s.

 

Please help. 

 

Log of cache1 says:

 

Jan 24 11:51:10 bronas kernel: BTRFS critical (device sdk1): corrupt leaf: root=2 block=756464238592 slot=123, unexpected item end, have 26532 expect 10148
Jan 24 11:51:10 bronas kernel: BTRFS critical (device sdk1): corrupt leaf: root=2 block=756464238592 slot=123, unexpected item end, have 26532 expect 10148
Jan 24 11:51:10 bronas kernel: BTRFS: error (device sdk1) in __btrfs_free_extent:6805: errno=-5 IO failure
Jan 24 11:51:10 bronas kernel: BTRFS: error (device sdk1) in btrfs_run_delayed_refs:2935: errno=-5 IO failure
Jan 24 11:51:10 bronas kernel: BTRFS: error (device sdk1) in btrfs_replay_log:2281: errno=-5 IO failure (Failed to recover log tree)
Jan 24 11:51:10 bronas kernel: BTRFS error (device sdk1): pending csums is 40960
Jan 24 11:51:10 bronas kernel: BTRFS error (device sdk1): open_ctree failed

 

Log of cache2 says:

 

Jan 24 11:50:23 bronas kernel: sd 3:0:0:0: [sdd] Attached SCSI disk
Jan 24 11:50:23 bronas kernel: BTRFS: device fsid 52e88b1f-a7d5-4cdf-9e74-56e1b2c7c19e devid 2 transid 26253320 /dev/sdd1
Jan 24 11:50:36 bronas emhttpd: KINGSTON_SKC600256G_50026B7783BADFAC (sdd) 512 500118192
Jan 24 11:50:36 bronas emhttpd: import 31 cache device: (sdd) KINGSTON_SKC600256G_50026B7783BADFAC
Jan 24 11:50:38 bronas root: /usr/sbin/wsdd
Jan 24 11:51:10 bronas kernel: BTRFS info (device sdk1): bdev /dev/sdd1 errs: wr 0, rd 5, flush 0, corrupt 0, gen 0
Jan 24 11:51:10 bronas root: mount: /mnt/cache: can't read superblock on /dev/sdd1.
Jan 24 11:51:13 bronas root: /usr/sbin/wsdd

 

I´ve just made a fresh balance 3 days ago so cache2 should be synced up.

 

Can i just let the cache2 ssd handle it? and simulate maybe that cache1 is "dead" ? and then make a fresh balance of cache2 to cache1?

 

best regards

 

schnub

 

 

PS: if you need more information, please let me know. 

 

bronas-diagnostics-20210124-1153.zip

Edited January 24, 2021 by schnub

[JorgeB]

Jan 24 11:51:10 bronas kernel: BTRFS info (device sdk1): bdev /dev/sdd1 errs: wr 0, rd 5, flush 0, corrupt 0, gen 0

This means there were read error before on cache2, see here for better pool monitoring.

 

Jan 24 11:51:10 bronas kernel: BTRFS critical (device sdk1): corrupt leaf: root=2 block=756464238592 slot=123, unexpected item end, have 26532 expect 10148

This means there's filesystem corruption, best bet is to backup cache, re-format and restore the data.

[schnub]

Thanks for the fast reply. How would be the best steps for that?

 

how can I access the file system or files from the cache in unraid to copy the contents over if it’s corrupted? 
 

if I got the files from the cache array I’m happy. 
 

reformat then is not the problem.

 

best regards 

 

schnub
 

 

[JorgeB]

See here for some recovery options.

[schnub]

Thanks again.

 

I´ve tried to mount it ro it say´s superblock not found.

 

if i try to btrfs restore it says on the vm img file: it seems like where looping a lot y n a to continue.

 

But this is the most important file on the cache. 

 

i´ve only btrfs restored one of the cache disks. should i btrfs restore the whole "cache array" ?

 

sorry for that stupid questions. im kinda overwhelmed.

 

PS: is it possible just to restore this one file with btrfs restore?

 

PS²: 

 

 

i´ve put a for looping retry. and this shows up. so the file is corrupt?

Edited January 24, 2021 by schnub

[schnub]

Thread can be closed. I formatted the cache an copied the backup files over to the cache share.