BTRFS error - Cache gone


Recommended Posts

Today i wanted to clear two drives and copyed some GB to another drive, with the goal to remove them from the array.

In this process my logs filled up with a lot of errors.

 

Apr 27 17:56:52 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091                                                                                                                                                         │
│Apr 27 17:56:52 serva4 kernel: message repeated 3 times: [ BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091]                                                                                                                            │
│Apr 27 17:56:56 serva4 kernel: blk_update_request: I/O error, dev loop3, sector 6427712 op 0x1:(WRITE) flags 0x100000 phys_seg 3 prio class 0                                                                                                                                             │
│Apr 27 17:56:56 serva4 kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 401, rd 0, flush 0, corrupt 0, gen 0                                                                                                                                                                  │
│Apr 27 17:56:56 serva4 kernel: blk_update_request: I/O error, dev loop3, sector 7603496 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0                                                                                                                                             │
│Apr 27 17:56:56 serva4 kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 402, rd 0, flush 0, corrupt 0, gen 0                                                                                                                                                                  │
│Apr 27 17:56:57 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091                                                                                                                                                         │
│Apr 27 17:56:57 serva4 kernel: message repeated 7 times: [ BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091]                                                                                                                            │
│Apr 27 17:57:00 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11791843328 wanted 894093 found 894087                                                                                                                                                         │
│Apr 27 17:57:00 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11791843328 wanted 894093 found 894087                                                                                                                                                         │
│Apr 27 17:57:00 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11790123008 wanted 894093 found 894087                                                                                                                                                         │
│Apr 27 17:57:00 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11790123008 wanted 894093 found 894087                                                                                                                                                         │
│Apr 27 17:57:02 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091                                                                                                                                                         │
│Apr 27 17:57:02 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091                                                                                                                                                         │
│Apr 27 17:57:07 serva4 kernel: verify_parent_transid: 6 callbacks suppressed                                                                                                                                                                                                              │
│Apr 27 17:57:07 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091                                                                                                                                                         │
│Apr 27 17:57:07 serva4 kernel: message repeated 7 times: [ BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091]                                                                                                                            │
│Apr 27 17:57:10 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11790123008 wanted 894093 found 894087                                                                                                                                                         │
│Apr 27 17:57:10 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11790123008 wanted 894093 found 894087  

 

so i stopped docker and vms and restarted unraid.

Now the cache cant mounted because of

 

Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): turning on async discard                                                                                                                                                                                                         │
│Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): using free space tree                                                                                                                                                                                                            │
│Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): has skinny extents                                                                                                                                                                                                               │
│Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): bdev /dev/sdf1 errs: wr 0, rd 0, flush 0, corrupt 40073, gen 0                                                                                                                                                                   │
│Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 27442, gen 0                                                                                                                                                                   │
│Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 40657, gen 0                                                                                                                                                                   │
│Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): bdev /dev/sde1 errs: wr 0, rd 0, flush 0, corrupt 27087, gen 0                                                                                                                                                                   │
│Apr 27 18:43:59 serva4 kernel: bond0: (slave eth0): link status definitely up, 10000 Mbps full duplex                                                                                                                                                                                     │
│Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): enabling ssd optimizations                                                                                                                                                                                                       │
│Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): start tree-log replay                                                                                                                                                                                                            │
│Apr 27 18:43:59 serva4 kernel: BTRFS error (device sdf1): incorrect extent count for 907568742400; counted 1347, expected 1338                                                                                                                                                            │
│Apr 27 18:43:59 serva4 kernel: BTRFS error (device sdf1): incorrect extent count for 907568742400; counted 1346, expected 1337                                                                                                                                                            │
│Apr 27 18:43:59 serva4 kernel: BTRFS: error (device sdf1) in convert_free_space_to_extents:454: errno=-5 IO failure                                                                                                                                                                       │
│Apr 27 18:43:59 serva4 kernel: BTRFS: error (device sdf1) in remove_from_free_space_tree:847: errno=-5 IO failure                                                                                                                                                                         │
│Apr 27 18:43:59 serva4 kernel: BTRFS: error (device sdf1) in btrfs_run_delayed_refs:2144: errno=-5 IO failure                                                                                                                                                                             │
│Apr 27 18:43:59 serva4 kernel: BTRFS: error (device sdf1) in btrfs_replay_log:2279: errno=-5 IO failure (Failed to recover log tree)                                                                                                                                                      │
│Apr 27 18:43:59 serva4 kernel: BTRFS error (device sdf1): open_ctree failed                                                                                                                                                                                                               │
│Apr 27 18:43:59 serva4 root: mount: /mnt/cache: can't read superblock on /dev/sdf1.                                                                                                                                                                                                       │
│Apr 27 18:43:59 serva4 emhttpd: shcmd (59): exit status: 32                                                                                                                                                                                                                               │
│Apr 27 18:43:59 serva4 emhttpd: /mnt/cache mount error: No file system                                                                                                                                                                                                                    │
│Apr 27 18:43:59 serva4 emhttpd: shcmd (60): umount /mnt/cache                                                                                                                                                                                                                             │
│Apr 27 18:43:59 serva4 root: umount: /mnt/cache: not mounted.                                                    

 

The cache has 4x500gb SSDs

 

Unmountable disk present:
Cache • SanDisk_SDSSDH3_500G_19438E803089 (sdb)
Cache 2 • SanDisk_SDSSDH3_500G_19438E804268 (sdc)
Cache 3 • SanDisk_SDSSDH3_500G_200207A01C89 (sde)
Cache 4 • SanDisk_SDSSDH3_500G_19438E800978 (sdf)

 

grafik.thumb.png.3e6fac33a83c3206475120d0042c381b.png

 

if i run btrfs check /dev i get a lot of checksum errors on each of the 4 drives.

 

root@serva4:~# btrfs check /dev/sdb1
Opening filesystem to check...
Checking filesystem on /dev/sdb1
UUID: 2ba1b265-e5ab-4e06-bafe-828647a53262
[1/7] checking root items
[2/7] checking extents
checksum verify failed on 908538724352 found 00000004 wanted 00000036
checksum verify failed on 908538724352 found 00000004 wanted 00000036
checksum verify failed on 908538724352 found 00000004 wanted 00000036
Csum didn't match
owner ref check failed [908538724352 16384]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
cache and super generation don't match, space cache will be invalidated
[4/7] checking fs roots
checksum verify failed on 908538724352 found 00000004 wanted 00000036
checksum verify failed on 908538724352 found 00000004 wanted 00000036
checksum verify failed on 908538724352 found 00000004 wanted 00000036
Csum didn't match

 

So how to fix this to restore the cache content?

thanks!

 

 

 

serva4-diagnostics-20210427-1859.zip

Link to comment

Thanks. I could manage to mount the cache and did a backup. Unraid can mount the cache, but only read only

If i try to

btrfs check --repair /dev/sdb1

 

i still get "Csum didn't match" errors.

 

Quote

root@serva4:~# btrfs check --repair /dev/sdb1
enabling repair mode
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/sdb1
UUID: 2ba1b265-e5ab-4e06-bafe-828647a53262
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
checksum verify failed on 908538724352 found 00000004 wanted 00000036
checksum verify failed on 908538724352 found 00000004 wanted 00000036
checksum verify failed on 908538724352 found 00000004 wanted 00000036
Csum didn't match
owner ref check failed [908538724352 16384]
repair deleting extent record: key [908538724352,168,16384]
adding new tree backref on start 908538724352 len 16384 parent 0 root 7
Repaired extent references for 908538724352
No device size related problem found
[3/7] checking free space tree
cache and super generation don't match, space cache will be invalidated
[4/7] checking fs roots
checksum verify failed on 908538724352 found 00000004 wanted 00000036
checksum verify failed on 908538724352 found 00000004 wanted 00000036
checksum verify failed on 908538724352 found 00000004 wanted 00000036
Csum didn't match

.

.

.

Csum didn't match
[5/7] checking only csums items (without verifying data)
checksum verify failed on 908538724352 found 00000004 wanted 00000036
checksum verify failed on 908538724352 found 00000004 wanted 00000036
checksum verify failed on 908538724352 found 00000004 wanted 00000036
Csum didn't match
Error going to next leaf -5
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 693552488448 bytes used, no error found
total csum bytes: 587268412
total tree bytes: 1140621312
total fs tree bytes: 351305728
total extent tree bytes: 110755840
btree space waste bytes: 154000473
file data blocks allocated: 1862878056448
 referenced 653472481280
root@serva4:~#

 

i found some websites that i should use

 

Quote

btrfs check --fix-crc

 

but i dont have the option.

Any Ideas?

Edited by corgan
Link to comment

some more logs

 

Quote

kernel: BTRFS info (device sdf1): turning on async discard                                               
kernel: BTRFS info (device sdf1): using free space tree                                                  
kernel: BTRFS info (device sdf1): has skinny extents                                                     
kernel: BTRFS info (device sdf1): bdev /dev/sdf1 errs: wr 0, rd 0, flush 0, corrupt 40073, gen 0         
kernel: BTRFS info (device sdf1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 27442, gen 0         
kernel: BTRFS info (device sdf1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 40657, gen 0         
kernel: BTRFS info (device sdf1): bdev /dev/sde1 errs: wr 0, rd 0, flush 0, corrupt 27087, gen 0         
kernel: BTRFS info (device sdf1): enabling ssd optimizations                                             
kernel: BTRFS info (device sdf1): checking UUID tree                                                     
emhttpd: shcmd (38271): sync                                                                             
kernel: BTRFS error (device sdf1): incorrect extent count for 907568742400; counted 1346, expected 1337  
kernel: BTRFS: error (device sdf1) in convert_free_space_to_extents:454: errno=-5 IO failure             
kernel: BTRFS info (device sdf1): forced readonly                                                        
kernel: BTRFS: error (device sdf1) in remove_from_free_space_tree:847: errno=-5 IO failure               
kernel: BTRFS: error (device sdf1) in btrfs_run_delayed_refs:2144: errno=-5 IO failure                   
kernel: BTRFS warning (device sdf1): Skipping commit of aborted transaction.                             
kernel: BTRFS: error (device sdf1) in cleanup_transaction:1942: errno=-5 IO failure                      
kernel: BTRFS error (device sdf1): incorrect extent count for 907568742400; counted 1347, expected 1338

 

Edited by corgan
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.