corgan Posted April 27, 2021 Posted April 27, 2021 Today i wanted to clear two drives and copyed some GB to another drive, with the goal to remove them from the array. In this process my logs filled up with a lot of errors. Apr 27 17:56:52 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091 │ │Apr 27 17:56:52 serva4 kernel: message repeated 3 times: [ BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091] │ │Apr 27 17:56:56 serva4 kernel: blk_update_request: I/O error, dev loop3, sector 6427712 op 0x1:(WRITE) flags 0x100000 phys_seg 3 prio class 0 │ │Apr 27 17:56:56 serva4 kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 401, rd 0, flush 0, corrupt 0, gen 0 │ │Apr 27 17:56:56 serva4 kernel: blk_update_request: I/O error, dev loop3, sector 7603496 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0 │ │Apr 27 17:56:56 serva4 kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 402, rd 0, flush 0, corrupt 0, gen 0 │ │Apr 27 17:56:57 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091 │ │Apr 27 17:56:57 serva4 kernel: message repeated 7 times: [ BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091] │ │Apr 27 17:57:00 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11791843328 wanted 894093 found 894087 │ │Apr 27 17:57:00 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11791843328 wanted 894093 found 894087 │ │Apr 27 17:57:00 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11790123008 wanted 894093 found 894087 │ │Apr 27 17:57:00 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11790123008 wanted 894093 found 894087 │ │Apr 27 17:57:02 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091 │ │Apr 27 17:57:02 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091 │ │Apr 27 17:57:07 serva4 kernel: verify_parent_transid: 6 callbacks suppressed │ │Apr 27 17:57:07 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091 │ │Apr 27 17:57:07 serva4 kernel: message repeated 7 times: [ BTRFS error (device loop3): parent transid verify failed on 11629772800 wanted 894093 found 894091] │ │Apr 27 17:57:10 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11790123008 wanted 894093 found 894087 │ │Apr 27 17:57:10 serva4 kernel: BTRFS error (device loop3): parent transid verify failed on 11790123008 wanted 894093 found 894087 so i stopped docker and vms and restarted unraid. Now the cache cant mounted because of Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): turning on async discard │ │Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): using free space tree │ │Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): has skinny extents │ │Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): bdev /dev/sdf1 errs: wr 0, rd 0, flush 0, corrupt 40073, gen 0 │ │Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 27442, gen 0 │ │Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 40657, gen 0 │ │Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): bdev /dev/sde1 errs: wr 0, rd 0, flush 0, corrupt 27087, gen 0 │ │Apr 27 18:43:59 serva4 kernel: bond0: (slave eth0): link status definitely up, 10000 Mbps full duplex │ │Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): enabling ssd optimizations │ │Apr 27 18:43:59 serva4 kernel: BTRFS info (device sdf1): start tree-log replay │ │Apr 27 18:43:59 serva4 kernel: BTRFS error (device sdf1): incorrect extent count for 907568742400; counted 1347, expected 1338 │ │Apr 27 18:43:59 serva4 kernel: BTRFS error (device sdf1): incorrect extent count for 907568742400; counted 1346, expected 1337 │ │Apr 27 18:43:59 serva4 kernel: BTRFS: error (device sdf1) in convert_free_space_to_extents:454: errno=-5 IO failure │ │Apr 27 18:43:59 serva4 kernel: BTRFS: error (device sdf1) in remove_from_free_space_tree:847: errno=-5 IO failure │ │Apr 27 18:43:59 serva4 kernel: BTRFS: error (device sdf1) in btrfs_run_delayed_refs:2144: errno=-5 IO failure │ │Apr 27 18:43:59 serva4 kernel: BTRFS: error (device sdf1) in btrfs_replay_log:2279: errno=-5 IO failure (Failed to recover log tree) │ │Apr 27 18:43:59 serva4 kernel: BTRFS error (device sdf1): open_ctree failed │ │Apr 27 18:43:59 serva4 root: mount: /mnt/cache: can't read superblock on /dev/sdf1. │ │Apr 27 18:43:59 serva4 emhttpd: shcmd (59): exit status: 32 │ │Apr 27 18:43:59 serva4 emhttpd: /mnt/cache mount error: No file system │ │Apr 27 18:43:59 serva4 emhttpd: shcmd (60): umount /mnt/cache │ │Apr 27 18:43:59 serva4 root: umount: /mnt/cache: not mounted. The cache has 4x500gb SSDs Unmountable disk present: Cache • SanDisk_SDSSDH3_500G_19438E803089 (sdb) Cache 2 • SanDisk_SDSSDH3_500G_19438E804268 (sdc) Cache 3 • SanDisk_SDSSDH3_500G_200207A01C89 (sde) Cache 4 • SanDisk_SDSSDH3_500G_19438E800978 (sdf) if i run btrfs check /dev i get a lot of checksum errors on each of the 4 drives. root@serva4:~# btrfs check /dev/sdb1 Opening filesystem to check... Checking filesystem on /dev/sdb1 UUID: 2ba1b265-e5ab-4e06-bafe-828647a53262 [1/7] checking root items [2/7] checking extents checksum verify failed on 908538724352 found 00000004 wanted 00000036 checksum verify failed on 908538724352 found 00000004 wanted 00000036 checksum verify failed on 908538724352 found 00000004 wanted 00000036 Csum didn't match owner ref check failed [908538724352 16384] ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space tree cache and super generation don't match, space cache will be invalidated [4/7] checking fs roots checksum verify failed on 908538724352 found 00000004 wanted 00000036 checksum verify failed on 908538724352 found 00000004 wanted 00000036 checksum verify failed on 908538724352 found 00000004 wanted 00000036 Csum didn't match So how to fix this to restore the cache content? thanks! serva4-diagnostics-20210427-1859.zip Quote
JorgeB Posted April 27, 2021 Posted April 27, 2021 Start by running memtest, then there are some recovery options here. Quote
corgan Posted April 27, 2021 Author Posted April 27, 2021 (edited) Thanks. I could manage to mount the cache and did a backup. Unraid can mount the cache, but only read only If i try to btrfs check --repair /dev/sdb1 i still get "Csum didn't match" errors. Quote root@serva4:~# btrfs check --repair /dev/sdb1 enabling repair mode 10 9 8 7 6 5 4 3 2 1 Starting repair. Opening filesystem to check... Checking filesystem on /dev/sdb1 UUID: 2ba1b265-e5ab-4e06-bafe-828647a53262 [1/7] checking root items Fixed 0 roots. [2/7] checking extents checksum verify failed on 908538724352 found 00000004 wanted 00000036 checksum verify failed on 908538724352 found 00000004 wanted 00000036 checksum verify failed on 908538724352 found 00000004 wanted 00000036 Csum didn't match owner ref check failed [908538724352 16384] repair deleting extent record: key [908538724352,168,16384] adding new tree backref on start 908538724352 len 16384 parent 0 root 7 Repaired extent references for 908538724352 No device size related problem found [3/7] checking free space tree cache and super generation don't match, space cache will be invalidated [4/7] checking fs roots checksum verify failed on 908538724352 found 00000004 wanted 00000036 checksum verify failed on 908538724352 found 00000004 wanted 00000036 checksum verify failed on 908538724352 found 00000004 wanted 00000036 Csum didn't match . . . Csum didn't match [5/7] checking only csums items (without verifying data) checksum verify failed on 908538724352 found 00000004 wanted 00000036 checksum verify failed on 908538724352 found 00000004 wanted 00000036 checksum verify failed on 908538724352 found 00000004 wanted 00000036 Csum didn't match Error going to next leaf -5 [6/7] checking root refs [7/7] checking quota groups skipped (not enabled on this FS) found 693552488448 bytes used, no error found total csum bytes: 587268412 total tree bytes: 1140621312 total fs tree bytes: 351305728 total extent tree bytes: 110755840 btree space waste bytes: 154000473 file data blocks allocated: 1862878056448 referenced 653472481280 root@serva4:~# i found some websites that i should use Quote btrfs check --fix-crc but i dont have the option. Any Ideas? Edited April 27, 2021 by corgan Quote
corgan Posted April 27, 2021 Author Posted April 27, 2021 (edited) some more logs Quote kernel: BTRFS info (device sdf1): turning on async discard kernel: BTRFS info (device sdf1): using free space tree kernel: BTRFS info (device sdf1): has skinny extents kernel: BTRFS info (device sdf1): bdev /dev/sdf1 errs: wr 0, rd 0, flush 0, corrupt 40073, gen 0 kernel: BTRFS info (device sdf1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 27442, gen 0 kernel: BTRFS info (device sdf1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 40657, gen 0 kernel: BTRFS info (device sdf1): bdev /dev/sde1 errs: wr 0, rd 0, flush 0, corrupt 27087, gen 0 kernel: BTRFS info (device sdf1): enabling ssd optimizations kernel: BTRFS info (device sdf1): checking UUID tree emhttpd: shcmd (38271): sync kernel: BTRFS error (device sdf1): incorrect extent count for 907568742400; counted 1346, expected 1337 kernel: BTRFS: error (device sdf1) in convert_free_space_to_extents:454: errno=-5 IO failure kernel: BTRFS info (device sdf1): forced readonly kernel: BTRFS: error (device sdf1) in remove_from_free_space_tree:847: errno=-5 IO failure kernel: BTRFS: error (device sdf1) in btrfs_run_delayed_refs:2144: errno=-5 IO failure kernel: BTRFS warning (device sdf1): Skipping commit of aborted transaction. kernel: BTRFS: error (device sdf1) in cleanup_transaction:1942: errno=-5 IO failure kernel: BTRFS error (device sdf1): incorrect extent count for 907568742400; counted 1347, expected 1338 Edited April 27, 2021 by corgan Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.