October 21, 20241 yr Hello, I realized several Docker services were down, and was surprised to see the Docker service had failed. Then looking at the array status, the cache pool of 2 SATA ssd's (BTRFS in raid 1) shows as "unmountable." I noticed several other issues on this forum with the same issue recently but am concerned about deleting data. What I've done so far: Stopped the array, removed the cache drives by setting pool slots to 0 (so it looked like the below screenshot), restarted the array, stopped the array, added cache drives back, restarted the array in maintenance mode Couple questions: 1. Was this the correct way to remove/re-add the cache pool? 2. The next step in the filesystem check is to run the btrfs check without the --readonly option, is there anything else I should do before this to ensure I don't cause further issues? I suspect a power loss was the root cause, the devices exist so I doubt an ssd hardware issue, (this was run while in maintenance mode) btrfs fi show Label: none uuid: b9325cba-fcee-4e86-8250-caa493fa4a24 Total devices 2 FS bytes used 99.92GiB devid 1 size 238.47GiB used 143.03GiB path /dev/sdb1 devid 2 size 238.47GiB used 143.03GiB path /dev/sdc1 After restarting in maintenance mode this is the output of the the filesystem check. btrfs check --readonly /dev/sdb1 Opening filesystem to check... Checking filesystem on /dev/sdb1 UUID: b9325cba-fcee-4e86-8250-caa493fa4a24 [1/7] checking root items [2/7] checking extents data extent[178439438336, 16384] referencer count mismatch (root 5 owner 260 offset 513245184) wanted 0 have 1 data extent[178439438336, 16384] bytenr mimsmatch, extent item bytenr 178439438336 file item bytenr 0 data extent[178439438336, 16384] referencer count mismatch (root 5 owner 260 offset 34872983552) wanted 1 have 0 backpointer mismatch on [178439438336 16384] ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space tree [4/7] checking fs roots [5/7] checking only csums items (without verifying data) [6/7] checking root refs [7/7] checking quota groups skipped (not enabled on this FS) found 107287035904 bytes used, error(s) found total csum bytes: 50747720 total tree bytes: 141934592 total fs tree bytes: 66371584 total extent tree bytes: 16416768 btree space waste bytes: 21003679 file data blocks allocated: 191448006656 referenced 93518864384 btrfs check --readonly /dev/sdc1 Opening filesystem to check... Checking filesystem on /dev/sdc1 UUID: b9325cba-fcee-4e86-8250-caa493fa4a24 [1/7] checking root items [2/7] checking extents data extent[178439438336, 16384] referencer count mismatch (root 5 owner 260 offset 513245184) wanted 0 have 1 data extent[178439438336, 16384] bytenr mimsmatch, extent item bytenr 178439438336 file item bytenr 0 data extent[178439438336, 16384] referencer count mismatch (root 5 owner 260 offset 34872983552) wanted 1 have 0 backpointer mismatch on [178439438336 16384] ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space tree [4/7] checking fs roots [5/7] checking only csums items (without verifying data) [6/7] checking root refs [7/7] checking quota groups skipped (not enabled on this FS) found 107287035904 bytes used, error(s) found total csum bytes: 50747720 total tree bytes: 141934592 total fs tree bytes: 66371584 total extent tree bytes: 16416768 btree space waste bytes: 21003679 file data blocks allocated: 191448006656 referenced 93518864384 system: Unraid 6.12.13 on an older i5-3570K system. unraid-diagnostics-20241021-1238.zip Edited October 21, 20241 yr by whomstdve clarified question
October 22, 20241 yr Community Expert Data corruption is being detected on both pool devices, you should run memtest, then type: btrfs rescue zero-log /dev/sdb1 And see if it mounts.
October 22, 20241 yr Author memtest did find a singular error, so unfortunately a hardware issue is likely present (the generated report is attached) btrfs rescue zero-log /dev/sdb1 The above command gave no output, checking with lsblk gave NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 63.4M 1 loop /lib loop1 7:1 0 339.2M 1 loop /usr loop2 7:2 0 1G 0 loop /etc/libvirt sda 8:0 1 7.3G 0 disk └─sda1 8:1 1 7.3G 0 part /boot sdb 8:16 0 238.5G 0 disk └─sdb1 8:17 0 238.5G 0 part sdc 8:32 0 238.5G 0 disk └─sdc1 8:33 0 238.5G 0 part sdd 8:48 0 7.3T 0 disk └─sdd1 8:49 0 7.3T 0 part sde 8:64 0 7.3T 0 disk └─sde1 8:65 0 7.3T 0 part sdf 8:80 0 7.3T 0 disk └─sdf1 8:81 0 7.3T 0 part md1p1 9:1 0 7.3T 0 md /mnt/disk1 md2p1 9:2 0 7.3T 0 md /mnt/disk2 sr0 11:0 1 1024M 0 rom After rebooting post-memtest-ing the server I noticed the same ssd's are surprisingly shown in 2 places 1. in the cache pool 2. in unassigned devices MemTest86-Report-20241022-150839.html
October 23, 20241 yr Community Expert Solution Before trying anything else you need to replace the RAM.
November 1, 20241 yr Author I managed to find some spare DDR3 that passed multiple memtest runs, so we should be all good to go! Both original ram sticks had a single corruption issue. Any thoughts on if recovery of the cache drives is possible? Not sure since the RAM was probably at fault.
November 1, 20241 yr Community Expert Post new diags after array start, and the output from btrfs fi show
November 1, 20241 yr Author btrfs fi show Label: none uuid: b9325cba-fcee-4e86-8250-caa493fa4a24 Total devices 2 FS bytes used 99.92GiB devid 1 size 238.47GiB used 120.03GiB path /dev/sdb1 devid 2 size 238.47GiB used 120.03GiB path /dev/sdc1 Looks like it worked (mostly), 120gb used instead of 143 originally, but I can browse the file explorer and although the logs had some errors I think they were mostly benign and not btrfs-related. Thanks for all your hlep unraid-diagnostics-20241101.zip
November 1, 20241 yr Author UUID: b9325cba-fcee-4e86-8250-caa493fa4a24 Scrub started: Fri Nov 1 12:13:30 2024 Status: finished Duration: 0:03:27 Total to scrub: 211.59GiB Rate: 1.02GiB/s Error summary: csum=4 Corrected: 0 Uncorrectable: 4 Unverified: 0
November 2, 20241 yr Community Expert Look in the syslog for the list of corrupt file(s), delete them or restore from a backup, then run another scrub to confirm 0 errors.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.