1 of 3 cache drives not actively used (issue)


johner

Recommended Posts

Hi Gurus,

 

So I have 3 mixed sized drives in a cache pool, they all report as part of a pool, but one is 100%/permantently spun down (although SSD), and never shows activity, please see snapshot below.

image.thumb.png.c034b6776762cffe4e49ea9f1b2056ab.png

 

Disk log for the unused disk is:

Oct 7 11:06:21 Tower kernel: sd 9:0:17:0: [sdae] 1000215216 512-byte logical blocks: (512 GB/477 GiB)
Oct 7 11:06:21 Tower kernel: sd 9:0:17:0: [sdae] Write Protect is off
Oct 7 11:06:21 Tower kernel: sd 9:0:17:0: [sdae] Mode Sense: 7f 00 10 08
Oct 7 11:06:21 Tower kernel: sd 9:0:17:0: [sdae] Write cache: enabled, read cache: enabled, supports DPO and FUA
Oct 7 11:06:21 Tower kernel: sdae: sdae1
Oct 7 11:06:21 Tower kernel: sd 9:0:17:0: [sdae] Attached SCSI disk
Oct 7 11:06:38 Tower emhttpd: OCZ-VERTEX4_OCZ-06K8P193JCCO7A2R (sdae) 512 1000215216
Oct 7 11:06:38 Tower emhttpd: import 32 cache device: (sdae) OCZ-VERTEX4_OCZ-06K8P193JCCO7A2R
Oct 7 11:22:33 Tower emhttpd: shcmd (169): /usr/sbin/hdparm -y /dev/sdae
Oct 7 11:22:33 Tower root: /dev/sdae:

for the 240gb:

Oct 7 11:06:21 Tower kernel: sd 9:0:19:0: [sdag] 468862128 512-byte logical blocks: (240 GB/224 GiB)
Oct 7 11:06:21 Tower kernel: sd 9:0:19:0: [sdag] Write Protect is off
Oct 7 11:06:21 Tower kernel: sd 9:0:19:0: [sdag] Mode Sense: 7f 00 10 08
Oct 7 11:06:21 Tower kernel: sd 9:0:19:0: [sdag] Write cache: enabled, read cache: enabled, supports DPO and FUA
Oct 7 11:06:21 Tower kernel: sdag: sdag1
Oct 7 11:06:21 Tower kernel: sd 9:0:19:0: [sdag] Attached SCSI disk
Oct 7 11:06:21 Tower kernel: BTRFS: device fsid 19b3ef51-b232-46af-ab0b-1bc724bd5a50 devid 2 transid 310842 /dev/sdag1
Oct 7 11:06:38 Tower emhttpd: OCZ-AGILITY3_OCZ-70M4R8854EQHXOE1 (sdag) 512 468862128
Oct 7 11:06:38 Tower emhttpd: import 31 cache device: (sdag) OCZ-AGILITY3_OCZ-70M4R8854EQHXOE1
Oct 7 11:07:27 Tower kernel: BTRFS info (device sdad1): new size for /dev/sdag1 is 240057376768

 for the 480gb: (which has some csum errors, hence my concern with this other disk not supporting the mirror)

Oct 7 11:06:21 Tower kernel: sd 9:0:16:0: [sdad] 937703088 512-byte logical blocks: (480 GB/447 GiB)
Oct 7 11:06:21 Tower kernel: sd 9:0:16:0: [sdad] 4096-byte physical blocks
Oct 7 11:06:21 Tower kernel: sd 9:0:16:0: [sdad] Write Protect is off
Oct 7 11:06:21 Tower kernel: sd 9:0:16:0: [sdad] Mode Sense: 7f 00 10 08
Oct 7 11:06:21 Tower kernel: sd 9:0:16:0: [sdad] Write cache: enabled, read cache: enabled, supports DPO and FUA
Oct 7 11:06:21 Tower kernel: sdad: sdad1
Oct 7 11:06:21 Tower kernel: sd 9:0:16:0: [sdad] Attached SCSI disk
Oct 7 11:06:21 Tower kernel: BTRFS: device fsid 19b3ef51-b232-46af-ab0b-1bc724bd5a50 devid 1 transid 310842 /dev/sdad1
Oct 7 11:06:38 Tower emhttpd: Crucial_CT480M500SSD1_134209547BCB (sdad) 512 937703088
Oct 7 11:06:38 Tower emhttpd: import 30 cache device: (sdad) Crucial_CT480M500SSD1_134209547BCB
Oct 7 11:07:27 Tower kernel: BTRFS info (device sdad1): allowing degraded mounts
Oct 7 11:07:27 Tower kernel: BTRFS info (device sdad1): disk space caching is enabled
Oct 7 11:07:27 Tower kernel: BTRFS info (device sdad1): has skinny extents
Oct 7 11:07:27 Tower kernel: BTRFS warning (device sdad1): devid 3 uuid 0e7d031f-09af-4f1b-bf4a-f215b1dd5671 is missing
Oct 7 11:07:27 Tower kernel: BTRFS info (device sdad1): bdev (null) errs: wr 158901028, rd 143240662, flush 620972, corrupt 0, gen 0
Oct 7 11:07:27 Tower kernel: BTRFS info (device sdad1): enabling ssd optimizations
Oct 7 11:07:27 Tower kernel: BTRFS info (device sdad1): resizing devid 1
Oct 7 11:07:27 Tower kernel: BTRFS info (device sdad1): new size for /dev/sdad1 is 480103948288
Oct 7 11:07:27 Tower kernel: BTRFS info (device sdad1): resizing devid 2
Oct 7 11:07:27 Tower kernel: BTRFS info (device sdad1): new size for /dev/sdag1 is 240057376768
Oct 7 11:07:27 Tower kernel: BTRFS info (device sdad1): relocating block group 491929993216 flags data|raid1
Oct 7 11:07:28 Tower kernel: BTRFS warning (device sdad1): csum failed root -9 ino 257 off 320339968 csum 0x3d37892e expected csum 0x7295014c mirror 2
Oct 7 11:07:28 Tower kernel: BTRFS warning (device sdad1): csum failed root -9 ino 257 off 320405504 csum 0x7980e4e9 expected csum 0xb4e9a8bb mirror 2
Oct 7 11:07:28 Tower kernel: BTRFS warning (device sdad1): csum failed root -9 ino 257 off 320299008 csum 0x0bb1ddc3 expected csum 0x71b5452d mirror 2
Oct 7 11:07:28 Tower kernel: BTRFS warning (device sdad1): csum failed root -9 ino 257 off 320344064 csum 0x489997f5 expected csum 0x07070da0 mirror 2
Oct 7 11:07:28 Tower kernel: BTRFS warning (device sdad1): csum failed root -9 ino 257 off 320409600 csum 0xb3a942a9 expected csum 0x48fa4231 mirror 2
Oct 7 11:07:28 Tower kernel: BTRFS warning (device sdad1): csum failed root -9 ino 257 off 320303104 csum 0xb770352e expected csum 0x6eddc163 mirror 2
Oct 7 11:07:28 Tower kernel: BTRFS warning (device sdad1): csum failed root -9 ino 257 off 320348160 csum 0x7236cd0a expected csum 0x63417da9 mirror 2
Oct 7 11:07:28 Tower kernel: BTRFS warning (device sdad1): csum failed root -9 ino 257 off 320413696 csum 0xcdf9b037 expected csum 0xa19e2b25 mirror 2
Oct 7 11:07:28 Tower kernel: BTRFS warning (device sdad1): csum failed root -9 ino 257 off 320352256 csum 0x8e04e6e0 expected csum 0x031711d6 mirror 2
Oct 7 11:07:28 Tower kernel: BTRFS warning (device sdad1): csum failed root -9 ino 257 off 320307200 csum 0xbc4ea35d expected csum 0xbf9066fb mirror 2
Oct 10 04:27:19 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740858368 csum 0x3d37892e expected csum 0x7295014c mirror 2
Oct 10 04:27:19 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740923904 csum 0x7980e4e9 expected csum 0xb4e9a8bb mirror 2
Oct 10 04:27:19 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740817408 csum 0x0bb1ddc3 expected csum 0x71b5452d mirror 2
Oct 10 04:27:19 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740862464 csum 0x489997f5 expected csum 0x07070da0 mirror 2
Oct 10 04:27:19 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740928000 csum 0xb3a942a9 expected csum 0x48fa4231 mirror 2
Oct 10 04:27:19 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740821504 csum 0xb770352e expected csum 0x6eddc163 mirror 2
Oct 10 04:27:19 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740866560 csum 0x7236cd0a expected csum 0x63417da9 mirror 2
Oct 10 04:27:19 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740825600 csum 0xbc4ea35d expected csum 0xbf9066fb mirror 2
Oct 10 04:27:19 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740932096 csum 0xcdf9b037 expected csum 0xa19e2b25 mirror 2
Oct 10 04:27:19 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740870656 csum 0x8e04e6e0 expected csum 0x031711d6 mirror 2
Oct 10 07:24:07 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740858368 csum 0x3d37892e expected csum 0x7295014c mirror 2
Oct 10 07:24:07 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740817408 csum 0x0bb1ddc3 expected csum 0x71b5452d mirror 2
Oct 10 07:24:07 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740928000 csum 0xb3a942a9 expected csum 0x48fa4231 mirror 2
Oct 10 07:24:07 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740862464 csum 0x489997f5 expected csum 0x07070da0 mirror 2
Oct 10 07:24:07 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740821504 csum 0xb770352e expected csum 0x6eddc163 mirror 2
Oct 10 07:24:07 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740932096 csum 0xcdf9b037 expected csum 0xa19e2b25 mirror 2
Oct 10 07:24:07 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740866560 csum 0x7236cd0a expected csum 0x63417da9 mirror 2
Oct 10 07:24:07 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740936192 csum 0x53e18b96 expected csum 0x73befbd4 mirror 2
Oct 10 07:24:07 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740870656 csum 0x8e04e6e0 expected csum 0x031711d6 mirror 2
Oct 10 07:24:07 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740825600 csum 0xbc4ea35d expected csum 0xbf9066fb mirror 2
Oct 11 04:20:01 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740858368 csum 0x3d37892e expected csum 0x7295014c mirror 2
Oct 11 04:20:01 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740817408 csum 0x0bb1ddc3 expected csum 0x71b5452d mirror 2
Oct 11 04:20:01 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740862464 csum 0x489997f5 expected csum 0x07070da0 mirror 2
Oct 11 04:20:01 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740821504 csum 0xb770352e expected csum 0x6eddc163 mirror 2
Oct 11 04:20:01 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740866560 csum 0x7236cd0a expected csum 0x63417da9 mirror 2
Oct 11 04:20:01 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740870656 csum 0x8e04e6e0 expected csum 0x031711d6 mirror 2
Oct 11 04:20:01 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740825600 csum 0xbc4ea35d expected csum 0xbf9066fb mirror 2
Oct 11 04:20:01 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740874752 csum 0xb74f1d9d expected csum 0xe7ee1d7f mirror 2
Oct 11 04:20:01 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740878848 csum 0x2aa33539 expected csum 0x0e96d84d mirror 2
Oct 11 04:20:01 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740829696 csum 0x52601703 expected csum 0xb8d2861f mirror 2
Oct 11 05:15:44 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740858368 csum 0x3d37892e expected csum 0x7295014c mirror 2
Oct 11 05:15:44 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740923904 csum 0x7980e4e9 expected csum 0xb4e9a8bb mirror 2
Oct 11 05:15:44 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740817408 csum 0x0bb1ddc3 expected csum 0x71b5452d mirror 2
Oct 11 05:15:44 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740928000 csum 0xb3a942a9 expected csum 0x48fa4231 mirror 2
Oct 11 05:15:44 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740862464 csum 0x489997f5 expected csum 0x07070da0 mirror 2
Oct 11 05:15:44 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740821504 csum 0xb770352e expected csum 0x6eddc163 mirror 2
Oct 11 05:15:44 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740932096 csum 0xcdf9b037 expected csum 0xa19e2b25 mirror 2
Oct 11 05:15:44 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740866560 csum 0x7236cd0a expected csum 0x63417da9 mirror 2
Oct 11 05:15:44 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740870656 csum 0x8e04e6e0 expected csum 0x031711d6 mirror 2
Oct 11 05:15:44 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740936192 csum 0x53e18b96 expected csum 0x73befbd4 mirror 2
Oct 12 03:42:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740858368 csum 0x3d37892e expected csum 0x7295014c mirror 2
Oct 12 03:42:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740923904 csum 0x7980e4e9 expected csum 0xb4e9a8bb mirror 2
Oct 12 03:42:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740817408 csum 0x0bb1ddc3 expected csum 0x71b5452d mirror 2
Oct 12 03:42:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740862464 csum 0x489997f5 expected csum 0x07070da0 mirror 2
Oct 12 03:42:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740928000 csum 0xb3a942a9 expected csum 0x48fa4231 mirror 2
Oct 12 03:42:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740821504 csum 0xb770352e expected csum 0x6eddc163 mirror 2
Oct 12 03:42:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740866560 csum 0x7236cd0a expected csum 0x63417da9 mirror 2
Oct 12 03:42:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740932096 csum 0xcdf9b037 expected csum 0xa19e2b25 mirror 2
Oct 12 03:42:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740870656 csum 0x8e04e6e0 expected csum 0x031711d6 mirror 2
Oct 12 03:42:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740936192 csum 0x53e18b96 expected csum 0x73befbd4 mirror 2
Oct 12 04:47:26 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740858368 csum 0x3d37892e expected csum 0x7295014c mirror 2
Oct 12 04:47:26 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740817408 csum 0x0bb1ddc3 expected csum 0x71b5452d mirror 2
Oct 12 04:47:26 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740928000 csum 0xb3a942a9 expected csum 0x48fa4231 mirror 2
Oct 12 04:47:26 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740862464 csum 0x489997f5 expected csum 0x07070da0 mirror 2
Oct 12 04:47:26 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740821504 csum 0xb770352e expected csum 0x6eddc163 mirror 2
Oct 12 04:47:26 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740866560 csum 0x7236cd0a expected csum 0x63417da9 mirror 2
Oct 12 04:47:26 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740932096 csum 0xcdf9b037 expected csum 0xa19e2b25 mirror 2
Oct 12 04:47:26 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740936192 csum 0x53e18b96 expected csum 0x73befbd4 mirror 2
Oct 12 04:47:26 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740870656 csum 0x8e04e6e0 expected csum 0x031711d6 mirror 2
Oct 12 04:47:26 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740825600 csum 0xbc4ea35d expected csum 0xbf9066fb mirror 2
Oct 13 04:05:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740858368 csum 0x3d37892e expected csum 0x7295014c mirror 2
Oct 13 04:05:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740817408 csum 0x0bb1ddc3 expected csum 0x71b5452d mirror 2
Oct 13 04:05:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740862464 csum 0x489997f5 expected csum 0x07070da0 mirror 2
Oct 13 04:05:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740821504 csum 0xb770352e expected csum 0x6eddc163 mirror 2
Oct 13 04:05:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740866560 csum 0x7236cd0a expected csum 0x63417da9 mirror 2
Oct 13 04:05:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740870656 csum 0x8e04e6e0 expected csum 0x031711d6 mirror 2
Oct 13 04:05:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740825600 csum 0xbc4ea35d expected csum 0xbf9066fb mirror 2
Oct 13 04:05:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740874752 csum 0xb74f1d9d expected csum 0xe7ee1d7f mirror 2
Oct 13 04:05:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740829696 csum 0x52601703 expected csum 0xb8d2861f mirror 2
Oct 13 04:05:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740878848 csum 0x2aa33539 expected csum 0x0e96d84d mirror 2
Oct 13 06:57:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740858368 csum 0x3d37892e expected csum 0x7295014c mirror 2
Oct 13 06:57:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740817408 csum 0x0bb1ddc3 expected csum 0x71b5452d mirror 2
Oct 13 06:57:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740862464 csum 0x489997f5 expected csum 0x07070da0 mirror 2
Oct 13 06:57:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740928000 csum 0xb3a942a9 expected csum 0x48fa4231 mirror 2
Oct 13 06:57:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740821504 csum 0xb770352e expected csum 0x6eddc163 mirror 2
Oct 13 06:57:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740932096 csum 0xcdf9b037 expected csum 0xa19e2b25 mirror 2
Oct 13 06:57:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740866560 csum 0x7236cd0a expected csum 0x63417da9 mirror 2
Oct 13 06:57:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740936192 csum 0x53e18b96 expected csum 0x73befbd4 mirror 2
Oct 13 06:57:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740870656 csum 0x8e04e6e0 expected csum 0x031711d6 mirror 2
Oct 13 06:57:57 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740825600 csum 0xbc4ea35d expected csum 0xbf9066fb mirror 2
Oct 15 04:11:32 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740858368 csum 0x3d37892e expected csum 0x7295014c mirror 2
Oct 15 04:11:32 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740923904 csum 0x7980e4e9 expected csum 0xb4e9a8bb mirror 2
Oct 15 04:11:32 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740817408 csum 0x0bb1ddc3 expected csum 0x71b5452d mirror 2
Oct 15 04:11:32 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740862464 csum 0x489997f5 expected csum 0x07070da0 mirror 2
Oct 15 04:11:32 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740928000 csum 0xb3a942a9 expected csum 0x48fa4231 mirror 2
Oct 15 04:11:32 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740821504 csum 0xb770352e expected csum 0x6eddc163 mirror 2
Oct 15 04:11:32 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740866560 csum 0x7236cd0a expected csum 0x63417da9 mirror 2
Oct 15 04:11:32 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740932096 csum 0xcdf9b037 expected csum 0xa19e2b25 mirror 2
Oct 15 04:11:32 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740870656 csum 0x8e04e6e0 expected csum 0x031711d6 mirror 2
Oct 15 04:11:32 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740936192 csum 0x53e18b96 expected csum 0x73befbd4 mirror 2
Oct 15 06:49:35 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740858368 csum 0x3d37892e expected csum 0x7295014c mirror 2
Oct 15 06:49:35 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740817408 csum 0x0bb1ddc3 expected csum 0x71b5452d mirror 2
Oct 15 06:49:35 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740928000 csum 0xb3a942a9 expected csum 0x48fa4231 mirror 2
Oct 15 06:49:35 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740821504 csum 0xb770352e expected csum 0x6eddc163 mirror 2
Oct 15 06:49:35 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740862464 csum 0x489997f5 expected csum 0x07070da0 mirror 2
Oct 15 06:49:35 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740932096 csum 0xcdf9b037 expected csum 0xa19e2b25 mirror 2
Oct 15 06:49:35 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740866560 csum 0x7236cd0a expected csum 0x63417da9 mirror 2
Oct 15 06:49:35 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740870656 csum 0x8e04e6e0 expected csum 0x031711d6 mirror 2
Oct 15 06:49:35 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740936192 csum 0x53e18b96 expected csum 0x73befbd4 mirror 2
Oct 15 06:49:35 Tower kernel: BTRFS warning (device sdad1): csum failed root 5 ino 19007 off 47740874752 csum 0xb74f1d9d expected csum 0xe7ee1d7f mirror 2

Diagnostics zip is being generated, it's still running after 20m.... I'll upload once complete.

 

EDIT: Attached diags download zip.

 

Thanks in advance!!

John

tower-diagnostics-20201015-1133.zip

Edited by johner
Added diags zip
Link to comment
6 hours ago, johner said:

so I 'think' it's not actually part of the poo

It's not, it's missing:

 

              Data      Metadata  System              
Id Path       RAID1     RAID1     RAID1    Unallocated
-- ---------- --------- --------- -------- -----------
 1 /dev/sdad1 229.00GiB   2.00GiB 32.00MiB   216.10GiB
 2 /dev/sdag1  83.00GiB   2.00GiB 32.00MiB   138.54GiB
 3 missing    146.00GiB         -        -  -146.00GiB
-- ---------- --------- --------- -------- -----------
   Total      229.00GiB   2.00GiB 32.00MiB   208.64GiB
   Used       197.49GiB 486.09MiB 48.00KiB  

It likely dropped offline at some point in the past but it's not being deleted possibly because of checksum errors, please run a scrub on the pool and post the results.

 

Link to comment

Hi, ah ok thanks.

 

I ran scrub once and it aborted with 42 uncorrectable errors. I ran it again:

 

UUID:             19b3ef51-b232-46af-ab0b-1bc724bd5a50
Scrub started:    Thu Oct 15 18:25:53 2020
Status:           aborted
Duration:         0:07:55
Total to scrub:   396.11GiB
Rate:             564.68MiB/s
Error summary:    csum=42
  Corrected:      0
  Uncorrectable:  0
  Unverified:     0

Aborted again but no uncorrectable errors this time. I didn't see if it actually got to the end of the run, the duration is a little short of what it predicted.

Edited by johner
typo
Link to comment

Ok so here is the output from scrub 1:

Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250333184 on dev /dev/sdad1
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250292224 on dev /dev/sdad1
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250337280 on dev /dev/sdad1
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250296320 on dev /dev/sdad1
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250341376 on dev /dev/sdad1
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250300416 on dev /dev/sdad1
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250345472 on dev /dev/sdad1
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250304512 on dev /dev/sdad1
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250349568 on dev /dev/sdad1
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250308608 on dev /dev/sdad1
Oct 15 18:16:49 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 11, gen 0

And scrub 2:

Oct 15 18:27:32 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 43, gen 0
Oct 15 18:27:32 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 44, gen 0
Oct 15 18:27:32 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 46, gen 0
Oct 15 18:27:32 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 46, gen 0
Oct 15 18:27:32 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 47, gen 0
Oct 15 18:27:32 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 48, gen 0
Oct 15 18:27:32 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 49, gen 0
Oct 15 18:27:32 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 50, gen 0
Oct 15 18:27:33 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 51, gen 0
Oct 15 18:27:33 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 52, gen 0

I think the different outcome was because I didn't click the repair option the 2nd time.

 

From a google search I think the above means that the issue is with the metadata as no file names are listed, is this your understanding?

Link to comment

So if it is the metadata that is corrupt (see previous reply), how will it manifest to Unraid given I'm running appdata etc. on here for the dockers and VM (just one at the mo). I'm running a nightly backup using the CA plugin (to the spinning rust unraid array), but everything seems to work, nothing 'functionally' complains with the dockers/VM.

 

Therefore I'm not sure if my backup is good, as there are no issues other than the csum errors - unless I'm missing some errors in the syslog that are indirectly related to these csum issues but I can't see anything obvious.

Link to comment

Apologies! Ok I've run it again, I only see 1 file impacted, if it is just this one then that is a result, as that file is the old backup from when i moved from Prox to Unraid, so it can go.

 

@JorgeB How do I tell if any metadata is corrupt? I note 42 csum errors, but not 42 line entries:

 

Oct 16 18:51:09 Tower ool www[10213]: /usr/local/emhttp/plugins/dynamix/scripts/btrfs_scrub 'start' '/mnt/cache' ''
Oct 16 18:53:25 Tower kernel: scrub_handle_errored_block: 32 callbacks suppressed
Oct 16 18:53:25 Tower kernel: BTRFS warning (device sdad1): checksum error at logical 492250292224 on dev /dev/sdad1, physical 49713471488, root 5, inode 19007, offset 47740817408, length 4096, links 1 (path: appdata/Plex-Media-Server/plexlib.zip)
Oct 16 18:53:25 Tower kernel: btrfs_dev_stat_print_on_error: 32 callbacks suppressed
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 85, gen 0
Oct 16 18:53:25 Tower kernel: scrub_handle_errored_block: 32 callbacks suppressed
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250292224 on dev /dev/sdad1
Oct 16 18:53:25 Tower kernel: BTRFS warning (device sdad1): checksum error at logical 492250333184 on dev /dev/sdad1, physical 49713512448, root 5, inode 19007, offset 47740858368, length 4096, links 1 (path: appdata/Plex-Media-Server/plexlib.zip)
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 86, gen 0
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250333184 on dev /dev/sdad1
Oct 16 18:53:25 Tower kernel: BTRFS warning (device sdad1): checksum error at logical 492250296320 on dev /dev/sdad1, physical 49713475584, root 5, inode 19007, offset 47740821504, length 4096, links 1 (path: appdata/Plex-Media-Server/plexlib.zip)
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 87, gen 0
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250296320 on dev /dev/sdad1
Oct 16 18:53:25 Tower kernel: BTRFS warning (device sdad1): checksum error at logical 492250337280 on dev /dev/sdad1, physical 49713516544, root 5, inode 19007, offset 47740862464, length 4096, links 1 (path: appdata/Plex-Media-Server/plexlib.zip)
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 88, gen 0
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250337280 on dev /dev/sdad1
Oct 16 18:53:25 Tower kernel: BTRFS warning (device sdad1): checksum error at logical 492250300416 on dev /dev/sdad1, physical 49713479680, root 5, inode 19007, offset 47740825600, length 4096, links 1 (path: appdata/Plex-Media-Server/plexlib.zip)
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 89, gen 0
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250300416 on dev /dev/sdad1
Oct 16 18:53:25 Tower kernel: BTRFS warning (device sdad1): checksum error at logical 492250341376 on dev /dev/sdad1, physical 49713520640, root 5, inode 19007, offset 47740866560, length 4096, links 1 (path: appdata/Plex-Media-Server/plexlib.zip)
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 90, gen 0
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250341376 on dev /dev/sdad1
Oct 16 18:53:25 Tower kernel: BTRFS warning (device sdad1): checksum error at logical 492250304512 on dev /dev/sdad1, physical 49713483776, root 5, inode 19007, offset 47740829696, length 4096, links 1 (path: appdata/Plex-Media-Server/plexlib.zip)
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 91, gen 0
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250304512 on dev /dev/sdad1
Oct 16 18:53:25 Tower kernel: BTRFS warning (device sdad1): checksum error at logical 492250345472 on dev /dev/sdad1, physical 49713524736, root 5, inode 19007, offset 47740870656, length 4096, links 1 (path: appdata/Plex-Media-Server/plexlib.zip)
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 92, gen 0
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250345472 on dev /dev/sdad1
Oct 16 18:53:25 Tower kernel: BTRFS warning (device sdad1): checksum error at logical 492250308608 on dev /dev/sdad1, physical 49713487872, root 5, inode 19007, offset 47740833792, length 4096, links 1 (path: appdata/Plex-Media-Server/plexlib.zip)
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 93, gen 0
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250308608 on dev /dev/sdad1
Oct 16 18:53:25 Tower kernel: BTRFS warning (device sdad1): checksum error at logical 492250349568 on dev /dev/sdad1, physical 49713528832, root 5, inode 19007, offset 47740874752, length 4096, links 1 (path: appdata/Plex-Media-Server/plexlib.zip)
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): bdev /dev/sdad1 errs: wr 0, rd 0, flush 0, corrupt 94, gen 0
Oct 16 18:53:25 Tower kernel: BTRFS error (device sdad1): unable to fixup (regular) error at logical 492250349568 on dev /dev/sdad1

The result from the scrum states 'aborted' again, does this mean it only found a sub set of the 42 errors, therefore there could be more damaged files I don't yet know about?

 

I'll delete the above file and scrub again.

 

UUID:             19b3ef51-b232-46af-ab0b-1bc724bd5a50
Scrub started:    Fri Oct 16 18:51:09 2020
Status:           aborted
Duration:         0:09:13
Total to scrub:   396.98GiB
Rate:             485.58MiB/s
Error summary:    csum=42
  Corrected:      0
  Uncorrectable:  42
  Unverified:     0

 

Link to comment

Ok I deleted the file, and ran scrum from the cli:

 

I think the abort is only on the missing disk, the 'aggregate' status is therefore aborted.

 

root@Tower:~# btrfs scrub status -d /mnt/cache/
UUID:             19b3ef51-b232-46af-ab0b-1bc724bd5a50
scrub device /dev/sdad1 (id 1) history
Scrub started:    Fri Oct 16 19:42:00 2020
Status:           finished
Duration:         0:08:00
Total to scrub:   197.03GiB
Rate:             284.85MiB/s
Error summary:    no errors found
scrub device /dev/sdag1 (id 2) history
Scrub started:    Fri Oct 16 19:42:00 2020
Status:           finished
Duration:         0:01:14
Total to scrub:   55.03GiB
Rate:             271.81MiB/s
Error summary:    no errors found
scrub device  (id 3) history
Scrub started:    Fri Oct 16 19:42:00 2020
Status:           aborted
Duration:         0:00:00
Total to scrub:   142.00GiB
Rate:             0.00B/s
Error summary:    no errors found

Now I need to figure out how to add the missing disk back in... google here I come...

Edited by johner
Link to comment
On 10/15/2020 at 9:13 AM, JorgeB said:

It's not, it's missing:

This status shows a RAID1. Your upper screenshot is more something like JBOD or RAID0?!

 

I would move all data to the array and restart the cache pool configuration. And if course you should choose a RAID5 setup, but this will result in only 480GB total size (the smallest disk defines the size per disk). So finally you could drop the 240GB disk and build a RAID1 of the biggest two disks.

Link to comment
  • 2 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.